Here’s Everything Chalkbeat Learned About Auditing Source Diversity
Two years ago, Chalkbeat joined a growing list of newsrooms trying to keep up with the diversity of our sources so that we can better understand the perspectives we elevate in our journalism.
We started off with a little bang – two recurring sets – and quickly realized we had some work to do. We have also learned that many, many newsrooms and individual journalists do this audit work alone in silos. We wanted to learn from each other and build a better system.
In recent months, Chalkbeat and Reynolds Institute of Journalism Student Tatyana Monnay and Sara Dingmann collected information on source tracking from 58 journalists and interviewed 14.
The journalists interviewed represent newsrooms ranging from small startups to old radio stations. Some have checked the sources for years, others had only started their process for a few months, and others performed periodic spot checks every few months.
The majority of those interviewed were from the public media, which has so far dominated the field of building source audits. All newsrooms assessed the portrayal of the people they interviewed and compared these results with their target audience and communities.
The information collected will guide Chalkbeat and RJI in creating an effective, sustainable and repeatable source diversity tracking tool. Want to get involved or receive updates on this project? Let us know here.
Lesson 1: Newsrooms need a clear direction at the start on the goals of source tracking, data collection, and how to use the results.
An effective source verification system should fit into a newsroom’s larger plan to diversify its staff and coverage to better reflect the communities it serves. Many newsrooms have launched source diversity audits in the past year (10 of 14 newsrooms surveyed) – and many audits are part of the newsroom’s broader diversity efforts, equity, inclusion and belonging, or DEIB.
KUER in Utah, for example, launched its audit this year as part of a larger newsroom culture project. And WHYY in Philadelphia paired up the staff cultural competence training with the extension of its source audits over three years.
But many audits didn’t have that kind of clear focus or connection to larger organizational initiatives, and afterward they seemed to be one-off projects that didn’t have a significant impact on the newsroom. Source tracking work can be tedious and only matters if it helps us understand our own models and blind spots as journalists and helps us hold ourselves accountable for more inclusive coverage.
Only a few of the newsrooms surveyed had been following the sources for more than a year – and most still struggled to compare their data – for example, should newsrooms use census data to assess results ? What is the correct reference?
Many newsrooms are also developing systems on how to use the results internally and publicly: when do you review the data with staff? Do you set goals for individual journalists and stick to them? Will you share your results publicly?
Key points to remember: Explaining the why and how of any source audit project is critical to its success. Rapporteurs should be involved in the design of audits or monitoring systems and in the analysis of results. Newsrooms that had these conversations with reporters early in the process seemed to get more buy-in and support from staff.
At the same time, newsroom managers need to develop a clear plan on how to embed results into culture and systems (e.g. will results play into the definition of journalists’ goals? Who is responsible? how and when will newsrooms publish publicly informing their audiences?)
Lesson 2: Investigations should be short and audits should be continuous.
These interviews confirmed that the shorter the survey, the better. Many newsrooms, including those that work with Amplified America, started with lengthy surveys, then narrowed down to five to eight key multiple-choice questions. Retroactive audits took longer than continuous monitoring and generally provided less reliable data.
Some newsrooms have used interns or college journalism programs to design the audit. KQED worked with Impact Architects to establish a baseline, and The Philadelphia Inquirer worked with Temple University to analyze 3,000 journalistic articles. While resorting to outside or temporary assistance can make retroactive audits less overwhelming, it can be difficult for editorial staff to continue work started by an outside entity after the initial audit.
Spot checks – or tracking sources for random periods of time – attracted poorer participation from journalists and more ad hoc data. Because the controls weren’t built into the workflow and regularly applied, journalists would forget. And a news cycle could skew the data.
To remember : A brief survey that tracks name, age, gender identity, race / ethnicity, source title or category, zip code and an option to register for a database source provides enough data. (Note: Many newsrooms have struggled with source titles, which should be set before launching an investigation. For examples, see the annex to this study by the Institute for Nonprofit News).
To remember : Continuously monitor and audit as part of the reporting process. If newsrooms believe that diversity of sources is important, they should strive for the best data.
Lesson 3: Automation saves time (but doesn’t mean journalists are less invested).
There was a clear divide among those interviewed as to who should be in charge of this work and how much newsrooms should invest in automating their diversity tracking systems.
The vast majority of editorial staff surveyed used Google Forms to record survey data. Resolve Philly used Airtable. Most, like Colorado Public Radio, require reporters to collect demographic information from sources during interviews and then record it in the form.
Some interviewees believed reporters should ask demographic questions and complete the survey. Rather than simply providing email addresses, journalists can then take more ownership of the diversification of their sources.
All editorial staff relying on reporters to fill out Google forms or separately interview sources acknowledged that none could last long term. Both are time consuming, can be overlooked, and raise data security issues.
The Wisconsin Public Radio Demographic Project integrates the source data into its content management system. WPR said this doesn’t tire journalists and makes them harder to forget. Yet WPR staff spend time each month figuring out which sources have been and have not been cited before. The data comes from emailed surveys that about half of the sources fill out.
Manual and more automated processes have raised security concerns. Some newsrooms are moving away from Google Forms because they don’t trust Google to protect sensitive information. Others have raised concerns about adding personal information from sources to their CMS – the system that publishes stories – that this information could accidentally be posted online.
To remember : Newsrooms that rely on reporters to collect data reported more problems with membership and data collection – often reporters either forgot or put it off. This speaks to the heavy load of journalists and why automation can help.
To remember : WPR is the closest to creating an automated system that is far more efficient than updating Google Forms manually. But it is not seamless. It could be further automated to lighten the burden on staff. Newsrooms also need to better assess the security risks associated with collecting and storing personal data.
Lesson 4: Newsrooms should design with a living source database in mind.
Very few newsrooms initially developed a source audit with the idea of creating a source database for future articles as well. Thanks to the influence of WPR, this is becoming more and more of a trend. WPR uses Google Sheets for its Rolodex newsroom.
A WPR audit manager oversees the database and helps journalists find sources. The database prioritizes sources that identify with an area under-represented in WPR coverage, but does not include demographic information from sources to protect privacy.
News agencies designing a sustainable source audit can think about how to create a source list that elevates the historically underrepresented people in this newsroom. Just as audits help newsrooms understand who they amplify, shared databases will make it easier for journalists to expand beyond their usual sources.
To remember : Editors should consider investing in a long-term source database. Having one person or team managing the source database could make this resource easier for journalists to use, but editors and journalists would also need training and access to ensure that the database is used properly.
Chalkbeat and RJI are incorporating all of these lessons into our next phase: Building a long-term system that will work for Chalkbeat newsrooms and hopefully any other newsrooms interested in this work. Questions? Suggestions? Want to get involved or say hello? Contact us at [email protected]