As I wrote last week, Jay Rosen is conducting a survey at MediaShift Idea Lab to see how many locally produced stories newspapers turn out on a given day to get an idea of what a new model, one in which newspapers do not exist, would need to “replace”. In my last post about this, I noted that simple story counts are a poor way to gauge journalism because it’s one of those things where quantity does not equal quality. I raised the same point in a comment on Ryan Chittum’s article at CJR about the survey. In Ryan’s reply, he noted that

… a closer critical accounting would be so subjective as to be pretty useless I think unless done by the same person or with the same standards somehow.

Obviously it would be a near-impossible task to have one person conduct some sort of quality survey on a large number of newspapers. However, I’ve been thinking about the “same standards” part and whether it is possible to develop some sort of metric that can be used to “code” or quantify each newspaper story to provide a relatively objective, numerical evaluation of its quality. Obviously, any such standard would not be able to cover all the facets of what makes good journalism, but having a standardized, objective metric for quality can be useful in providing at least part of a picture of the level of journalism that would be lost if newspaper companies disappear. It would also be useful if anyone is ever to attempt such a study as it would provide a tool that doesn’t hinge on personal subjectivity and therefore can be used by anyone who participates in the study.

So what would go into such a metric? First here are some thoughts on what shouldn’t be in it:

• Word count: This is the ultimate illustration of quantity not equaling quality. For just about every story I’ve written, for every word that made it into the story, I probably had at least two that didn’t, for various reasons. Any reporter can tell you that writing 20-30 inches is easy; writing 10-15 inches and saying just as much, now that’s hard, and that’s part of what good journalism is about — concise, to-the-point communication. Therefore, a higher word count doesn’t reflect quality. All it shows is that a newspaper has the room to run a longer story.
• Number of corrections: This is very dependent on each paper’s policy regarding corrections, and I’ve known some editors who were gun-shy about printing corrections. So its effectiveness as a measurement for accuracy would vary greatly from paper to paper.
• Anything that relies on subjective evaluation: It’s not that subjective evaluations are wrong, but putting these factors into the formula would defeat the purpose of this metric — developing a system that can be used by anyone and produce evaluations based on the same set of measurable factors.

As for what should go into our formula, basically I’m trying to identify quantifiable characteristics of good journalism and put them in the formula. here are some possibilities (we’ll worry about the mathematical equation later):

• Staff-produced story counts: Quantity doesn’t equal quality, but when it comes to a newspaper’s productivity, the number does have some relevance, but we should be careful not to assign too high a value to this number, for the reasons I stated in my earlier post. Also, this should be juxtaposed against the paper’s circulation (which can be found here), since bigger circulation = bigger revenue + bigger staff + bigger coverage area + more potential story ideas = (supposedly) more stories. So we should be looking at a ratio of story count to circulation to get a better idea of how productive a particular newspaper is.
• Number of sources cited in the story: This could be used as a measurement of the amount of research and legwork done by the reporter. This number can include both humans (experts, government officials, etc.) and documents. This can be a bit tricky to track, however, as it would not always be obvious just from reading the story that a reporter used a source document to obtain certain facts. To accurately measure this attribute, researchers would need to do more than just read the paper. They would need to communicate with the journalists who produced the stories to ask them how many sources they used.
• Number of typos: Obviously typos mar a story, but that’s not the main reason this is listed as one of the criteria. In my experience, typos get into newspapers because of lack of time on the part of both the reporter and the copy desk. Speaking from countless nights spent editing sports stories that are filed minutes before the presses start, the very minimum a copy editor does when editing a story is to just proofread it for typos and grammatical errors. Obviously there will be a typo that just eludes the eye now and then no matter how many times you read a story, but a high number of typos would be an indication of copy editors not having much time to spend with each story. If they didn’t have enough time to properly do the very minimum on a story, then it follows that they didn’t have time to do the more time-consuming levels of editing, which includes fact-checking, name-checking, editing to provide better context, etc. It’s for that reason that the number of typos merits inclusion in our metric.
• Number of investigative, long-term stories over a period of time: This requires tracking a paper’s content for a certain period of time (weeks, months, a year) rather than just one day, which I think is the proper way to conduct a study of newspaper productivity anyway. Look for stories that aren’t just spot/breaking news reports (e.g.: car crashes, meetings, games, etc.), but rather stories that took multiple days or weeks to plan and research. Again, this would probably require some communication with the journalists who produced the stories.
• Number of stories following up on a breaking news story: After news breaks, track how many stories the paper runs in the next X weeks/months following up on the initial report of the news that aren’t breaking-news stories. For instance, after reporting that somebody sued the city police department yesterday, how many follow-up stories does the paper run in the following month that aren’t reports on a particular day’s court proceedings? How many stories does it run that takes a deeper look into the issues in the case?
• The 5 W’s and H: Does the story meet that most basic of journalistic principles — does it tell you the who, what, where, when, why, and how? Assign a value for each one, and extra emphasis should be placed on why and how, since those are harder to get.
• Action items: How often does the paper provide information that makes it easier for readers to take action on the issue being reported? For example, in reporting about a controversial bill being debated in local government, does the paper provide contact information for local officials so its readers can reach them and make their opinions heard on the issue? For a feature about a local dance troupe, does the story provide dates for its upcoming performances, or at least a link to its Web site?

It goes without saying that this is not a all-inclusive list. So what else do you think should go into this formula?