I live-tweeted the session and Storified some key points he shared:
#bcni13: Andrew Mendelson’s "Beyond metrics" talk
Temple University journalism chair Andrew Mendelson presented a talk called "Beyond Metrics: Thinking more broadly about the measurement of journalism impact (or building journalism’s theory of change)."
Storified by Greg Linch· Mon, May 06 2013 21:16:35
This session is being led by @andrewmendelson. #bcni13Greg Linch
He’s interested in public interest, investigative, accountability journalism. "Theory of change." — @andrewmendelson #bcni13Greg Linch
Two ideas: Public interest journalism goal to create informed & engaged citizens and "journalism as a curriculum" @andrewmendelson #bcni13Greg Linch
Last year I led a more general discussion about impact, which Andrew Spittle expertly documented in his notes. Springboarding from that, Wendy Warren and I did a follow-up #BCNI12 session to dive into specific, qualitative possible news metrics for success.
The thinking and conversations about impact have grown and evolved in the past year, such as with a conversation about impact at #ONA12. With this progress, we’re going to take a much more hands-on approach at the unconference this time, as Erika wrote in her preview post:
It’s pretty easy for the conversations around impact to go in circles, but the aim is for this discussion to be more focused. Greg suggested an “impact-a-thon” format where folks share case studies, discuss them in small groups, and then report back on the findings.
Do you have any case studies of stories you’ve worked on? Analytics and/or anecdotes to share of impact of the stories? Think back to those awards application cover letters, what did you describe for why your story should be honored?
The session, rather than theory, will focus on implementation. How have organizations monitored impact, and what are some specific examples of things they can do for future stories? We look forward to an insightful, action-oriented discussion, and want your help in designing the session. Please email the Google Group, share your thoughts, and join us Saturday morning.
Brian will share his experiences thus far exploring impact measurement as an Open Newsfellow at The New York Times:
In preparation for the discussion, I’m hoping to get your thoughts and concerns about impact measurement (read up here) – what have we done right so far and what’s missing from the conversation?
If you’re interested, please up-vote/comment soon! Voting closes Thursday, April 12.
It would begin with an overview of interesting metrics in other fields, how we can learn from those and how we can possibly apply them to journalism. After that groundwork is laid, there would be a moderated discussion among the participants of other examples and how we could best implement new metrics.
With this question, I wanted to broaden the possibile metrics beyond just impact. Higlights from the discussion are below. Enjoy!
Sheree Martin offered four important questions we must first consider:
What is journalism?
What is impact?
How do we measure?
Who is measuring?
Kathy E. Gill asked about similar fundamentals, “What matters? What is the role of journalism, our purpose, our challenge?” and raised a good question of balance:
how does “but it matters!” coexist in an environment where assessment is measured in large part by short-term page views and click-throughs?
Denise Cheng noted an opportunity to measure “inert engagement,” described as “the engagement that doesn’t want to come out of hiding as big steps like shares and comments.” She said:
measuring impact by measuring engagement are manifold. First and foremost, the modus operandi on my patch of the Internet is that journalism’s highest ideal is to equip its readership with information from which they can take judicious action
Specifically, she recommended defining metrics before you embark, measuring topical importance to the audience and acts accordingly, and that the sturdier your metrics become over time, the more of a road map you have.
The root of journalism is truth, and the time-tested method that journalists have to uncover that truth is verification. If we want to measure journalism, it must begin here.
Addressing the quantitative vs. qualitative measurement distinction, he said:
Measurement assumes quantification, and some ideas — such as verification — are better evaluated qualitatively. Creating a measure requires including some attributes and excluding others; inevitably, such measures are always imperfect approximations, especially when it comes to complex concepts.
If we want to measure the impact of media and online journalism, we need to consider action. Action is what defines Effective Media (EM), and Effective Media can be measured by the Action that is a direct result of Quality Dialogue that is Shared
So, if we want to consider Impact by measuring Action, that measurement has to be proportional.
Michael Rosenblum emphasized the importance of finding niche instead of mass audiences:
As more and more content begins to fill the blogosphere and cyberspace and the cloud and wherever else ‘it’ all is, the competition for the Holy Grail of mass audience becomes ever more intense, and as such, the content itself becomes ever more amorphous.
Yet where is the ‘real’ value?
The web gives us access to discrete groups with specific interests. Our goal should be ‘narrowing the field’, not expanding it. Creating affinity groups with a common interest and common goals, and then, making it possible for those people to achieve those goal – whether its contributing to a new project – as in Kickstarter, or going on a golfing trip to St. Andrews.
Steve Outing looked at how social media is gaining an edge in the impact realm:
When I look at the question, I can’t help but get sidetracked into thinking how social media (i.e., “the crowd” utilizing digital social tools like Twitter, Facebook, and Change.org, among others, to amplify their voices) in a growing number of cases is having more impact than the traditional news media can achieve themselves — or is driving the mainstream news media to pay attention to stories that their editors fail to recognize as important.
Carnival ringmaster David Cohn also proposed an alternative approach :
I want to measure a different kind of impact. The impact of the dollars we spend in pursuit of journalism and its meaningful impact.
What we don’t appreciate is the strength of the little guy. What they don’t have in “impact” they do have in efficiency.
Steve Fox challenged the assumption of measuring something like impact:
Perhaps we all need to remember that the true “impact of journalism” rests with the impact we have on people’s lives. Have we given readers/viewers an amazing piece of writing or video that makes them appreciate parts of their life more? Have we created an “Oh, wow” moment for readers/viewer? Have we expanded someone’s universe? Isn’t that why we got into this business? Isn’t that what journalism has always done?
Perhaps the real question should be: “Why are we spending so much time measuring the “impact” of journalism?” Because, it really isn’t quantifiable now, is it?
Before Isaac Newton, words like mass and force were general descriptors, as James Gleick writes in The Information:
“the new discipline of physics could not proceed until Isaac Newton appropriated words that were ancient and vague—force, mass, motion, and even time—and gave them new meanings. Newton made these terms into quantities, suitable for use in mathematical formulas.”
The term information was similarly amorphous until Claude Shannon, while working at Bell Labs, quantified the concept in bits.
* * *
The journalism goals and business goals for news organizations are out of sync.
Pageviews. Unique visitors. Time on site.
Some journalism might be best quantified partly or wholly by one or more of those ways, but we need to explore deeper beyond these fairly simplistic metrics.
We know how these terms are defined, but what do they really mean? What do they help us achieve?
In creating a theory of information and quantifying information in bits, Shannon aimed to remove meaning. “Shannon had utterly abstracted the message from its physical details,” Gleick says.
For journalism, the goal should be to add more meaning to the information we use to measure our work. Granted, our current metrics aren’t meaningless. We use them because they do have meaning: views, comments, shares, etc. each has a meaning and can be measured based on that one-dimensional measure. The quantities of metrics increase because the works of journalism they describe are meaningful. Or, put another way, impactful.