Guardian ends run of smart science journalism discussions with scientists’ self-congratulatory essay about peer review*

Since September, I’ve been teaching in MIT’s Graduate Program in Science Writing. So far, it’s a wonderful experience — about which more in the weeks to come.

Over the past week, we’ve been talking about two intertwining themes: A journalist’s proper relationship with a source and the ways in which the Internet can foster important discussion and debate. PLoS Blog Network aficionados likely know where this is heading: Straight to “Trine Tsouderos on This Week in Virology: When do you fact-check article content with sources?,” which David Kroll posted on Take as Directed on September 19. Some very quick context: TWiV is a podcast hosted by Columbia University virologist Vincent Racaniello, who also runs Virology Blog; Tsouderos is a science reporter for the Chicago Tribune. I’ll let David take it from there:

…what got a Twitter discussion going last night and this morning was Trine’s discussing her practice of sometimes running quotes, paragraphs, and even full articles past scientists she’s interviewed for fact-checking purposes. Particularly in cases where she is interviewing someone about complex original research literature, she expressed her motivation as the desire to get it right ‘because you can’t retract 300,000 newspapers.’ (Trine, please correct me if I misrepresented what you said.)

Well, the response to this little Twitter banter leads me to think it may be valuable to bring the discussion out to more than 140-character bursts. For example, I know that Maggie Koerth-Baker, Science Editor at BoingBoing, has proposed a ScienceOnline2012 session on the problem of journalists growing too close to their sources as stimulated by her reading of the book, Wrong.

David’s post got such a voluminous response that six days (and 110 comments — and counting!) later, he posted a follow-up titled “How do scientists view fact-checking by science writers?“; that, in turn, sparked several dozen more comments, along with a handful of considered (and, um, less considered) responses, replies, replies, retorts, RTs, and rejoinders. Included among those was a Sept. 29 story in The Guardian by Nature online editor Ananyo Bhattacharya titled, “Scientists should not be allowed to copy-check stories about their work.” Alistair Dove, a senior scientist at the Atlanta Aquarium (also known by his nom-de-Twitter, @para_sight), didn’t agree with Bhattacharya’s logic; one of his first tweets about the Guardian story began with this succinct assessment: “What a poor argument.” Within a few hours, Dove had mined his and Bhattacharya’s (Twitter handle: @Ananyo) back-and-forth for a post on Deep Sea News titled “Getting on the same page as science journalists.”

So, to recap: You have a Columbia University researcher (Racaniello), who, as it happens, is something of an evangelist when it comes to making science accessible to the public. When he runs into one of the country’s best, and most fearless, science reporters (Tsouderos) at the International Conference on Antimicrobial Agents and Chemotherapy, he decides to interview her for his podcast. Much tweeting ensues; soon, the chair of the pharmaceutical sciences department at North Carolina Central University (Kroll) writes up a post about some of the chatter. The result is a freewheeling, deeply considered discussion between some of the smartest and best science writers working today, folks like Ed Yong and John Rennie and Deborah Blum and Maryn McKenna and Carl Zimmer. A direct response to all these back-and-forth results in a debate between a marine scientist (Dove) at what is perhaps the country’s preeminent aquarium and an editor (Bhattacharya) at what is arguably one of the most respected scientific publications in the world.

If that doesn’t make you proud to let your geek flag fly, I don’t know what will. It’s an open-sourced master’s class in the intersection between the scientific and journalistic processes, conducted by some of the most exciting and engaged thinkers around. So you can imagine my excitement yesterday afternoon, when I saw this tweet from @Open_Notebook (another great and valuable resource for science writers/science writing):

“Someone was having trouble with argument development here.” More on the should-scijournos-copycheck fracas bit.ly/p4I1dF

That link leads to a post on Emily Willingham’s The Biology Files, which, in turn, critiques a new Guardian story about the whole quote-check imbroglio. Emily’s piece is typically excellent. Unfortunately, that’s where this particular run of incisive commentary ends; the Guardian piece, “Scientists should be allowed to check stories on their work before publication,” is a jaw-dropping mixture of ignorance and arrogance.

Now, as I’ve hopefully made clear, I think there are many strong, even persuasive arguments for why journalists should check quotes and information with sources; there are also many strong and persuasive arguments on the other side of the ledger. The Guardian piece — written by Petroc Sumner, Frederic Boy, and Chris Chambers, all of whom are psychologists at Cardiff University in Wales — contains what just might be the single worst argument in this whole debate:

If journalists were to allow governments or companies to vet their stories, it would surely destroy the credibility of the press. Why should science be treated differently?

Science is different for four reasons, one categorical, three of degree. The categorical difference is the process of peer review. Every research article in a reputable scientific journal has been through a process in which between two and five independent experts (normally anonymous) have made extensive comments.

These ‘reviewers’ are looking for flaws and are often extremely critical. … This process typically goes through two or more rounds, with the revised article returned to the reviewers for further comments.

This system of expert critical scrutiny doesn’t exist for most other types of journalistic ‘source’ – for example, statements by politicians. So in most areas journalists are the review process (or a crucial part of it), and independence is paramount. But in science journalism…the essential role of critical review has already been performed.

Wow. Even if I hadn’t spent the last three years writing about the repercussions of Andrew Wakefield’s potentially fraudulent, ethically unsound, scientifically bankrupt Lancet study — a study which faced “expert critical scrutiny” before it appeared in print — the notion that reporters should rely on the peer review process as a substitute for their own critical evaluation is myopic at best and dangerous at worst.

And with that…we break for an actual class. More on this later today…

* Edit, October 18, 11:29 pm: The end of this headline originally read, “…publishes exercise in idiocy.” In a comment, reader (and writer) Matt Carey correctly pointed out that terms like “idiot,” “moron,” and “imbecile” are insensitive and insulting. I should have known better and I apologize for my choice of words.

Related Posts Plugin for WordPress, Blogger...

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.

This entry was posted in Journalism, Media, science and tagged , , , , , , . Bookmark the permalink.

22 Responses to Guardian ends run of smart science journalism discussions with scientists’ self-congratulatory essay about peer review*

  1. Pingback: Science Outreach Links of the Day « Galileo's Pendulum

  2. Misha says:

    Holy cannoli…we’ve been reading Janet Malcolm and talking about this same stuff in my class. I’ve gotta crawl out of my spider hole more often! Thanks for the synthesis!

    • Seth Mnookin says:

      Are you doing Journalist and the Murderer? I was just debating whether to add that to the reading list for this semester…

  3. Blake Stacey says:

    I really don’t get where Sumner et al. are coming from. I mean, flawed papers can get through peer review — all it takes is for two or three people to f!#k up. And as for this:

    The second reason to distinguish between scientists and, for example, politicians, is that there are no organised parties, and so no “party lines”. Scientists are a loose collection of individual thinkers, as difficult to herd as cats.

    To paraphrase Barney Frank, on what planet have they been doing their science? Even if we don’t carry around cards stamped with party membership, we sort ourselves by temperament and inclination. Communities can overlap, but that doesn’t mean they coincide completely. We can get sucked into petty turf wars which consume time and generate nothing but bitterness and enmity. I’ve been on the periphery of an ongoing argument which is basically Team Edward vs. Team Jacob, and even being CCed on the e-mails is dispiriting.

  4. Pingback: “To inform and surprise” | genomeboy

  5. Pingback: Scientist copy-checking: Point-counterpoint at the Guardian | Take As Directed

  6. In total agreement. That is the most moronic argument that could be made! Good lord. Just check out Ivan Oransky’s Retraction Watch for more proof. Or take a look at how the whole XMRV/CFS story has unfolded…

    http://retractionwatch.wordpress.com/

  7. Jon Brock says:

    I’ve been following this discussion with great interest. As a scientist rather than a journalist, it’s a real education. While I can’t comment on the whole issue of copy checking, I did write this response on the limits of peer review, coincidentally with (non-vax-related) examples from the autism literature.

    http://crackingtheenigma.blogspot.com/2011/10/on-peer-review.html

  8. Tracy Staton says:

    As a former magazine editor, I’m a little flummoxed by part of this debate. Higher-quality magazines have fact-checkers that routinely check quotes and facts, and no one thinks this is a problem. *Not* checking them would be. As an ex-daily newspaper reporter/editor, I understand a bit more; newsroom types look askance at any whiff of undue influence from a source, and rightly so.

    Maybe the difference lies in the once-removed nature of a fact-checker’s follow-up. It’s not the writer herself or himself looking for feedback, but someone who has no vested interest in impressing the source. The argument over changes then happens in the office, between editorial staffers, and not between a writer and his/her source. The only problem in this scenario is that, while fact-checkers aren’t too close to the material, they may not be close enough. That is, the writer always understands more about the subject than the fact-checker does.

    Also, fact-checking is unrealistic for daily news online or in print. Bare-bones staff as it is, after so many layoffs.

  9. Pingback: Readbacks and Researchers | Alan Dove, Ph.D.

  10. Al Dove says:

    It’s been a fascinating conversation to be a part of. I have to admit that when I got involved after reading Ananyo Bhattacharya’s piece, I had no idea of the amount of discussion that had already taken place, and had not read David Kroll’s original piece. I’ve learned a lot since then about how science journalists view their role and have reconsidered my position on their allowing scientists to COPY check (I still think fact checking is a good idea and I think most would agree). What I DO maintain is that journalists spending too much time trying to challenge science is largely a waste of time as junk science gets left by the wayside anyway. The recent Kraken story from the GSA meeting that Brian Switek has written about is a good example. That work wasn’t published yet and was clearly a pretty dubious bit of sensationalism, so the media stories that arose definitely qualify as churnalism. I don’t think any amount of journalistic criticism would have made it a better story though; it’s just a far-out and unsupported idea that would have been left by the roadside of science like so many others. It should never have been any kind of media article at all. And all the while that we’re talking about it, we’re NOT talking about any number of other worthwhile scientific stories and it’s that opportunity cost that bothers me. Ananyo, Ed Yong, Martin Robbins and others have made it abundantly clear to me that they don’t think communicating science to the public in that way is what science journalism is about though, so I guess its up to science communicators (however they are defined) and scientists themselves to get the message out.

  11. Pingback: October 12, 2011 – Science Policy Around the Web « Science Policy For All

  12. Kathryn says:

    Something else the Guardian article missed: It is all too easy for a science journalist to inadvertently misinterpret what the original source said. This was covered very well at David Kroll’s blog, I thought.

    I understand that the press can’t give politicians a chance to retract something they wish they hadn’t said. But there’s a big difference between letting a political source convince a reporter “No, he didn’t compare Obama to Hitler” when it’s right there on video, and letting a scientist correct some misunderstood scientific jargon.

    I was interviewed for a blog and made the mistake of discussing two separate research projects involving different types of worms–my own thesis research in genetically-altered glowing green nematodes, Caenorhabditis elegans, and my labmates’ research using non-glowing annelids, Lumbriculus variegatus. Somehow those ended up merged into one project in the blog post.

    Worms are worms, right? Well, not really. They’re in different phyla, and we use them for completely different experiments to take advantage of various characteristics. The resulting mishmash didn’t make much sense, though her enthusiasm was flattering.

  13. David Kroll says:

    Thanks, Seth, for a beautiful summary of the discussion to date. Fabulous to hear that Trine Skyped in to your class yesterday – lucky students!

    I’m equally confused by the scientists’ argument that peer-review somehow places scientists above other sources. That’s not only an arrogant stance but one that is highly flawed. In fact, science journalists have been at the forefront of revealing and/or popularizing fraudulent work or published papers that are highly at odds with the entire body of scientific data.

    As a scientist who has been interviewed about a hundred times, I’ve only been offered to review my quotes once. I don’t expect it because I respect the concept of journalistic objectivity. Hence, I prepare carefully for interviews. And if I disagree with how I am attributed in the resulting work, I have my blog to elaborate and clarify – yet another reason why scientists should have an academic blog.

    I wish I could be taking your class – thanks so much for discussing this important issue to both scientists and science writers!

  14. Pingback: What Communication Pros Want From Fact-Checking Journos | Common Sense

  15. The Paris Review goes backwards and forwards with its interview subjects so that they can clarify what they mean; this isn’t a bad system in light of the research on error in science reporting identified by Tankard and Ryan in the 1970s. They showed that journalists made substantially more error in science reporting than any other beat – only 8.8 percent of stories were error free. Many scientists have been so burned by misquotation that they won’t even speak without a review of their quotes. I’ve found numerous cases supporting their contention that their research is distorted by journalists grinding an ax, be it political or merely dramatic, even at news organizations considered to be at the top of the journalistic quality league table. If the goal is accuracy, there is no real philosophical justification for not letting scientists preview the accuracy of their research as they reported it. Whether their research is true is another matter – and this is a crucial distinction. In this regard, I usually send a study to my mathematician colleague at STATS.org to read through its statistical data. Sometimes the weakness of the research is appallingly obvious; sometimes not so. Usually when we identify a problem and ask the researcher about it, they don’t want to talk; so that clears that up. The thing is that scrupulous scientists tend to be scrupulous about identifying the weaknesses in their methodology and writing up their results. When you contextualize new findings through the rigor of their experimental design in light of the weight of evidence, you can make a reasonable call about trustworthiness. The result of collaboration in this regard is still largely superior to the result of a journalist protecting the integrity of the reporting process (sic) by excluding the scientist from the fact check process.

  16. How says:

    As a non-scientist non-journalist, merely someone with an interest in science, I find this subject and the commentary on this page absolutely fascinating. Thanks to you all!

  17. Matt Carey says:

    Seth,

    If I can go off-topic here: for many of us, terms like “idiot”, “moron”, “imbecile”, etc. are insulting on a very different level. These are former clinical terms for various levels of intellectual disability.

    When a person with the gift of intelligence choses to not use those gifts, it is a shame. It is, however, nothing like when a person doesn’t have the gifts to begin with.

    • Kathryn says:

      Matt,

      Thank you for saying what I should’ve said, and more elegantly at that.

      Yesterday someone (at a public science lecture) called some aspect of city politics “retarded” and I did call him out. But I was gobsmacked by his response: “We can’t call people that any more, so it is OK to use it as a term to describe things that are deliberately made to work badly.” So does he think people with intellectual disabilities are mean-spirited and obstructionist? A friend who job-coaches people with IDs enjoys working with this population because they’re so much nicer, apolitical, and common-sense than her “normal” cow-orkers.

  18. Pingback: Introducing SciWriteLabs. Today’s installment: Kroll and Racaniello discuss the journalism/factchecking debate | The Panic Virus

  19. tecnik says:

    As a scientist who has been interviewed about a hundred times, I’ve only been offered to review my quotes once. I don’t expect it because I respect the concept of journalistic objectivity. Hence, I prepare carefully for interviews.

  20. Pingback: Science reporting: Covering the environment, technology and medicine Journalist's Resource: Research for Reporting, from Harvard Shorenstein Center

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>