Peer Review at the Scholarly Kitchen

The Scholarly Kitchen is a group blog started by the Society for Scholarly Publishing in 2008. The blog posts by authors Kent Anderson, Phil Davis, David Crotty, Michael Clarke, etc. are an always interesting – and often thought-provoking – read about scholarly publishing. Two recent posts looked at peer review.

The “Burden” of Peer Review
In this blog post David Crotty argues that we overestimate the amount of time the typical researcher spends doing peer review. In his informal survey most researchers review 1-3 papers per month, and those reviewing many more often do so voluntarily (e.g. because they sit on an editorial board). I haven’t yet checked whether there are any formal surveys on the workload of peer review.

Post-publication Review: Is the Dialog of Science Really a Monologue?
In August the BMJ published a paper that looked at the adequacy of author replies to electronic letters to the editor. The cohort study found that authors are reluctant to respond to criticism to their work. In a companion editorial, David Schriger and Douglas Altman write about the possible reasons for this inadequate uptake of post-publication peer review, and that we need a change in culture to value public discussion. Philip David summarized the two papers and concluded that post-publication review may continue to be spotty and unreliable.

I would disagree with Philip Davis about the conclusions that can be drawn from the cohort study. We all know that many papers receive few if any online comments. But we should rather think about where we could be 3-5 years from now, and how to get there. A recent editorial by Thomas Liesegang in the Journal of Ophtalmology is very relevant to this discussion (Peer review should continue after publication, link via EASE Journal Blog). And Richard Smith gives a wonderful and much broader definition of post-publication peer review in a response to the BMJ editorial:

I would define post-publication review as the process whereby scientists and others decide whether a piece of work matters or not. I suggest that this doesn’t happen much through debate in the correspondence pages of journals, but rather through scientists and other consumers of research recommending others to pay attention to a piece of research, conducting other studies off the back of it, absorbing it into systematic reviews, beginning to act on its conclusions, throwing it in the bin, and taking a thousand other actions that constitute the “market of ideas.”

This entry was posted in Conferences, Interviews, Presentations, Recipes, ResearchBlogging, Reviews, Snippets, Thoughts and tagged , . Bookmark the permalink.

9 Responses to Peer Review at the Scholarly Kitchen

  1. Interesting post Martin! We’ve set up a site called the Third Reviewer for just such post publication discussion on published work ( Readers can look by subject area (right now our limited offerings are neuroscience, microbiology, and immunology)- and find relevant journals, look at abstracts, link back to the full text, then return to our site and leave their comments (fully anonymous, no name or registration required) at our site. We hope that this will begin to encourage a wider discussion on the published scientific literature.

  2. Martin Fenner says:

    Helene, that was a quick reply, I just posted this minutes ago! The Third Reviewer is on my list of things to write about in more detail.

  3. Coturnix says:

    I have a feeling that a lot of people, used to online commenting elsewhere (blogs, other media), expect that all the commenting will happen within the first 24 hours, after which the paper is “off the front page” and forgotten.

    But a scientific paper is not a blog post. It takes several years for the research community to do further work, publish it, think about it, and then come back and evaluate a paper if it turned out if was important or novel (or importantly wrong), etc.

    Also, very few journal publishers have comments enabled right now. If everyone had them, if it became common-place, scientists would much more quickly get used to the idea that this is something they could and should use.

  4. Martin Fenner says:

    Bora, I agree that we have to be patient with this in several ways. And that post-publication peer review can’t be reduced to comments written on a journal webpage. Or to citation counts.

    Something that bothers me is that it is still so difficult to track all the blog posts that talk about a particular paper. This post is a recent example.

  5. Ha Ha! I’m procrastinating what I’m really supposed to be doing. Right about tracking multiple conversations… doi? Can’t that be used somehow to link all comments back somewhere? Pubmed? I’m just talkin’ . I have no idea if that is even possible.

  6. Pingback: Quick Links | A Blog Around The Clock

  7. Martin Fenner says:

    Helene, using the DOI when discussing a paper is the first step, and it frustrates me that many blogs and other websites still don’t use it. DOIs sometimes are broken right after a paper is published, and many news sites don’t provide any links to the paper they discuss, but that are different issues.

    The next step would be to aggregate all webpages that link to a particular DOI, and no single system (Google or other search engines, Nature Blogs, is doing a good enough job with this.

    Finally, that aggregated information should be made available, e.g. to journal publishers to link to the blogs or web pages that discuss a particular paper. Places like Third Reviewer or Faculty of 1000 should provide an API for that.

  8. David Crotty says:

    Bora, you make an excellent point about commenting and immediacy. Mark Cuban wrote an excellent post last year about “Participation Value and Shelf Life” that we covered on the Scholarly Kitchen:

  9. Martin Fenner says:

    David, thanks for the link. And while we are at it, here is a blog post by Shirley Wu about a comment analysis of BMJ and PLoS articles (and a related paper by her and Cameron Neylon in PLoS Biology) and here is your critique of star ratings of articles.

    But the most authoritative blog post on the subject is of course this one (background story here).