The Value of Peer Review

This weekend I was at SciFoo, an invitation-only unconference by O'Reilly Media, Nature and Google that took place at Google. I was fortunate to be invited, and I'm still digesting all the impressions and discussions that I had (there were many). This post is the indirect result of two sessions and several related discussions on one particular topic that I'm most interested in – the process of scientific publication.

Flickr photo by Duncan Hull.

Peer review is usually seen as essential for the quality of a published paper. At the same time peer review puts a large burden of work on the research community, and in general is unpaid work. But peer review is not living up to it's full potential. We should start to see it not as a necessary, costly and time-consuming burden, but rather as a business opportunity. But for this the information communicated in the peer review process needs to leave the (digital) drawers of the journal editorial offices.

The published research paper is the most important piece of information used to evaluate the reputation of a scientist (the published book takes that role in many social sciences). This evaluation is most important when large sums of money and the personal careers of the involved scientists (in the form of research grants and jobs) depend on it. We usually begin with applying some sort of metrics to the publications of the scientist(s) under evaluation, most often the Journal Impact Factor. Once we have reduced the number of scientists and papers to a small enough number, we can start reading the fulltext papers in order to evaluate the quality of the science. Both approaches have shortcomings that I don't want to go into detail here (citation metric: artificial number, delay of several years if relying on citation counts; reading fulltext paper: time constraints, often needs outside experts as science has become so specialized). The information contained in the peer review process is a large untapped resource that could potentially overcome many of these shortcomings.

The typical revenue streams of a scientific journal are currently subscriptions and author submission fees, and to a lesser extend advertising and subscription-based added value in the form of news items and editorials. The information contained in the peer review process is extremely valuable to granting agencies and job search committees and has the potential to become an additional major revenue source for scientific journals, thus allowing a reduction in author submission fees and/or free fulltext access without a subscription. Many research organizations and funding agencies currently pay for journal subscriptions and author submission fees. If they would pay the journal publishers similar amounts of money for peer review information, they would obviously get much more value out of their money. Whereas this revenue would probably come mainly from granting agencies, large research organizations or companies would also pay for this information, as would the typical academic journal reader (obviously a much smaller sum) if it helps in filtering out the most relevant scientific papers.

The peer review process obviously would have to change in this model. If peer review becomes a major source of revenue for the journals, reviewers need to get paid for their efforts. And it will affect of what reviewers and editors write in their assessment if that information might later be seen by third parties – even if they remain anonymous. It is also not clear if the peer review information of rejected papers should be used (a large body of information at journals with high rejection rates). And if journal publishers don't buy in that model of selling peer review information, nobody stops other parties from doing additional peer review of papers already published and selling that information. Which sounds pretty much like what Faculty of 1000 is doing, although they seem to be targeting the academic journal reader rather than the much more important funding agencies.

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Thoughts and tagged . Bookmark the permalink.

32 Responses to The Value of Peer Review

  1. Maxine Clarke says:

    Peer review can be seen as a benefit of the commons – I have spoken to many scientists who see it as part of their professional role – they provide reviews of work and in return they get reviews of their work. If everyone contributes, the system works, is self-sustaining and is relatively efficient and “cheap” for everyone.
    For a journal that provides a high-quality filtering service, a large part of the effort is selecting what to send out for review. Reviewers like to review for a high-quality journal as they see interesting work. The editors do not send many papers out for review at all.
    I think some of these factors are forgotten in some of the debates that go on about peer-review. It is one of those systems that works because everyone sees it as part of the “commons” – everyone eventually benefits from the service if everyone contributes to it.
    There are an increasing number of variants on the traditional peer-review model, and most people welcome these experiments. However, this is not the same as regarding peer-review as “unpaid work”, because everyone benefits from it. (And the journals or grant agencies already spend money on pre-peer-review filtering.)

  2. Richard P. Grant says:

    *cough* “much more important funding agencies”
    Let’s substitute, say ‘Qiagen’ in that last sentence and see if it works.
    Exactly. F1000 is a service for researchers. To our minds, researchers and clinicians _are_ more important than funding agencies. They’re the people doing the actual work, after all; they’re the ones who _really_ need to know the literature.
    Having said that, we are going to be offering something in October that funding agencies will find _very_ useful, along with something else later in the year. But if I said any more, I’d have to shoot you.
    –’Dr F1000′

  3. Brian Derby says:

    Martin
    I think one of the reasons peer review works well is because it is anonymous and confidential. Despite that (maybe because of that?), it can also be spiteful. If you are a journal editor or if you sit on a grant awarding panel, you can moderate the review. That would be much more difficult if the reviews become public. I could also see a rich seam of legal work when we find that someone did not get a grant/job/promotion because of a perceived flawed review in the past.
    Anyway, t5his has reminded me that I should go an review the paper I said I would do for Phil. Mag.

  4. Martin Fenner says:

    @Maxine. I did not want to suggest that there is anything wrong with the peer review process. It just occurred to me that we should think beyond subscriptions and author submission fees as ways to generate revenue for paper production costs.
    @Richard. I was referring to who I think would be most interested in paying significant amounts of money for this kind of information.
    @Brian. The reviewer could write a separate version of his review that is intended for that other use. And the review would still be anonymous. I would not think that any decision for a grant or job would be based on a single review.

  5. Craig Rowell says:

    So where does this leave (or begin) supplemental information (see “post”:http://network.nature.com/people/scurry/blog/2009/07/07/vital-supplements)? Is there a way to actually leverage SI as a revenue stream? There was much discussion at Scifoo about SI and the importance of all data from an experiment being available – i.e. open lab books. What if we move some sections, or at least, some details of a paper to SI can we reduce the size of publications and thus make the review process a bit more manageable? (sorry this comment is a bit disjointed)

  6. Richard P. Grant says:

    I’m not sure funding agencies _do_ want to spend significant amount of money on such things. I think scientists might get _very pissed off_ if they realized that large pots of potential grant money were going to journals instead of to them.

  7. Martin Fenner says:

    Richard, this I don’t understand. What is the difference for the scientist involved if the same amount of money is spent on a peer review evaluation rather than the author submission fee (or indirectly the journal subscriptions)?

  8. Richard P. Grant says:

    I got the impression this was extra money the FAs would have to spend.
    You should also remember that many scientists are opposed to author submission fees and don’t give a rat’s arse about free access.

  9. Martin Fenner says:

    _You should also remember that many scientists are opposed to author submission fees and don’t give a rat’s arse about free access._
    Then these scientists should not be surprised if the institutional subscription to their favorite specialty journal is dropped because of an increase in subscription cost.

  10. Mike Fowler says:

    Ahhh, these peer review discussions always bring out the best in people! There’s a couple of points I’d like to ask about:
    1) _”The information contained in the peer review process is extremely valuable to granting agencies and job search committees and has the potential to become an additional major revenue source for scientific journals”_
    Funding bodies also generate lots of their own peer review in application processes – essentially the same research that will later turn up in journals. Surely this would conflict and potentially overlap with the peer review info that journals have, reducing the value to both parties?
    2) Making peer review public would undoubtedly change the way it is carried out. Referees would (hopefully) spend much more time and effort ensuring that their critical reviews are watertight, to avoid embarrassment or the legal problems mentioned if mistakes arise. If peer review becomes harder, I think fewer people will willingly do it. And if other people are making money from it, reviewers may start to ask why they should put in extra effort for someone else’s financial gain. How do you see this going, Martin?
    Richard, how many scientific authors don’t give a rat’s ass about free access? As self-important egomaniacs who don’t (directly) make money from publication, I get the feeling we all want as many people to read and cite our work as possible. To put a number on it, I know 0 (zero) scientific authors who are opposed to free access.

  11. Mike Fowler says:

    By the way, Martin, I meant to say I think it’s very valuable to have these discussions – good post!

  12. Richard P. Grant says:

    Mike, I think most if not all authors want free access… to give and to receive. But when you turn round and say right, that’ll be three thousand quid please, well it’s a different matter. I’m not entirely convinced that all people will perceive the extra money they get in their grants (because the libraries aren’t taking so much) as sufficient.
    While I am, for the record, in favour of open access as a matter principle, I do recognize that there are huge problems in implementing it. I’m fully behind the idea of ‘equal access’ for all UK researchers and en bloc negotiations with publishers.

  13. Maxine Clarke says:

    Brian (and Mike) on the “spiteful” point you make. Although clearly there are issues with the way some journals run their peer-review processes, I hold that a good-quality journal (and I am not talking about impact factors here!) can run an impartial and fair anon peer-review process. The most effective ways to do this are (1) to have an independent editor select the peer-reviewers (ie if the ed is also a card-carrying researcher, the journal has clear conflict of interest policies so the ed selects reviewers independent of him/her); (2) papers are seen by two independent peer-reviewers; (3) peer-reviewers are shown each other’s comments if there is any question in the ed’s mind of spite or other personal judgement interfering with the report – and in any event, if the ed asks for a revision, both/all peer-reviewers see all reports with the revised ms and authors’ repsponse to reviewers’ concerns.
    In my experience, this is a simple and clean system to run, as opposed to various types of “open” peer review which do open up rather ghastly lobbying of various kinds by the authors to the peer-reviewers. People who propose “open” peer review often do not think beyond the first round of review. It may be fine for a ms that is not revised before publication, but all good manuscripts, like anything else written, benefit from at least one round of revision based on independent feedback, often more. If the author knows the identity of the reviewers, the independence and neutrality of this process is compromised and it becomes tainted by personality and power issues.

  14. Martin Fenner says:

    Maxine, I would think that the criteria you define for good peer review can also be applied to the revised peer review system I propose. Reviewers could remain anonymous, and the peer review communication remains confidential between authors, editor and reviewers until a final decision has been made.
    I would also think that the information communicated to third parties would have to look differently. This could for example be a summary of the peer review written by the editor in charge of handling the manuscript once the paper is accepted. The main point is that the strengths and weaknesses of the manuscript identified in the peer review process are not lost.

  15. David Papapostolou says:

    @Martin
    _What is the difference for the scientist involved if the same amount of money is spent on a peer review evaluation rather than the author submission fee (or indirectly the journal subscriptions)?_
    i agree from the scientist’s perspective, but what about the journal’s? If you really mean that peer review can be a source of EXTRA cash for the journal, then you can’t possibly scrap out subscription fees? Or i am missing something?
    @ Maxine
    _If the author knows the identity of the reviewers, the independence and neutrality of this process is compromised and it becomes tainted by personality and power issues._
    In my experience there is more often than not a cloud of certainity around reviewers’ identity, simply because people bump into their peers all the time at conferences, or even coffee breaks, or invite them around to give talks and vice versa. This is imho what makes it worthwhile to discuss and experiment with alternative models or reviewing research. The current one relies on very strong assumptions: a sort of ideal world where service to the scientific community primes any other interest. But anyway, i am kind of sidetracking the discussion here ;)

  16. Maxine Clarke says:

    Martin – I’m now confused about what you are proposing, sorry. I don’t think it is necessary for money to change hands directly for the reason explained. In fact you may get more bias in a payment system.
    David – I am aware of numerous occasions where authors have been convinced of who did or didn’t review their paper and have been completely wrong. It happens all the time- happened to me only the other day in fact.

  17. Martin Fenner says:

    Maxine, I agree that is not necessary that money needs to change hands for the current peer review system to work. But if we change the peer review system just a little bit, it could become an interesting revenue source. Why is that good? I hope not so that the journal generates more revenue, but rather that it reduces the subscription rates and/or author submission fees.

  18. David Papapostolou says:

    @ Maxine
    It is obviously not an all black or white situation, but just the fact that it is clearly more often then not as i say is enough to signify that the current system is far from the ideal one hope it was. As the whole academic research chebang relies on peer-reviewing we can sit on or afford a _statu quo_ and i am sure you agree with me here.
    Will have a couple of comments and questions to ask people on the peer-to-peer forum soon!

  19. Irene Hames says:

    Thanks for the thoughtful post, Martin. I’m coming at this from the perspective of a professional editor, and I second many of the points made by Maxine.
    I have real concerns about peer review being considered ‘a business opportunity’ and becoming a potential additional revenue stream for journals that could, in theory, be used to reduce subscription charges and author submission fees.
    Rather than being a positive step forward it could introduce unwelcome and unhealthy elements into a process that, as Maxine points out, generally works, is self sustaining, and benefits many in a give-and-take way. There are of course problems now. Peer-review standards, and levels of knowledge about how to run a good, fair and effective system, vary enormously. But peer review as operated by many good-quality journals (and I’m also, as Maxine, not talking about Impact Factor) is a sophisticated process that is complex, flexible and understanding. It’s also a system whose success is based on trust and goodwill, sometimes built up over many years with the community of the journal, with many members acting as both authors and reviewers and so benefiting in kind rather than by direct monetary recompense.
    I just wanted to concentrate on one point – that if journals were to sell peer-review information, then reviewers would need to get paid for their efforts. How would this remuneration be worked out across journals? Would it lead to reviewers tending to review for journals that paid the most? Would those higher fees come from the larger and more general journals with high Impact Factors that could afford more? But they’d also have to charge more for their peer-review information because they’re more expensive to run and their rejection rates are higher – I’m assuming it’s only the information for accepted and published articles that would be available for sale. The ethical issues and problems inherent in using the new model for rejected manuscripts are enormous. And, as Brain Derby hints, it could be lawyers who end up making the most. Would journals have to grade returned reviews for quality and pay accordingly? Would that lead to disputes with reviewers who felt their efforts weren’t being fully appreciated, with possible consequent breakdowns in relationships?
    Many journals are finding it harder to find appropriate reviewers. What would happen to those journals that were only able to afford a small remuneration for reviewing? Would finding reviewers become even more difficult? I hear from some authors that they are being put under pressure by their institutions to submit to journals based on Impact Factor rather than where they feel their work is best placed and will be seen by the right audience. That would be a double whammy for small, niche journals.

  20. Maxine Clarke says:

    Just to pick up on the revenue point again. I still am not clear about the exact proposal. David seems to think it is that the author pays for the peer-review process. All other considerations apart (I do not think much of the idea), what of those papers that are not sent for peer-review? That’s an investment by the journal, especially one that only sends a small proportion of its submissions to reviewers. And of course it woudl incetivise the journal to send out more papers to review and annoy reviewers who’d receive even more (and less filtered) submissions.
    I’m not sure that this is exactly what Martin is proposing. Who pays who for what, in the suggestion? Running a peer-review process is enormously expensive already for any journal that does it properly, yet I fail to see how charging for it can do anything other than introduce bias and risk of bias. Paying the reviewers themselves seems superficially attractive but there are downsides to that proposal too.

  21. Martin Fenner says:

    I know that asking for changes in the peer review system is a dangerous proposition as peer review in general works well. But I want to have free access to all (or most) scientific papers in 5-10 years. As a university researcher I can get (almost) all the papers I need, but there are many people that are not in the same privileged position. More importantly, wider access to fulltext papers will spawn a number of useful applications (from text-mining to real-time usage statistics), some of which we haven’t even thought about yet. Therefore I don’t see a long-term future for subscription-based access to research papers. The author pays model works for some journals (_PLoS One_ is an example) and some areas of research, but as the only source of revenue wouldn’t sustain journals such as _Nature_, _Science_ or _PLoS Biology_.
    It is possible that we will see a mix of different revenue sources in the future, including author pays fees, subscription fees for additional content such as news and editorials (the _BMJ_ allows free access to research but charges for additional content), and advertising. But we have to think about other sources of revenue, as for many journals this is probably not enough. Journals are an important revenue source for many societies, so most of them would probably not want to financially support a journal that is losing money. It is also conceivable that journals – even those in very good standing today – that continue to use subscriptions as their only source of revenue will run into financial problems in the future, as readers and authors move to other journals.
    In this context I see a role for my proposed model of paying for peer review information. I would think that many journal readers would pay for a subscription if the peer review information is added as subscription-only content (similarly as they would pay for news and editorials). And that research organizations would pay for that if it helps them to decide where to put their research money. But there are many open questions, some of which were raised in this discussion.

  22. Martin Fenner says:

    A recent _Nature_ essay by Toby Murcott (“Science journalism: Toppling the priesthood”:http://dx.doi.org/10.1038/4591054a) wants to _allow journalists access to (anonymous) referee comments alongside a final paper_ for different reasons, i.e. to improve the quality of science journalism that now too often relies on press releases.

  23. Martin Fenner says:

    A June _PLoS One_ paper from the Wellcome Trust (“Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs”:http://dx.doi.org/10.1371/journal.pone.0005910) stressed the importance of expert review and concluded that
    _Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation._

  24. Maxine Clarke says:

    Ah, I get it now. You are suggesting that the peer-reviewers’ comments are made visible to people who pay to see them.
    Well…….not sure. I certainly believe that there is added interest for readers in posting the various iterations and views (not just of peer-reviewers, but of editors and authors’ responses) with the manuscript. But whether anyone would want to pay for this I don’t know. The organisations you mention probably value the filtering the journal has provided and the time-saving more than all the nitty-gritty. (And some people disagree completely with the principle of publishing referees’ reports and other discussion. One such is Roger Macy in his response to Toby Murcott in the NN Opinion forum, but he’s not alone in that.)
    Martin, I’m not going to respond to all the points you’ve just made about access, because that’s a whole different ballpark from the point of your post about peer-review having some monetary value. But just to note that in your paragraph about text mining, subscription publishers are making their content freely available to machine readers for semantic matching and so on. Well, some of them are.

  25. Irene Hames says:

    _A recent Nature essay by Toby Murcott (Science journalism: Toppling the priesthood) wants “to allow journalists access to (anonymous) referee comments alongside a final paper” for different reasons, i.e. to improve the quality of science journalism that now too often relies on press releases._
    As not everything in reviewers’ reports may be right there’s a danger in publishing them alone. For their true accuracy and worth/weighting to be gauged they need to be assessed in the light of other information available in the peer-review process. So, to be of most value (and to avoid inaccuracies in use and reporting), they really need to be viewed along with the insight and interpretation of the editor, for example as at “The EMBO Journal”:http://www.nature.com/emboj/index.html, which is attempting to make its editorial process more transparent. For articles submitted from January 2009 and published since June it’s providing a “Review process file”:http://www.nature.com/emboj/journal/v28/n13/extref/emboj2009155s7.pdf as part of the supplementary information. This contains the editor’s decision letter(s), reviewers’ reports and authors’ responses.

  26. Martin Fenner says:

    Irene, thanks a lot for pointing to the _EMBO Journal_ review process files, which I somehow have missed. They are good model for what I have in mind.

  27. Henry Gee says:

    I have a feeling that people are forgetting how complicated the peer-review process can be. Which reports would Martin propose are sold?
    Papers that are seen just once by referees – who would write just one report each – are those most likely to be rejected. Papers that get accepted invariably go to at least two rounds of review, during which time reviewers might have a chance to respond to specific points raised by other reviewers, the editor and the author. Quite a correspondence can build up. Sometimes, referees choose to waive anonymity and invited direct interaction with the authors.
    The job of a manuscript editor is not to oversee some simple, linear process, but to act as a chairman (or, perhaps, something like a ringmaster at a circus), co-ordinating a number of related (and sometimes conflicting) interests. It is my view that this process is best done in confidence, behind closed doors – not because we editors are trying to impose some kind of cover-up, or keep information away from people, but because that’s the best way to get the job done. I suspect that to put referees’ reports up for sale would compromise the quality of such reports and act to degrade a journal’s quality, as well as introducing yet another complication into what is already a much more multi-faceted process than people generally imagine.

  28. Richard P. Grant says:

    According to some people, the job of a manuscript editor is to make money for a journal. _sigh_

  29. Henry Gee says:

    We editors don’t stoop to that level. When we go anywhere, we have some wallet-carrying flunky follow us around (at a discreet distance).

  30. Irene Hames says:

    Martin, you might be interested in some of the ideas in “this new approach”:http://math.rejecta.org/vol1-num1/1-3 – publishing (open access, maths) papers that have been rejected by other journals, along with an open letter from the author discussing the original review process and including, amongst other things, the reasons for rejection.

  31. Mike Fowler says:

    Irene, thanks for putting up the EMBO links – it’s fascinating to see the level of detail that some Editors go into in their comments. Is this typical of EMBO, or Mol Biol journals more generally?
    I’m starting to wonder if there are large differences amongst fields. I’m an ecologist (mostly mathematical modelling of population biology), and I’d say I rarely receive such considered comments from editors. Likewise, when I’ve received editorial feedback after acting as a reviewer, I don’t always see such useful, directed comments. I certainly appreciate seeing the other reviewers’ comments, and the Ed’s if they have any. Whenever I have had such useful feedback to my own work (as the EMBO link), I’ve really appreciated it.
    Seeing how other papers have fared through the review process can break down a sense of isolation that can build up from only seeing feedback on your own work. It’s also an extremely useful learning resource to see how good reviews are carried out and good responses are presented.
    Introducing some financial scope into the review process could feedback positively to the reviewers who do the hard work. If journals did try to make money from publishing these, they can easily identify which university a reviewer/editor is associated with, and reduce their library’s subscription fees accordingly for each paper handled.

  32. Irene Hames says:

    I think, Mike, that the level of effort editors go to and the level of detail in their letters probably varies more from journal to journal, even within a journal, than it does from field to field. An editor who only ever says ‘please address all the reviewers’ comments and make the appropriate revisions’ isn’t doing his or her job. The editor’s input and guidance are crucial when reviewers’ reports are complex, divided or asking for lots of additional work. They also need to set the goalposts for acceptance so everyone knows where they stand and what chance of success they have.
    Many journals do nowadays send their reviewers notification of the outcome of the review process and the other reviewers’ comments, sometimes with the decision letter. That’s courteous as well as educative. But I hear from some reviewers that not all journals do, and they only find out by chance what the outcome was for manuscripts they’ve reviewed when they see the article published in that journal or somewhere else. There’s no excuse for journals not to send out feedback as it’s very easy to do with online submission and review systems, where it can be automated. We used to do it by hand, photocopying up letters and reports and sending them out by regular mail – that was a real effort.
    If any journal you review for doesn’t let you know the outcome or provide you with the other reviewers’ comments, write to them and ask for this information – you deserve to have it after your hard work for them. If they don’t treat you well, then you may decide you don’t want to review for them again …
    If you want to see more published reviews, then look at the “BioMedCentral medical journals”:http://www.biomedcentral.com/browse/journals/. They have a ‘Pre-publication history’ section where they mount the reviewers’ comments (many, not sure if all, operate open peer review, so the reviewers are named), authors’ responses, and also the various versions of the manuscript.