No Comment?

Why is it that comment forums provided by online scholarly journals for post-publication review are so consistently underutilized, even though online forums concerning countless subjects, from culture to politics, are booming? Explanations for this deafening silence in journal comment forums are complex, and reviewed here and here.

My lab’s recent PLOS ONE paper on the bioaccumulation of the antidepressant Zoloft in brewer’s yeast cells was a departure from prevailing wisdom, and so I thought it especially comment-worthy. Determined to buck the trend of no-comments, I proactively solicited comments from other scientists most likely to have informed opinions, reasoning that a personalized email would boost some of them over the commenting threshold, and in turn generate a virtuous cycle of others jumping in.

So I emailed 166 professors worldwide with research interests aligned with mine. Response rates are depicted in the pie charts below. 49 of the 166 were past email correspondents or professional acquaintances (“prior contact”), while the remaining 117 were perfect strangers (“no prior contact”).

Half of the 44 replies were short-lived, terse and congratulatory, e.g., “This looks like an interesting study – thanks for bringing it to my attention! Congrats.” Encouraging, but not informative.

 

My entreaty was politely declined several times, e.g., “I will not be posting a comment. Sorry – I just don’t do that sort of thing.”

 

The most negative response was this remark from a full professor with whom I had no prior contact: “Most of us just let our published work speak for itself.”

 

In the end, I parlayed 22 email exchanges into eight comments, meaning an overall email-to-comment conversion rate of 4.8% (8/166). Of those eight, only two were posted pseudonymously.

This might sound like a poor response rate, but my paper’s comment breakdown was noticeably better than the average profile described in a 2009 analysis, according to which 40% of all PLOS ONE comments were author-generated, and 18% of PLOS ONE papers had at least one comment. Since 2009, however, there has been a torrent of PLOS ONE publications. The most recent statistics indicate that 90% of PLOS ONE articles have zero comments, and that an article with 14 comments (like mine) is in the top 0.001%.

I come away with two conclusions. First, the 90-9-1 online engagement rule basically gets it right. To use a brick-and-mortar analogy, imagine the last conference you attended: for every 100 people who stopped by your poster, 90 said nothing, 9 conversed for a few minutes, but only 1 person became a collaborator. It’s sobering to reflect on comparable ratios applying for online articles, where the convenient option of ongoing asynchronous group conversation exists.

Second, online academic discussion does not come easy, so if you want to cultivate discourse on your paper, be prepared for multiple rounds of gentle prodding to seal the deal. My advice to corresponding authors is simple: activate specific scientific networks, establish rapport and in a handful of cases you’ll be rewarded. Over time, norms can change. I was most encouraged by an associate professor who agreed to move our scientific discussion from emails to a comment thread: “An old dog has learned new tricks and I have posted my comments on your page. I like the idea of an open review process though. I have reviewed a paper for PLOS ONE before but I have never submitted one to them. I may try this out sometime.

About the Author: Ethan Perlstein is an evolutionary pharmacologist and Lewis-Sigler Fellow at Princeton University. He received his Ph.D. in molecular and cellular biology from Harvard University in 2006, working in the lab of Professor Stuart Schreiber.  Ethan’s lab website is http://perlsteinlab.com and he can be followed via Twitter @eperlste.

This entry was posted in Aggregators, Functionality, Topic Focus. Bookmark the permalink.

55 Responses to No Comment?

  1. Peter Byass says:

    Maybe if it’s true that “Most of us just let our published work speak for itself.” then that explains why science funding is difficult and impact outside research circles often fails to be achieved! (Sorry I’m not qualified to say anything erudite about your article!)

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  2. I have raised this very question with colleagues and at a departmental talk. One explanation is that researchers already know each other well, and are more likely to email each other about a paper. I see the value of having that out in the open, but there’s another thing in particular keeping me from commenting on PLoS ONE papers: I don’t read them ;)

    There are so many papers in PLoS ONE that it’s really hard to keep up, and studies in my area so rarely appearin PLoS ONE that I don’t try to keep up with it. Perhaps information saturation is a factor.

    Other PLoS journals are a different story, so I’ll say that most of the time if I find a paper interesting, all I can offer “this is really interesting” until a few months later when I’m writing a paper and figure out how that one is important to my own research.

    Another factor is that even if the research is interesting, people may be reluctant to publicly question the authors, for several reasons: perhaps they can’t find a nice way to do it and don’t want to piss anybody off. Some people think that the only reason to read research papers is to find flaws, and all they would have to offer would be negative comments. However at the same time they realize that they might be wrong, make themselves look dumb, etc.

    There’s a number of barriers here that just don’t operate on Youtube ;)

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks for commenting, Joel!

      I agree that a lot of excellent post publication reviews takes place off-line or online but invisible to comment threads. All I’d like to do is ask my colleagues who are willing to engage in email-based colloquy to consider moving the conversation to comment threads. I talk more about comment-thread fragmentation — and what we might be done about it — on my blog in a companion post: http://perlsteinlab.com/round-table/if-you-email-it-they-will-comment.

      I’ve heard the “don’t want to look stupid” argument a lot. That doesn’t stop people from asking questions after a public seminar, though granted most people stay mum. The solution to my mind is pretty simple: think before you speak, or this case, comment.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Addressing Joel’s comments about potential risks and pitfalls of public commenting, perhaps one easy solution would be to actually go to emails, see what happens, and IF the email exchange proves fruitful, noncombative, and also seems, in hindsight, to be of possible value to others interested in the topic, the email exchange can be cut and pasted into the public comment thread, where, hopefully, it would prompt further public participation…..
        But it requires some commitment on the part of at least one of the private emailers to the greater good of a richer public interaction…..

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  3. Michael S. says:

    I think the two most pressing reasons are far simpler:

    1. Time: Most PIs in particular don’t have a lot of “free” time to perform a task that may or may not get you in trouble (see below). Just as most people recognize how much time is required to appropriately craft a constructive peer review for a paper, a comment on someone’s paper would require the same care or else risk serious fallout.

    2. Risk/reward: There is far greater risk in a comment than reward (does reward even exist?). If you do happen to know the authors and they know you, even a carefully worded comment could strike a nerve in the wrong way, which could lead to long term consequences for your relationships. Would they then be vindictive for your grants, papers and seminars? It is certainly possible. Why take the risk? Now, I’m sure some will say that the greater scientific community suffers as a result, but science is a social construct and we operate in it. No one wants to potentially create unnecessary obstacles to their own career, so few if any comments are left.

    To me it seems those are the primary reasons people don’t/won’t leave comments.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks for the comment, Michael!

      No doubt academics have “limited bandwidth.” But I’m not asking for the world, here. I’m asking academics to use the same amount of time they’d devote to reading a colleague’s paper and them emailing him/her about it, but instead of clicking “send,” click “post.”

      Of course the incentives are not there to promote article commenting, and I acknowledge that there’s a deep cultural resistance. Again, I’m not asking for the world, just for academics to adjust to the Internet Age reality in the same way they must have readjusted a generation ago when correspondence moved from pen and paper to email.

      That said, scientific research is a risky proposition. Aren’t all scientists risk takers to some extent? If academics are too timid to comment on a colleague’s paper, then why are we entrusting them to tackle the enormous scientific challenges of our time?

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • What if posters were allowed to post anonymously, as long as they meet certain basic standards of decorum and politeness, and if they provide some actual substantive content (whether agreeing or disagreeing with the article writer)? That might encourage more participation.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  4. I think perhaps the biggest problem is that there’s no institutional value to these interactions. As a young scientist (I received my phd in 2011) I write a blog, I spend hours learning twitter bootstrap so I can write my own websites, I have an RSS stream with both 15 journals and 20 blogs that I check everyday and I try and tweet at least everyday. But what’s my ROI on these activities? Last time I checked hiring committees care about how many publications I have and that’s it. At my institution (UBC) the best predictor for getting short-listed for a job is having > 10 publications. Until there’s institutional value on being actively engaged in the community, only those who believe in community and engagement for their own sake will create that community. I would love to have online professional communication count for something, but as it stands, nobody cares if you comment on a paper except maybe you and the author. Until that changes, it’s hard to see anyway that people will regularly comment.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks for the comment, Edmund!

      When looking out over the vast academic landscape, article commenting does seem like an impossible task. But the beauty of my approach of soliciting comments is that it shrinks that vast landscape to a manageable terrain. And if everyone would email >100 colleagues the next time they published a paper, the network effects would start to kick in.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Edmund, your comment made me realize that a good place to push this idea of community participation would be to borrow a page from high school, when students were required to get a certain number of community service hours in order to graduate. If forward thinking science departments were to make it mandatory for program participants to demonstrate some degree of community participation in one or more of different ways, then it would happen.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
    • Jim Woodgett says:

      Edmund,
      While I empathize with your attitude, it is not the right way to think. You are viewing activities that are not direct “CV-able” publications as having no value. But you can say the same about email, conference questions, manuscript reviewing, etc. Most of what we do isn’t directly of value TO OUR CV, however, they each have other values such as helping to build networks, generating awareness and even changing your own thinking. If you are willing to comment, you likely have something worthy to say and others may reply, modifying your own stance – hopefully like this particular reply :)

      If we look to monetarize our academic activities, we’ll withdraw from so much of the functions and benefits of our academic environments. I am sure that’s not what you really mean and I do recognize the challenges of finding time and energy, but don’t give up yet! Publishing is in the beginnings of a new revolution and building post-publicaiton communications and debate will take time but will likely be a true value-add.

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Michael S. says:

        I disagree. “Academia”, as it is framed, has become as much about a business as one about more philosophical-scientific pursuits. More papers does lead to more grants. More grants lead to more papers. More papers AND grants lead to career advancement. People who diverge from this path can be severely penalized, and as such, a very narrow approach to academic pursuits is preferable to one that allows for such frivolous things. I doubt that even in a purely open-access future, devoid of physical journals, that you will see much in the way of commentary. People don’t have the time, or the willingness to stick their necks out to risk moving out of the 8% pay line because they made new enemies.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
      • I agree with you Jim. I feel I’ve reaped plenty of intangible benefits from twitter and blogging. I’ve been able to maintain a network of colleagues, and feel connected to the scientific community at large. Not just that, I believe intrinsically in the “new vision” of open science. Where we all blog and comment about research, publish our projects as they happen on Figshare or Github, and publish in OA journals like PeerJ or PlosOne and my CV get’s measured by citation count and things like Altmetrics. I personally believe in serving the community of science, it’s why I review every time I’m asked to, and work to build up a virtual network of likeminded scientists. BUT….

        I’m cynical that will account for much beyond sating my own personal morals about the online science community. The truth is I could put in far fewer hours doing the extra curricular online things that I do and instead write and I’d be better off in terms of my career. While online activities MAY pay-off in the form of collaborations or invited seminars, etc…. turning off the blogs and twitter WILL pay off in terms of more publications, and better jobs. I’m probably just on the wrong end of the demographic wave. If I get a faculty job, I’ll value things like blogs, commenting, twitter, etc…but as it stands my senior peers have told me it’s a waste of time.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
        • Edmund, I’m right there fighting the good fight with you! At some point there will be turnover in departments and junior faculty will have experienced the added value of social media firsthand as trainees. Question is how long do we have to wait, and is our cohort languishing in the middle of the transition?

          VN:F [1.9.22_1171]
          Rating: 0 (from 0 votes)
        • Bala Iyengar says:

          Here is an idea. May be PLoS/OA-journals can hand out “points” that will lead towards a subsidised submission fee (for the next paper) to labs that posts comments? Surely, a few top insights/comments can be harvested from any reasonable Journal Club that discussed an OA paper.

          VA:F [1.9.22_1171]
          Rating: 0 (from 0 votes)
      • Abdallah Al-Hakim says:

        It is great to run into you on this interesting discussion. I hope you are right about future of commenting on publications. The 90-9-1 is true across the spectrum for all types of articles not just the scientific ones. The one extra hurdle that science might have is the paranoia that many PI have about being scooped and they unfortunately distill into their students and post-docs minds. This leads to widely practiced behaviour in the scientific circles of staying in stealth mode to the point of ridiculousness. I think it all stems from the way scientific institutes and funding agencies value and rank scientist. Until the system changes to start measuring a scientist work by more than just his/her publication, I fear that science will stay a very closed ecosystem.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  5. Graham Steel says:

    Great post, Ethan.

    Whilst I can’t speak on behalf of PLoS, I would say the following.

    I am aware of the 2009 analysis as my name is mentioned and I’ve seen it before.

    PLoS would like to see more commenting directly on Manuscripts but as you describe in this post, it never took off. This is disappointing.

    Altmetrics is a newer approach and something that they are very much engaged with which I find more encouraging. I do note also that you touch upon this on your Lab Blog about this post.

    I’m sure PLoS are now aware of your posts on this subject and most certainly will take note and act upon this great “food for thought”.

    Thanks, Ethan. PLoS love feedback this this.

    Graham

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  6. Graham Steel says:

    Oops – error on last sentence which should read

    “Thanks, Ethan. PLoS love feedback like this”.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  7. Jason Rihel says:

    Hi Ethan– I think a lot of us look at places where on-line comments are booming and don’t see much intellectual value there. Even in places where there are sometimes great comments, e.g. the New York Times (to name just one), those comments have to be culled by a willing staff both for the best and for representative ones. Without that kind of editorial selection, the comments section becomes a sea of people speaking, but not talking. That implies to me that volume of commenting doesn’t bring quality, especially without additional work to sort it out. Maybe having few comments is actually a good thing– it might mean those comments that do exist are of higher quality. (And, given the deluge of science papers, maybe only a few are reading all those PLoS papers).

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks, Jason!

      We’re not drowning in a deluge of ill-informed comments. The problem is obviously persistent drought. By soliciting comments from colleagues in my field, I applied a pretty stringent selection, so my universe of potential commenters is relatively small by design. The number of times a PDF of an article is downloaded is, I think, the best proxy for the size of the universe of potential commenters. With PLoS papers you can see the PDF downloads (and HTML views, which are a proxy for casual readers) as part of the standard suite of article-level metrics.

      The low-hanging fruits in terms of commenting are the colleagues with whom you already correspond via email and who are amenable to relocating the convo to comment threads.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • MLR says:

        It turns out that there are lots of ways to moderate comments using the readers of those comments, once their numbers are large enough to become unruly. For instance, see an old survey from Wired:

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  8. Jim Woodgett says:

    Not sure I am on-side with self-promoting my own published work but I do think that post-publication commentary has a lot of value. I can see why you received a rather frosty set of replies. I’d argue it’s not the job of the author to wave her/his flag but that of the journal. PLoS One does have a popularity problem and the social experiment has not scaled well. That said, F1000 and other post publication initiatives are helping to cast the spotlight on studies that are not necessarily right under the streetlamp. Journals that do this successfully, will have an advantage.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Thanks for the comment, Jim!

      I see what you’re saying but then isn’t giving a poster at a conference a 3×4 glossy flag? And for that matter isn’t the whole enterprise of giving invited seminar talks an act of self-promotion, albeit solicited?

      I’m not sure why there’s a distinction for using the tools of the Internet Age, e.g., social media, to make colleagues and the research-funding public aware of the fruits of one’s lab’s labor.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Jim Woodgett says:

        (This made be a duplicate post as there was an Internet drop-out at my end).

        I don’t email everyone I know in a city I’m giving a talk in or everyone I know who’ll be at a conference I will attend. In both cases, I depend on the host or the organizers to “advertise”. I think the same is true of papers and journals. You’ve done your bit by getting through the reviewers and paying the open access fee.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
        • Why don’t you?! To think that the cream spontaneously rises to the top in the noisy Internet Age seems a bit quaint.

          Also, your laissez-faire approach actively causes harm by trapping knowledge in academic silos. The average taxpayer can’t make heads or tails of research but what about citizen scientists? Or disease foundations that don’t have journal subscriptions? Or intrepid high school biology teachers who want to expose their students to cutting edge research? And the list goes on and on…

          VN:F [1.9.22_1171]
          Rating: 0 (from 0 votes)
          • Jim Woodgett says:

            Is it the authors job to advertise/self-promote their work? I don’t think it is (although I agree we do this in various ways – usually as a barter where the audience gets to hear your new stuff in exchange for you getting to profile your work). Remarkably (and perhaps naively) I do think the cream rises and when it does so under its own “buoyancy” it is often more efficient in sorting the real cream from the fake topping. The mechanisms by which great science is popularized are varied but all have more objectivity than self-promotion. The worry in author-initiated advertising is that we’ll all turn into great spin doctors. Of course, the Internet has a way of cutting you off at the knees as well (i.e. objectivity has limits).

            I must say that fundamentally the wheat/chaff problem is due to too much stuff being published. If we stopped linking our “productivity” to numbers of papers and instead relied on quality of work, we’d have a lot less dross to swim through.

            VA:F [1.9.22_1171]
            Rating: 0 (from 0 votes)
    • Abdallah Al-Hakim says:

      Jim,
      I am really encouraged by your words and by your participation in this discussion. I completely agree with you that journals who can provide a good platform for commentary and public feedback will have an advantage. There are tremendous value to commenting that is almost immediately felt when one starts doing it. It is also a terrific way to build online communities and might even lead to some collaboration between scientist. FYI. I have been working with Engagio which is social conversations network and one of its aims is to help users track, engage and discover new conversations. I will send you the invite for it soon. It is a free tool for anyone to use

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  9. R. Head says:

    I was asked several times to leave a comment but was reluctant to do so because I disagreed with the premise that the study was built on and the conclusions that were drawn. The media coverage that accompanied the publication of this paper also made leaving negative comments more difficult than this would have otherwise been because of the wider audience reading them.

    Hence, I confess to succumbing to that general politeness that takes over when someone asks what you think about something you’re not very enthusiastic about and ‘interesting’ is as good a word as any in that sometimes awkward situation…

    The reason I am commenting now is because having had the author’s blog drawn to my attention I am concerned that there has been a breach of the ethical guidelines pertaining to studies involving human participants.

    That is the author invited responses from colleagues for the purposes of a study that he has now published in his blog but he did not inform them that they were taking part in a study.

    Hence his correspondents were not asked to give their informed consent, told of their rights to withdraw from the study at any point nor given any sort of debriefing that could explain or excuse this deception.

    It also seems that this study was not approved by an appropriate ethics committee beforehand as is also required under APA guidelines (linked here for reference) because without these safeguards in place the study would not have been granted permission to go ahead.

    What the author does with this information is up to him, but an apology sent to all his unwitting participants informing them of his deception would seem to be the minimum

    I note that comments are moderated, hence I hope that PLOS will, in the interests of maintaining its own ethical standards, take this blog down once it sees that the data reported in it has been obtained unethically.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  10. Hello,

    I worked at PLoS for three years (2007-2010) and my job was to take care of the comments at PLoS ONE (the other six PLoS journals added commenting capability a little later and had their own moderators). Perhaps I should chime in with a few diffuse thoughts of mine….

    First – the commenting has also gone from most blogs (and bloggers are noticing and complaining about it). Most of the discussion of papers, just like of blog posts or media articles, now happens on social media – Twitter, Facebook and Google Plus – all three of which have vibrant science communities. A 100-comment thread on a blog is becoming a rarity, while it used to be a norm. Capturing those discussions and linking them from the paper itself is a technological problem that, from what I can see, PLoS is trying to solve.

    Second – there is no one unitary thing called “Science”. Each discipline has its own culture. At my job interview at PLoS I was asked how to get more commenting on PLoS ONE articles. My reply, pretty brave considering that the room was full of biomedical people, was: “forget the biomed”. Biomed is the most competitive, secretive and closed of all the science communities. I suggested trying to get more publications in paleontology, ecology, evolution and animal behavior, as those communities already have a much more open culture of public commentary (they are also unscoopable, thus fearless – everyone knows who dug up which fossil when and where, or who has ten field seasons under their belt, so that cannot be replicated by someone who has more postdocs and can work faster). A month later, I met and talked to Paul Sereno about it, and he decided to publish his Nigersaurus paper in PLoS ONE. This opened up the floodgates for more paleontology papers. I eye-balled it at the time, and situation may have changed since then, and I’d love to see someone do a proper by-discipline analysis of commenting, but fossil papers tended to have decent commenting threads right from the start.

    Third – there are different commenting cultures that strongly dislike each other and are weary about the prospect of meeting each other in the comment threads. The academics tend to write long comments in a very academic style, written almost like short scientific papers, with references at the bottom. They are outwardly extremely polite, but the substance is often devastating: “Let me first congratulate my esteemed colleagues for publishing this excellent paper, but…” followed by the sound of knife twisting. I call that style “academic kabuki”. The other style is used by people who are experience on the Web, e.g,. veteran science bloggers. They are short, direct, efficient with language, brutally honest, and go straight to the point. The first group, the “kabuki” practitioners, see the “bloggers” as impolite and lacking proper reverence and decorum. The “bloggers” see the “kabuki dancers” as dishonest mealy-mouthers. Every time those two cultures met in the PLoS comments, the academics disappeared from the thread, not to come back again.

    Just my 2c.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Wow, great perspectives! (Btw, I really enjoyed your chronicle of blogging a few weeks ago on the SciAm blog).

      I know what you mean about scoop paranoia. I’ve experienced it firsthand every day of my scientific career. I take comfort in the fact that younger scientists — and a few intrepid senior faculty — are increasingly open to the idea of sharing and online scientific community more generally. Unfortunately, the god-awful academic job climate is turning away what would be a revitalizing force for change operating within academia.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  11. Martin Fenner says:

    Ethan, I have posted a comment on your blog. Or should I have posted the comment here? Or in the comments section of your article? On Twitter, Facebook, Reddit or Friendfeed? It helps a little bit that PLoS is at least linking to the tweets about your paper, but this is a big dilemma.

    As I said in the cross-posted comment on your blog, building a community is the most important step to attract repeat visitors, and at least some will eventually comment. One important ingredient could be a central place where all comments are listed, think FriendFeed.

    VN:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  12. Bill Hooker says:

    http://friendfeed.com/api/

    but Facebook owns Friendfeed and general feeling is that the latter is expected to die of neglect if it’s not simply killed off. For a while it was a thriving center of online science discussion but since the buyout, traffic there has dropped dramatically. Maybe a self-fulfilling prophecy but the end result is the same.

    There is still no good central conversation “control center” online, where one can track and take part in conversations across blogs, journal websites, Twitter, Facebook, etc etc from one interface. Friendfeed was almost there when Facebook struck, imo. Google Plus might eventually replace it but Google’s track record of abandoning projects (Wave, anyone?) and a general dislike of too many eggs in the Google basket keeps me away.

    (I’m interested to hear your response to R Head, btw.)

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Interesting. I didn’t know FriendFeed fell into the clutches of FB. Oh well.

      As for R Head, I’m not sure whether to take him/her seriously or not. Apparently, they disapproved of my lab’s PLOS ONE paper but were too fearful to comment, but this guest post was the last straw. I’m not surprised that he/she didn’t post openly; given the severity of my transgression, you would think he/she has the moral high ground.

      The bottom line is I didn’t out anyone or divulge identifying information. Besides, my intent isn’t to shame but to demonstrate the efficacy of a targeted email approach. If people are morally outraged, I’d love to see more evidence of it.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  13. Bill Hooker says:

    Spoken, I’m sorry to say, like someone who has never felt the wrath of a human subjects review board.

    I’m personally inclined to agree with you that, since all the data you have presented is aggregated/de-identified, there’s no ethical breach. But by the same token, we have IRB’s so that we don’t find ourselves making those sorts of assessments post-hoc. And they take their jobs seriously, and they don’t like to be short-circuited.

    Were you “conducting a study” that warranted IRB oversight, just because you’re a PI — or does sending a few dozen emails not rise to that level of “experimenting on humans”, whether or not you are an academic? Should you have let people know upfront that you planned to run some numbers on response rates? I really don’t know (and you may well have done so — I haven’t seen the emails you sent out).

    I do know that what you did there was interesting, and I bet you’re planning to do more things like it. If I were you, I’d have a chat with the relevant people at your school, just to find out where the boundaries really are.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  14. Jim Birch says:

    If we apply a little loose game theory to this question we’d want to look at cost, risk and payoffs of the interaction and try to optimize these, ie, make it easy to comment in both quick and considered modes, reduce the risks associated with being wrong, and improve the payoff. In particular, a system is needed to allow commenters to capture kudos from their comments. I’ve yet to see a comment cited, but this is exactly where things should be heading for a real open collaboration system. I like the strategies that are used by the likes of Reddit and Slashdot that allow comments to be up and down voted; the next step would be to have those votes retained in some way, analogous to a citation count. The full loop would score commenters and adjust their coefficients to remove the noise from the system. (There’s also a potential for gaming the system.)

    On the risk side, the process should acknowledge different levels of the comment effort ranging from a full review of a paper to quick note. It should be possible to be wrong without too much loss of face a casual comment but take a full claim of rebuttal more seriously. Maybe authors could rate their own comments.

    Building these kind of things into a system is not just a matter of getting a software design right but it’s also critical mass, culture and time.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  15. MRR says:

    I was motivated a few times to comment on a PLOS article, but was discouraged by the barriers that the system puts up.
    First, you don’t see the comments when you look at the article. You have to click on a Comment tab, to see a list of comment titles, and click on each comment to read it.
    Second, when I click “Make a new comment” I get redirected to a login page. Not sure whether I have a login for this, don’t want to look for it. Stop there. End of storty.
    Contrast this to the ease of reading and adding comments on a blog such as this one, or the ease of tweeting about a paper.
    These are small barriers, but with little incentive to comment in the first place, they are way too big.
    Since I’m an academic, I’ll give a reference to my comment:-)
    http://s3.amazonaws.com/theoatmeal-img/comics/websites_stop/4.png
    (from http://theoatmeal.com/comics/websites_stop)

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  16. Nick Morris says:

    Interesting piece, thanks.

    In my limited experience there are a number of reasons why readers don’t engage and leave comments:

    1. The difficulty of login in to leave a comment (which doesn’t seem to be a problem here – but I guess I’ll know for sure when I try to click the Post Comment button!)

    2. The risk/reward problem as stated above. Why should someone leave a comment? What is in it for them? Chances are it may lead to more risk than reward in their career.

    3. At the risk of sounding ageist, most PIs are not familiar with “social media”. I know very few PIs that use Twitter or Facebook, and even less that regularly write a blog. Therefore, the idea of leaving a comment on a blog post, or an online scientific paper, is particularly alien.

    As part of one of the courses I teach I try to introduce undergraduates to the idea of using social media such as Twitter and Facebook (and Google+ this year) in science. The students find this a bit odd as they don’t connect social media with science, and I’m convinced that some of my colleagues just think I’m “playing”.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Nick, as must be apparent from my name, I am participating in this blog not as a hard scientist, but as a personally interested lay person who happens to be closely related to the author of this blog post. But I think that is the beauty of this technology, there is a role for an interested layman to play in such discussions as well.

      Your comment, above, suggests to me that you would be a great person to ask the following question—has the topic of “science as a social process in the era of the Internet” been addressed in a coherent way in any recent article(s) or book(s)? This thread can’t be the first place where topics like the pros and cons of promoting comments on articles have been addressed.

      As I reflected on the above comments so far, an analogy came to my American mind of recent US Presidential election primaries, of which there are two basic kinds—direct secret balloting by individual voters, versus caucuses in which there is vigorous very public public debate leading to some sort of consensus decision.

      I would like to see any serious arguments trying to justifying the longstanding status quo of science article publishing being solely of the first variety, when the Internet has made the latter process so readily achievable on a worldwide basis for the first time in the history of science? It seems particularly absurd, given that science has been the motor for radical societal change throughout history, and yet now, it paradoxically is lagging badly behind other areas of joint human endeavor….

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  17. I do sometimes post comments in PLOS ONE, but it’s rare. Lack of time is certainly one reason I rarely comment, but another is that I simply don’t expect that many individuals will read my comments. (I’m surprised enough as it is that anyone reads my papers!) As academic scientists, we are professional writers. Writers don’t reap benefits unless someone reads their work. While a comment on a paper is hardly a major piece of work, it is still a net loss (effort in for no reward) if no one ever bothers to read it.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  18. Roderic Page says:

    @eperlste Two thoughts.

    Firstly, “what’s in it for me?” If I read something I want to discus, I’ll write about it on my own blog. That way I can flesh out the ideas, control how it looks, and engage with readers of my blog. The blog provides me with a way to aggregate my commentary. It’s not scattered over multiple publishers’ web sites, it’s in one place. People comment on my blog, but because I use http://disqus.com they can choose their identity, and can aggregate their own comments across other sites that use this tool. Disqus also has other nice features such as being able to tweet individual comments (as a way of broadcasting the discussion).

    Secondly, I’m puzzled as to why publishers think they can/should be a locus of discussion. Why? I suspect that communities form around ideas or topics, not publishers. They may form around journals if that journal is, say, linked to a scientific society, but even then that’s not a given. It wonder whether publishers have somewhat misunderstood their role in the academic ecosystem. It’s like music companies thinking their websites should be the locus of discussions on music.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • I admire your contrarian spirit, Roderic! Though I wouldn’t go as far to say that commenting on the journal website is superfluous (yet). For one, not every academic maintains a science/research blog. Until we get to that point, the article page appears to be the natural locus for discussion, even more so with universal commenting platforms like DISQUS being embraced. But without a countermeasure for comment-thread fragmentation (for paths forward see http://engag.io, and also Sciencescape), if you post about an article on your blog how will readers outside your social network or non-DISQUSers find it? Tweeting helps but it’s an imperfect alert system.

      Now, in my self publishing utopia, the natural gathering place for scientific colloquy would be one’s own lab website, where all “published” articles would reside: http://perlsteinlab.com (sorry for the shameless plug). I’m also perfectly fine uploading a copy to the arXiv.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Roderic Page says:

        So the issue for journal publishers is finding the commentary. They thought they could solve this by having the commentary on the article web page, but that’s not working. The discussion is typically elsewhere, on blogs, Twitter, email lists, Google groups, etc. So one approach is to have tools to aggregate that external discussion, such as PLoS using Twitter searches for article DOIs to display tweets about a paper, and services like http://altmetric.com/. My sense is that it is hard to predict where communities will form, but easier to predict where they won’t, namely places constructed for the “community” by organisations that aren’t seen as part of that community.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
    • Zen Faulkes says:

      Like Roderic, I rarely commented on the PLoS — oops, PLOS — website because I have my own blog. I am more interested in writing stuff for my own site rather than a publisher’s site. Writing at my own blog gives me more ability to edit and refine my own posts.

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
    • I am with Rod on this … I have been saying for years that the whole concept of people commenting at a journal web site, seems, well, silly. People comment on their blogs, on Twitter, on Facebook, and in places where their networks are. Their networks are unlikely to ever be at journal sites.

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Jim Woodgett says:

        I can see journal comments as having two potential benefits (over blogs/Tweets, etc). Firstly, they are paper-centric so can act as a nucleator for relevant comments on that paper. This may help the authors of that paper respond to questions and comments without them scraping the web for feedback. But secondly, they may also capture other eyeballs that are specifically brought into view by the paper itself. It’s still a chicken and egg problem and it would be interesting to see whether there is any correlation between metrics like JIF and number of comments per paper.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  19. People should look at the Nature News article by Apoorva Mandavilli called “Peer review: Trial by Twitter” which discusses related issues.

    I am quoted in there

    “Who in their right mind is going to log on to the PLoS One site solely to comment on a paper?” asks Jonathan Eisen, academic editor-in-chief of PLoS Biology, and a prolific blogger and tweeter. “I guarantee that there are more comments on Twitter about a PLoS paper.”

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  20. One suggestion above was “wouldn’t it be great if everyone emailed their closest 100 colleagues every time they published a paper”… the problem here is that the people you want to notice (prolific scientists) have big networks, and are swamped by emails already. It simply becomes annoying to receive unsolicited emails from colleagues at that level. A much better approach is to select the 4-5 colleagues most interested in your paper and send them a copy. Even better, wait until one of them publishes something similar and send them a personal email congratulation them on their paper, make some intelligent comments and suggestions and then end with “oh by the way attached is my own similar paper that no doubt you have already seen”.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • That was my suggestion! I think my experience provides some decent evidence for the claim that emailing 4-5 people is not enough because of the stochasticity of small numbers. Even among academic colleagues with whom I had prior contact, half of my inquiries went unanswered.

      So you might get lucky and all 4-5 will reply, but the odds are still against you that their replies will be more than a few sentences of generic encouragement, or that one question you always get about your project.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  21. Pingback: Crowdsourcing and the value of curation – Mendeley pairs with F1000 Prime to recommend great research. | Mendeley Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>