CrowdoMeter – or trying to understand tweets about journal papers

Last Friday Gunther Eysenbach published a very interesting paper in the Journal of Medical Internet Research (JMIR):

Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact

Gunther Eysenbach analyzed a total of 4208 tweets citing 286 distinct JMIR articles. 60% of the tweets were sent the day the paper was published, or the day after:

Figure 3. Tweetation dynamics. From Eysenbach 2011, distributed under the terms of the Creative Commons Attribution 2.0 License.

There was a correlation between the number of tweets about a JMIR paper, and the number of citations in Google Scholar or Scopus (analyzed 17-29 months later).  Highly tweeted papers were more likely to become highly cited, but the numbers were to small for any firm conclusions (12 out of 286 papers were highly tweeted).

This is a great study because it shows empirically what many of us felt already: Twitter is one of the fastest tools to discover newly published scholarly papers, and the number of tweets is an important measure of scholarly impact. This is an important paper for the altmetrics movement, even though Gunther Eysenbach in the paper says that he doesn’t like the term. In the paper he coins the term tweetations for tweets citing a paper – I personally prefer the term citation for all content linking to and discussing a scholarly work. Not surprisingly the paper has been tweeted more than 250 times in the first few days after publication, and will certainly become highly cited.

Are altmetrics only about counting citations?

A few weeks ago Euan Adie from altmetric.com suggested to me that we do a little research project for the altmetrics session we moderate together at the ScienceOnline2012 conference in January. We both felt that altmetrics could be improved by taking a closer look at the content of social media discussing scholarly works, and that using the Citation Typing Ontology (CiTO) would be a good start. CiTO is an ontology for the characterization of citations, and we agreed that crowdsourcing would be the only way we could do this with a large enough number of social media contributions, and with only a few weeks left before ScienceOnline2012. The project launched today at http://crowdometer.org.

To get this project started in time we had to make two other simplifications: a) we use only a subset of 7 CiTO relationships (see below, cito:usesConclusionsFrom is new in CiTO 2.1, and was added for us by David Shotton), and b) we focus on short content (Twitter, Facebook Walls, Google+), as it would be much more difficult to crowdsource the semantic meaning of a blog post and similar longer content.

The social media data for this project were all collected by Euan Adie at his altmetric.com site, and consist of 500 tweets, Facebook Wall posts and Google+ comments about scholarly papers from the month of October. In the end we used only tweets, but they made up more than 90% of the posts anyway.

Users can classify these tweets in two ways, either by browsing or searching the list of nearly 500 tweets, or by getting presented random tweets (hint: search for “I’m feeling lucky” in CrowdoMeter). You have to register via your Twitter account to classify tweets.

CrowdoMeter is doing a basic analysis of the crowdsourcing effort in real-time, allowing everyone to follow along:

We are still learning how to attract as many users as possible to help with this effort, but a long list of tweets about hopefully interesting papers, an uncluttered user interface, the real-time analysis and a high score list seem to be good start. We hope to have enough classifications together when we discuss this and other altmetrics projects at ScienceOnline2012 next month.

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Conferences, Interviews, Presentations, Recipes, ResearchBlogging, Reviews, Snippets, Thoughts and tagged , , , . Bookmark the permalink.

10 Responses to CrowdoMeter – or trying to understand tweets about journal papers

  1. tjvision says:

    Cool idea. The issue I encountered when I tried it out, and one reason I didn’t continue past two (thanks for broadcasting my feeble contribution, btw) was that the CiTO options available didn’t seem to capture the most common context. That is, someone just reporting the existence of an article without further comment. Presumably the article is of interest to them for some reason, but since they only report the title with a link, I can only guess why, and wouldn’t presume to interpret that as agreement or disagreement.

  2. Great idea! According to an announcement from Twitter’s developer conference Chirp in 2010, it should be possible (but perhaps is not yet possible, I don’t know?) to add arbitrary metadata to any tweet. What about building a twitter web client (or a browser extension for twitter.com) that let twitterers add CiTO metadata as they tweet?

  3. Pingback: CrowdoMeter goes Mobile | Gobbledygook

  4. Pingback: Facebook and Twitter as Infrastructure for Dissemination of Research Papers (and More) « UK Web Focus

  5. @littlesimon says:

    I assume you have a way to identify other factors that are associated with higher citations, such as:

    *Journal sends round link to paper in weekly e-roundup (or other direct marketing)
    *Press release about paper
    *Subsequent media coverage
    *Breadth/impact of subject area

  6. Pingback: The Personalized Journal | Gobbledygook

  7. Pingback: Crowdsourcing the analysis of scholarly tweets | Gobbledygook

  8. Pingback: Crowdsourcing the Analysis and Impact of Scholarly Tweets « BioMed 2.0

  9. Pingback: Linktipps der Woche 8/2012: Fangewinnung für Facebook-Page, Kooperation süddeutscher Archive und Grenzen des Wissenschaftsmarketings | Wissenschaft und neue Medien

  10. Pingback: Why I still like FriendFeed, why Twitter is important and other thoughts about Altmetrics | Gobbledygook