Moving article-level metrics forward

In September PLoS started to show usage data (downloads, citations, but also use of social bookmarking services and blog posts) for all their published papers (article-level metrics at PLoS – addition of usage data). PLoS is not the first publisher to do that, but certainly the largest to date. Two Nature Network bloggers wrote about these changes back in June (The Scientist: On article-level metrics and other animals) and August (Gobbledygook: PLoS One: Interview with Peter Binfield), and a number of blogs commented on this new feature, including:

There are a number of reasons why article-level metrics are a good idea, and I hope that many other journal publishers will follow. But in this blog post I want to talk about some of the shortcomings of the current implementation of article-level metrics.

Article-level metrics should be combined from different places
Fulltext articles live in more than one place. Obviously at the journal publisher's website, but in many cases also in one or more institutional repositories and at PubMed Central (or similar places for papers not published in the life sciences). Which of these places produces the most reliable article-level metrics or should the HTML views, PDF downloads, etc. from all these places be combined? The decentralized nature of institutional repositories makes it especially difficult to combine usage statistics from them, but there are projects that try to tackle this problem. A unique identifier is required to combine the usage data from these different sources, and we have the DOI for that. PubMed Central and similar large repositories could not only start to provide their own usage data, but also combine them with the usage data from those journal publishers that already provide them.

Article-level metrics need author identifiers
Evaluating the “impact” of a researcher is one obvious use for article-level metrics. In order to be able to do that for more than a handful of researchers, we need unique author identifiers. This year we have had many dicussions about author identifiers (including this blog and at the Science Online London Conference), and I hope that in 2010 we will finally see an evolving standard that is picked up by journal publishers. It would be in the interest of PLoS to combine their article-level metrics with an author identifier as soon as possible, most likely the proposed CrossRef ContributorID, rather than the Elsevier Scopus Author Identifier or the Thomson Reuters Researcher ID.

Article-level metrics should enhance literature searches
We all know how Google became the most popular search engine (Pagerank). And article usage data would be a tremendous boost for scientific literature databases such as PubMed. A literature search should sort the results by usage data (e.g. a combination of number of citations, HTML views and PDF downloads) rand not the rather boring publication date, author or journal name. Normally I would think that Google Scholar would be the first place to implement such a functionality, but I haven't seen much innovation from Google Scholar lately.

Article-level metrics should not only be numbers
As we don't want to reduce a paper to simple numbers, it is important to provide more than HTML views and PDF download counts. Citations counts are useful numbers, but linking to the citing papers is even more interesting. Similarly we want to see links to Faculty of 1000 recommendations and blog posts aggregated at ResearchBlogging.org. If we extend this further, we should probably start to think about a better name for article-level metrics. And I hope we never start to call this ALM.

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Snippets and tagged . Bookmark the permalink.

6 Responses to Moving article-level metrics forward

  1. Richard P. Grant says:

    Interesting post Martin, thanks. As you know, we’re very interested in this.
    /understatement

  2. Maxine Clarke says:

    I agree that article-level metrics are great in many ways, but how do article-level metrics cope with all the “gaming”, web-scraping robots etc, that can or do go on? And how do they cope with site licenses? There seem to be some quite problematic technical and behavioural issues facing their uptake as an objective number.

  3. Martin Fenner says:

    Maxine, these are of course valid points. I would suggest that we shouldn’t overestimate the value of any metric. PLoS uses several sources for citation counts. Citations are less prone to manipulation, but there are already differences in the numbers. For the same reason I like to expand qualitative information like F1000 recommendations and blog posts.

  4. Zen Faulkes says:

    I blogged a bit on article measures “here”:http://neurodojo.blogspot.com/2009/11/rating-journals-and-articles.html. In addition to the questions here, I would add, “What do we want to know about an article, and why?”
    In many cases, the “Why?” is money, and that can deeply affect what information might be measured.

  5. Martin Fenner says:

    Zen, thanks to the link to your blog post. I would also add the _PLoS Biology_ paper by Cameron Neylon and Shirley Wu about article-level metrics that was published this Tuesday: “doi:10.1371/journal.pbio.1000242″:http://dx.doi.org/10.1371/journal.pbio.1000242.
    To try to answer your question: I think we want to know two things about a paper: a) is it an interesting paper I should read (the filtering function that places like PubMed do poorly) and b) I want to evaluate the research output of a particular person for a grant or job application.
    For b) there are many good reasons to be very careful with using any metrics instead of actually reading the papers of that particular person and making an informed decision yourself. Metrics would be helpful in a more general way, e.g. using four categories (top 25%, top 50%, etc.).

  6. Zen Faulkes says:

    Personally, a good title and abstract is infinitely more helpful in telling me if a paper is interesting enough to read than an article metric. ;) That said, I’m in a relatively low-key research field, and maybe people in faster moving fields with higher publication volumes might appreciate the tools.
    Which leaves us with metrics being a way to decide who gets money.