Baby steps toward better metrics

Article-Level Metrics provide new ways to look at the impact of scholarly research. Two important concepts are a) to track metrics for individual scholarly articles instead of using numbers aggregated by journal, and b) to go beyond citations and also include usage stats and altmetrics.

Article-Level Metrics is also doing something else: instead of tracking impact by year, it looks at usage, altmetrics and citations in real-time. There might have been technical reasons to do so 20 years ago, but there really is no longer any reason why scholarly impact should be tracked on a yearly basis in 2013. Unfortunately there is one big stumbling block:

The publication date of a scholarly article is often difficult or impossible to obtain. Publication year may be the only available information.

A good example is CrossRef. They provide a lot of interesting metadata about an article and make this information available in a very nice search interface. But they only require the publisher to provide the publication year, information about the publication month and day is optional. There are many other examples of journals and services that just can’t tell you when exactly an article was published. This might have made sense when periodicals were printed on paper, but doesn’t work for digital content.

 

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Snippets and tagged . Bookmark the permalink.

3 Responses to Baby steps toward better metrics

  1. I think there are some misunderstandings between disciplines and amongst non-academics about what metrics are *for* in scientific works.

    For example, my work in JACS (http://www.ncbi.nlm.nih.gov/pubmed/18031038) has been cited about 5 times. Only a few labs on the face of the planet have the laser setups to do this sort of work, and therefore we were excited we were getting cited at all. It’s a very interesting field, since ultimately there are applications for the manipulation of gene expression with light by the photoactive protein. However, it is of interest mostly to a rather narrow audience of physical chemists at the moment; not the much larger biological one.

    It is far more important *who* is citing you than how many total citations you have, and whether or not the other independent experts agree with your conclusions. They could be citing you because they are proving you to be wrong!

    This is a much more difficult “metric” to calculate, as one must actually read the article and understand the citing author’s reasons for citing your study. This is why I am liking post publication review sites like pubpeer.com (http://pubpeer.com/recent). Only corresponding authors can register to comments on an article, and the comments are screened for things like collegiality and expertise. This ensures that it is not just some “random person on the internet” leaving the comment, which can be both anonymous and non-anonymous. (I do non-anon, here’s mine: http://pubpeer.com/publications/6123ABFE4A1BB6C628618D2F5EFAB1 .)

    There is no quick and easy way to judge the merits of scientific literature; it requires time, expertise, and independent replication of the work. The general public and non-experts would do well to recall this.

  2. Martin Fenner says:

    Allison, thanks for the excellent comment. I think we are in agreement: I don’t want to collect just numbers about papers (whether it is citations or something else), but rather learn about the individual comments, blog posts, tweets, citation, etc. Ideally I would like to know when who said what, and that is why I need something like a publication date. Among other things this is important to bring the post-publication activity on a paper in the right order.

  3. Yep. Esp. your idea of who-said-what-1st; since this is how “credit” is usually allotted in scientific conversations. These are usually ideas anyone past a certain level of expertise would be able to come up with, so it’s important to know who’s the “quickest on the uptake”.

    It’s almost a measure of reading speed and comprehension than much else! And this is important; “keeping up with the news” is an essential part of the job of any good active (ie, non-emeritus) scientist.