What is the scientific paper? 3: The metric

This is a guest post by Joe Dunckley
Continuing the series exploring the question "what is the scientific paper?", reposted from my old blog, and originally written following Science Online 2009. The topic of this post was originally discussed on FriendFeed, here.
On my recent post, what is wrong with the scientific paper?, Steve Hitchcock said that the most important problem with the paper is access, and that when we solve the problem with access, everything else will follow. I agree that access is hugely important, I recognise that we haven't won everyone over yet, and I know we do have to continue working away at the access problem, so I will devote a future post to reviewing that topic. But having thought about it a little longer, I am more convinced than ever that it is not access that is the big problem which is holding back the paper and journal, and open access is not the solution from which all others follow and fall into place.
There is one big problem, a single great big problem from which all others follow. The great ultimate cause is not, as I said last week, the journal. It is more basic than that. It is the impact factor. The journal is the problem with disseminating science, but the reason it has become the problem, the reason people let the problem continue is the impact factor. The impact factor is a greater problem than the access problem, because the former stands in the way of solving the latter. The impact factor is a great big competition killer; by far the greatest barrier to innovation and development in the dissemination of science.
Scientists can look at all of the problems with disseminating science, and they can look at us proposing all of these creative and extravagant solutions. They might agree entirely with our assessment of the state of the scientific paper and of the journal, and they can get as excited as us at the possibilities the flow from new technologies. But blogs and wikis are mere hobbies, to be abandoned when real work starts piling up; databases a dull chore, hoops to jump through when preparing a paper. So long as academics can get credit for little else besides publishing in a journal — a journal with an impact factor — any solution to publishing science outside of the journal will never be anything more than a gimmick, a hobby that takes precious time away from career development.
In a worse position than blogs and wikis, where cheap easy products are openly available, are the wonderful but complicated ideas that would benefit from financial backing to implement — the databases, and open lab notebooks, and the like — but which are currently artificially rendered unviable because no scientist could ever afford to waste time and money on a product that isn't a journal with an impact factor. No scientist can try something new; no business can offer anything new. Even such an obviously good idea and such a tame and simple advance as open access to the scientific paper has taken over a decade to get as far as it has in part because it takes so long for start-up publishers with a novel business model to develop a portfolio of new journals with attractive impact factors.
I am not a research scientist. I don't have to play the publish-or-perish game. So I have no personal grudge; no career destroyed or grant lost by rejection from a top-tier journal. It doesn't bother me how much agony, absurdity, and arbitrary hoop-jumping research scientists have to go through in their assessments and applications. But it bothers me greatly that, by putting such weight on the publication record — not actual quantity and quality of science done, but a specific proprietary measure of the average impact of the journals (and journals alone) that it's published in — public institutions across the world are distorting markets, propping up big established publishers, and destroying innovation in the dissemination of science. End the malignant metric and everything else will follow.

The Scientist has an attack of CNS disease

The Scientist this week tells us that

"Peer review isn’t perfect [who knew?]— meet 5 high-impact papers that should have ended up in bigger journals."

Wait, what? These high-impact papers got those citations despite ending up in "second tier" journals, so I doubt the authors have been crying into their beer about this "injustice". This is an example of CNS Disease, a term coined by Harold Varmus to characterise the obsession with Cell, Nature and Science. Not all high-impact papers must published in one of these journals, and not all papers published in these journals will be high impact. Biomedical publishing is not just a game in which editors sort articles by predicted future impact - at least, I hope it's not.

Authors chose their publication venue for all sorts of reasons, and it's hard to predict which new work will set the world on fire. Take BLAST - it was a "quick and dirty" algorithm that gave similar results to the Smith and Waterman algorithm only much faster, and the gain in speed came at a loss of accuracy. Only use by scientists in practice could decide whether this was a good approach. Focussing on the umpteen thousand citations to BLAST is missing the point: the important thing about BLAST is the millions or billions of hours of computer time saved by using it. As Joe, the other denizen of Journalology Towers, said recently:

"Lord protect us from the idea that an academic publication might have any value beyond its ability to accumulate citations."

Article-level metrics

This is a guest post by Joe Dunckley
Guys, are you sure you've thought this through? I mean, they're nice. They're fun. Data is fun. Seeing that somebody somewhere has read something you've written is satisfying and reassuring. It's good to know that you've sparked a conversation, and gotten people recommending you to their friends. But you think that it can't possibly go wrong? You think we should roll it out as the universal metric right now, and sort out the details later?

The impact factor is just data. It's nice for a publisher to know that people are reading the papers that they publish, and that all their hard work is having some effect. I can't believe that it would ever have been intended that such an absurd situation as the current practice of making and breaking careers according to a journal citation index should have arisen. Give out article-level metrics and they're soon going to stop being a bit of fun. People are going to use them and abuse them. It's what people do with data.

If you're going to reduce somebody's life's work to a number, it would certainly be less absurd to pick a number that is in some way relevant to that work, rather than relevant to the work that a whole bunch of other people did several years earlier. But only a little bit. What are article-level metrics representing? The quality of the work, or the controversy you've stirred up? The web-savviness of the field? The number of friends you have? There is already huge variation between the kinds of impact factors that medical journals get compared to, say, the sort that ecology journals get. What if, at the article level, breast cancer turns out to be inherently more comment-worthy than bowel cancer? If tenure and funding committees are willing to use something as absurd as an impact factor when making a decision, do you think that they're going to give a damn about the inherent variation in readership between fields? All those bowel cancer researchers better start reading up on their breast cancers now.

But what happens when article-level metrics really start to mean something? Give a large enough number of people an incentive to cheat and some of them are going to cheat. Remember when the World Journal of Gastroenterology boosted its impact factor with a little citation loading? How are you going to stop academics from doing what flickr users do to get themselves into the site's front-page "Explore" section for the day's best photos -- posting their stuff to "I'll leave an inane comment on yours if you leave an inane comment on mine" groups? What happens when academics spend ever increasing hours marketing their work to each other? How long is it going to be before journals and universities are competing for researchers by advertising how good their average article-level metrics are? Before journals and universities open departments dedicated to pressuring people into reading and commenting and blogging their articles?

What happens when pharmaceutical companies get in on the act? What about the ideas that get ignored for ten years, before it becomes apparent how important they are -- do the metrics count for the original paper, or for the review article that reignites the interest? What about the assholes, the trolls, the groupthink...?

Article-level metrics are a bit of fun. It's possible for them to remain a bit of fun. But it's going to take a lot of forethought and vigilance to make sure that is so.

CNS disease, or [ney-cher-sahy-uhns-uhnd-sel]

I've read some comments to the effect that PLoS ONE is a new competitor to Nature. You know who you are. The confusion appears to be wrought by the fact that PLoS ONE is a general science journal, but in reality it is poles apart from Nature, Science or Cell. If Nature is aiming to be at the tip of the publications pyramid, PLoS ONE is the broad base, much as the BMC series is also part of the broad base. And that's a Good Thing.

Harold Varmus has complained about ‘CNS disease’, the tendency to regard publication in these journals too highly. These three journals are mentioned in one breath so often that perhaps a new word could be coined: naturescienceandcell /ˈneɪtʃər/ Pronunciation Key - Show Spelled Pronunciation[ney-cher-sahy-uhns-uhnd-sel] -noun: 1. General science journals that cause researchers to temporarily lose their sanity.

Jan Velterop estimates that 1 million scholarly articles are published each year, and I read somewhere this week that there were around 680,000 abstracts added to PubMed in 2005, so that estimate looks reasonable. A quick look at PubMed tells me that the hallowed trio of Nature, Science and Cell between them published in the ballpark of 3,000 research articles that year. As only around 0.5% of publications appear in these journals, the attention paid to them is a little bit unwarranted. I'd wager that at least some of the other 99.5% of articles have some merit.

In a similar vein, Doug Altman has pointed out that although randomized controlled trials in general medical journals get such attention paid to them, 93% of trials are not published in general medical journals and 90% of medical publications are not trials. The focus on these "big headline" RCTs that make up 0.7% of the medical literature appears to be due to reprints being bought by pharmaceutical companies -- the Vioxx article by Merck brought in $700,000 to the NEJM -- and due to what Ben Goldacre calls Humanities Graduates In The Media hyping medical stories in the press.

It is worth remembering that although these journals have high impact factors, the impact factor doesn't determine the number of citations an article published in a journal will receive: the causation is the other way around. To their credit, Nature have been honest about the fact that their 2004 impact factor mainly (89% of it) derived from 25% of their articles, including the mouse genome paper that has been cited over 1,000 times. Not all articles published in Nature receive that kind of response, yet people still refer to Nature publications in awed tones. Some people might "read" Nature each week, but, seriously, does anyone actually read the research articles if they're not in the field?

Science depends more on a slow and steady accumulation of knowledge than upon "breakthrough" papers. Geoff Watts has argued in the BMJ that we should "pension off the major breakthrough". I'd echo this, and I'd agree with Harold Varmus: we need a cure for CNS disease.

Does peer review work?

There are now a reasonable numbers of studies from journals such as the BMJ and JAMA on the factors affecting peer review. For example, we know due to a piece of work done by my colleagues that while author-suggested reviewers appear to return reports of equal quality to editor-suggested reviewers, they are significantly kinder to the authors in their recommendations on publication.

One of those authors, Pritt Tamber, regularly makes clear his belief that peer review doesn't work, most recently arguing in a BMJ Rapid Response that "Much of the research conducted at the BMJ [...] showed that there is little or no objective value to the process, yet journals and their editors persist with—and advocate—peer review; their only defence is that "there's nothing better," even though few have tried to find an alternative".

As Pritt notes, one alternative is the system used by Biology Direct, published by BioMed Central. The idea is that authors obtain reviews from three members of the reviewing board. If the author cannot find three members of the board to agree (or to themselves solicit an external review) the manuscript is considered to be rejected. If they can get three reports, then the manuscript will be published, no matter what the reviewers say. The twist is that the comments of the reviewers will be included at the end of the manuscript, as an integral part of the manuscript, and signed by the reviewers. The author can make revisions to the manuscript if they wish or even withdraw it, but equally they can ignore the comments and publish despite them. This is with the knowledge that readers will be able to see the reviewers' dissent. Other alternatives include the community peer review being tried by Philica, PLoS ONE and Nature (Nature's experiment appears to have been unsuccessful, but that is no reason to write-off the idea). More journals, publishers and researchers need to go out on a limb to explore new and better ways to assess and critique scientific research.

Before we go too far with condemning peer review, it is worth remembering that without an evidence base, we won't be able to work out where peer review works, where it doesn't and why, and how to improve it.

Much of the research done into the effects of peer review has been, in my opinion at least, quite superficial. Reading it has really only told me what I knew already from working as an editor.

My wish-list for studies of peer review are:

1. Creating a metric of "on-topicness" that editors can use to assess how relevant a reviewer's expertise is to a piece of research or an aspect of that work. This could be by simple similarity analyses, comparing their PubMed abstracts to the abstract of the submitted manuscript, or by more complicated semantic analyses.
2. Comparing manuscripts that were accepted to those rejected to examine the predictive factors. Some of these have been done, but the analyses always strike me as simplistic. The sample size needs to be greater, and the journals chosen need to not be so highly selective - is it really that interesting to see the factors that influence publication in journals like Nature, The Lancet or NEJM? I really want to see are the factors that affect whether a study is ever published in a reputable journal.
   
3. A side-by-side comparison of published articles with the original submitted version (before any peer review in any journal). This could be done by a paid panel who would be able to spend the time to do an in depth analysis; an alternative would be to invite journal clubs at universities worldwide to analyse manuscripts in this way (a sort-of Seti@Home for journalology). Did peer review noticeably improve the work?
   
4. Examine the fate of articles rejected by journals. Several studies of this nature have been conducted, but they mainly focus only on the journal it is eventually published in and the Impact Factor of the publishing journal. Why not examine whether any changes had been made since rejection? What about whether the rejected work is cited and read? Do a panel and journal clubs agree that the work is now sound, even if it might be uninteresting?
5. Compare the ability of different editors to assess a manuscript and select appropriate reviewers under time pressure, pitted against some of the new semi-automated tools available, like etBLAST. This would be like a peer review Olympiad.
   

It is tough to design and conduct good studies to examine peer review, but editors need to make the effort, else skeptics like Pritt will have a point. Now, just as soon as I have some spare time...