CNS disease, or [ney-cher-sahy-uhns-uhnd-sel]

I've read some comments to the effect that PLoS ONE is a new competitor to Nature. You know who you are. The confusion appears to be wrought by the fact that PLoS ONE is a general science journal, but in reality it is poles apart from Nature, Science or Cell. If Nature is aiming to be at the tip of the publications pyramid, PLoS ONE is the broad base, much as the BMC series is also part of the broad base. And that's a Good Thing.

Harold Varmus has complained about ‘CNS disease’, the tendency to regard publication in these journals too highly. These three journals are mentioned in one breath so often that perhaps a new word could be coined: naturescienceandcell /ˈneɪtʃər/ Pronunciation Key - Show Spelled Pronunciation[ney-cher-sahy-uhns-uhnd-sel] -noun: 1. General science journals that cause researchers to temporarily lose their sanity.

Jan Velterop estimates that 1 million scholarly articles are published each year, and I read somewhere this week that there were around 680,000 abstracts added to PubMed in 2005, so that estimate looks reasonable. A quick look at PubMed tells me that the hallowed trio of Nature, Science and Cell between them published in the ballpark of 3,000 research articles that year. As only around 0.5% of publications appear in these journals, the attention paid to them is a little bit unwarranted. I'd wager that at least some of the other 99.5% of articles have some merit.

In a similar vein, Doug Altman has pointed out that although randomized controlled trials in general medical journals get such attention paid to them, 93% of trials are not published in general medical journals and 90% of medical publications are not trials. The focus on these "big headline" RCTs that make up 0.7% of the medical literature appears to be due to reprints being bought by pharmaceutical companies -- the Vioxx article by Merck brought in $700,000 to the NEJM -- and due to what Ben Goldacre calls Humanities Graduates In The Media hyping medical stories in the press.

It is worth remembering that although these journals have high impact factors, the impact factor doesn't determine the number of citations an article published in a journal will receive: the causation is the other way around. To their credit, Nature have been honest about the fact that their 2004 impact factor mainly (89% of it) derived from 25% of their articles, including the mouse genome paper that has been cited over 1,000 times. Not all articles published in Nature receive that kind of response, yet people still refer to Nature publications in awed tones. Some people might "read" Nature each week, but, seriously, does anyone actually read the research articles if they're not in the field?

Science depends more on a slow and steady accumulation of knowledge than upon "breakthrough" papers. Geoff Watts has argued in the BMJ that we should "pension off the major breakthrough". I'd echo this, and I'd agree with Harold Varmus: we need a cure for CNS disease.

The Evil Empire Strikes Back

Picture, if you will, Darth Vader. Big, black armour. Heavy breathing. Imagine that, tired of being right hand man to the Emperor, Darth Vader has decided to venture into scientific publishing. Imagine what his publishing house would be like.

Elsevier has a reputation as "evil" (as seen here, here, here , here, here, here, here and here, but not here or here). This is mainly due to making massive profits (total revenues for 2005 were £1.4 billion) while restricting access to scientific research. They have also been noted for an involvement in the arms trade, and the censorship of published work.

Open access has been a thorn in Elsevier's side, much as the Rebel Alliance were to the Imperials in Star Wars. Open access threatens the ability of publishers like Elsevier to maintain their hold over library budgets, which is why Elsevier has repeatedly criticised OA (several society publishers have joined in too as they perceive OA to be a threat to their society's income, drafting the Washington DC Principles).

However, Elsevier recently announced a Sponsored Article option on some of their journals. Their hand was effectively forced by CERN, doing in effect what PLoS' boycott failed to do. Elsevier refuse to call it open access, and authors do not retain copyright. They even allowed authors to post preprints, allowing self-archiving. It might have been thought that Elsevier had laid to rest their hostility to open access.

We'd have been wrong. In a brilliant piece of investigative journalism, Nature have revealed a fruitful relationship that Elsevier, Wiley and the American Chemical Society have with a PR guru, advising them on how to take on the open access movement. Not just any guru, but the same one who represented Enron and ExxonMobil. A director at Wiley is quoted as saying that "Media messaging is not the same as intellectual debate". This explains some of the outrageous spin we've seen about open access ("government censorship"; "no peer review").

The blogosphere is beginning to react. The IWR blog understates it somewhat when they say that this will "do little for the reputations of the publishers involved". Chris Leonard predicts that "they won't be able to use these arguments even if they wanted to", while Jonathan Eisen has confidently exclaimed that "Their ship is sinking and they are grabbing at the last little pieces of wood they can find".

Hopefully, this news thoroughly discredits the smears of these publishers against open access. Was this their open exhaust port?

Response to '10 Problems with the Peer-Review Publishing Process'

Kevin Dewalt's blog on the 19th January includes 10 criticisms of peer review. I've posted a comment on his blog with my response to each of the points, but I'll copy them here as well.

Kevin's original points are italicised, and I've made a couple of additional comments since I replied on his blog that are indicated by square brackets. I hope I've corrected some misconceptions about peer review.

---
1. Unstated real or perceived conflicts of interest. Reviewers and authors can have relationships with entities that have an ulterior motive in getting material published.

True, but many journals, such as mine, require authors and reviewers to declare their competing interests - in our medical journals, these interests are published with the article. Editors are used to watching out for this.
---
2. Peer-review process advances slower than scientific progress.

Yes, but peer review doesn't stop someone first posting their article on their own web-site, discussing their work at conferences, or posting their work on a pre-print server like ArXiv. Anyway, scientific progress isn't as rapid as people believe, and without the checks and balances that peer review gives, all sorts of rubbish would be published, and scientists would have to follow even more blind alleys than they do already after reading profoundly flawed research. Peer review adds some rigour into the process of communicating scientific research. Less haste, more speed is an apt concept here.
---
3. The current process does not provide authors and reviewers with basic collaborative web tools.

That's nothing to do with peer review, just the delays in the Web 2.0 revolution getting to publishers. PLoS ONE (published by Public Library of Science, another OA publisher) does now offer reviewers and authors interactive tools to annotate articles. Many journals, like mine and the BMJ, allow any reader to comment on a published article.
---
4. Authors lose copyright privileges when publishing yet are often forced to publish to continue career advancement.

Traditional journals insist on copyright transfer. Many open access journals, including those published by BioMed Central and PLoS, allow the authors to retain copyright. The article is published under a Creative Commons Attribution License.
---
5. Peer-review networks tend to form around cliques. Those “outside the club” of a particular discipline - where often the best ideas surface - cannot get published because new ideas are rejected by the current establishment. As a result great ideas are often lost.

I don't believe that this complaint is really that valid. The complaints I've read about were by top scientists who couldn't get their idea published in Nature, Cell or Science. Well, just publish it elsewhere. There are plenty of journals that aren't as picky as those journals, and if authors had a little more self-awareness they'd recognise that they aim too high sometimes. Besides, many journals don't use established lists of reviewers, but go straight to those publishing related work and ask them. So, yes, you usually have to be a published scientist to review, but then it is called *peer* review, isn't it? I doubt that "the best ideas" surface outside academic research, the lone researcher is more likely to be a kook than a genius. There are some geniuses out there, but they are the ones you read about in the news - there's a teensy bit of a selection bias going on...
---
6. Precedence is often establish by those with the best personal contacts and not those who first introduce new theories.

I don't see the basis for this argument. Precedence does go to those who first raised a theory, so long as scientists are aware of it [this is the idea of 'priority']. Those who publish in languages other than English are at a disadvantage, admittedly, but some journals allow republication of work in English that was previously published elsewhere in another language, so that gives authors the possibility to widen their audience. Peer reviewers go out of their way to alert authors to work that first demonstrated something, and I have also insisted that authors cite certain studies. Scientists are very attuned to giving due credit for the origin of ideas or techniques.
---
7. There is no medium for wider, instant dissemination. Doctors or researchers who prepare a presentation or speech cannot “publish it” to a wider audience.

Yes, they can. ArXiv and other pre-print servers allow the publication of non-reviewed work (see e.g. Public Knowledge Project). Theses and dissertations can be published electronically (e.g. NDLTD, MIT on DSpace). This Portugese university repository, for example, allows the publication of reports, presentation etc. If a university doesn't have a repository for this kind of material, then it should do! Staff and students can take the lead, rather than waiting for journals to do it for them - journals are traditionally for peer-reviewed research, why would we necessarily expect them to post presentations? That said, Nature has recently launched Nature Protocols, so publishers are making some effort to include material that is outside their usual range.

8. Participating in the review process has little benefit for the reviewer. Performing reviews can take an enormous amount of time and the written reviews are not themselves “published”.

Reviewing takes between 2-6 hours, according a survey I read [an average of 3 hours]. I've seen reviews done in 10 minutes...
Here are a few reasons for participating in peer review:
- Allows a researcher control over what is published in their field - they are the "gatekeepers of science".
- Allows researchers to ensure that what is published accurately reflects and acknowledges their field.
- Can help a scientist get promoted and get grants, as journals often list the names of their reviewers annually.
- In the case of the medical journals in the BMC-series, published by BioMed Central, we *do* publish the reports, along with the name of the reviewer.
- Reviewers are actually paid by a small minority of journals [the BMJ pays £25], and more commonly can get other perks such as discounts on reading or publishing in the journal.
- Reviewers get the opportunity to read their competitors' work months before it will be published, and unscrupulous reviewers can deliberately block publication.
- It's interesting! They're scientists, they enjoy critiquing science!

9. Reviews and reviewers are not “reviewed”. An author who receives a biased review or one based on poor critical thinking has no recourse to publicly respond or invite others to comment.

Not true. Editors assess the reviewer reports and qualifications. Authors who receive what they perceive to be a biased review can appeal to the editor, and request a further opinion. If they are badly treated and the journal is a member of the Committee on Publication Ethics (such as BioMed Central, BMJ, Lancet) then authors can even take a case to that body [currently only editors can submit a case, but often will in cases of a dispute]. Some journals (BMJ, Lancet) have an ombudsman.

10. Journals can be prohibitively expensive for some in the developing world.

Yes - this is one of the reasons why open access is a good idea! The research is free to read for anyone with Internet access. Traditional pay-to-view journals are also members of a scheme called HINARI, a WHO project that allows some people in developing countries to read the research for free (but it does have limitations, as they need to be connected to an institution).

Does peer review work?

There are now a reasonable numbers of studies from journals such as the BMJ and JAMA on the factors affecting peer review. For example, we know due to a piece of work done by my colleagues that while author-suggested reviewers appear to return reports of equal quality to editor-suggested reviewers, they are significantly kinder to the authors in their recommendations on publication.

One of those authors, Pritt Tamber, regularly makes clear his belief that peer review doesn't work, most recently arguing in a BMJ Rapid Response that "Much of the research conducted at the BMJ [...] showed that there is little or no objective value to the process, yet journals and their editors persist with—and advocate—peer review; their only defence is that "there's nothing better," even though few have tried to find an alternative".

As Pritt notes, one alternative is the system used by Biology Direct, published by BioMed Central. The idea is that authors obtain reviews from three members of the reviewing board. If the author cannot find three members of the board to agree (or to themselves solicit an external review) the manuscript is considered to be rejected. If they can get three reports, then the manuscript will be published, no matter what the reviewers say. The twist is that the comments of the reviewers will be included at the end of the manuscript, as an integral part of the manuscript, and signed by the reviewers. The author can make revisions to the manuscript if they wish or even withdraw it, but equally they can ignore the comments and publish despite them. This is with the knowledge that readers will be able to see the reviewers' dissent. Other alternatives include the community peer review being tried by Philica, PLoS ONE and Nature (Nature's experiment appears to have been unsuccessful, but that is no reason to write-off the idea). More journals, publishers and researchers need to go out on a limb to explore new and better ways to assess and critique scientific research.

Before we go too far with condemning peer review, it is worth remembering that without an evidence base, we won't be able to work out where peer review works, where it doesn't and why, and how to improve it.

Much of the research done into the effects of peer review has been, in my opinion at least, quite superficial. Reading it has really only told me what I knew already from working as an editor.

My wish-list for studies of peer review are:

1. Creating a metric of "on-topicness" that editors can use to assess how relevant a reviewer's expertise is to a piece of research or an aspect of that work. This could be by simple similarity analyses, comparing their PubMed abstracts to the abstract of the submitted manuscript, or by more complicated semantic analyses.
2. Comparing manuscripts that were accepted to those rejected to examine the predictive factors. Some of these have been done, but the analyses always strike me as simplistic. The sample size needs to be greater, and the journals chosen need to not be so highly selective - is it really that interesting to see the factors that influence publication in journals like Nature, The Lancet or NEJM? I really want to see are the factors that affect whether a study is ever published in a reputable journal.
   
3. A side-by-side comparison of published articles with the original submitted version (before any peer review in any journal). This could be done by a paid panel who would be able to spend the time to do an in depth analysis; an alternative would be to invite journal clubs at universities worldwide to analyse manuscripts in this way (a sort-of Seti@Home for journalology). Did peer review noticeably improve the work?
   
4. Examine the fate of articles rejected by journals. Several studies of this nature have been conducted, but they mainly focus only on the journal it is eventually published in and the Impact Factor of the publishing journal. Why not examine whether any changes had been made since rejection? What about whether the rejected work is cited and read? Do a panel and journal clubs agree that the work is now sound, even if it might be uninteresting?
5. Compare the ability of different editors to assess a manuscript and select appropriate reviewers under time pressure, pitted against some of the new semi-automated tools available, like etBLAST. This would be like a peer review Olympiad.
   

It is tough to design and conduct good studies to examine peer review, but editors need to make the effort, else skeptics like Pritt will have a point. Now, just as soon as I have some spare time...