A piece of peer review history

This is a guest post by Joe Dunckley
I love browsing PubMed Central. How about this, from May 1985. The paper, Exaggerated responsiveness to thyrotrophin releasing hormone: a risk factor in women with coronary artery disease, by Fowler and Dean in the BMJ, is not in itself a work of unusual historical significance. But the associated sections in the document could be: is this where open peer-review began?

In a comment posted letter published the day after I was born, Thomas Walever said:

SIR,-I congratulate you on your publication of the paper by Drs J W Dean and P W B Fowler (25 May, p 1555) complete with the referees' comments and authors' replies. This correspondence indeed illustrates some of the problems with the peer review system. It was a help to know that others have had some of the same problems that we have had as authors. While comments made by referees are often helpful, it is indeed distressing when one does not agree with the criticism of the methods. This is especially true, it seems, for statistical matters; and this was well illustrated here. It has even happened that a major criticism was that something was not done which in fact had been done and was clearly stated to be so in the manuscript.

As a suggestion, perhaps matters might be improved if we had to sign our names to our reviews. There would be many problems with this, but the comments made to the authors would probably be more careful, considerate, and constructive. Some journals already suggest this as an option; perhaps the practice should be encouraged?

Twenty-five years on, and we do not have universal open peer-review. Further experiments have been conducted, though, led by the BMJ, which now uses signed reviews (but, so far as I can tell, does not publish the reviewer reports). In 1999, a blinded trial of open review by the BMJ editors found that open review didn't really make a difference to the quality of reports, and probably lengthened the time taken for review. Editor Richard Smith, campaigner for open-science and publishing reform, stubbornly decided to adopt it anyway, citing ethical reasons: you can not be tried by an anonymous judge. (In his editorial, Smith also declares his intentions to publish reports alongside the accepted papers, and introduce live community review, neither of which have come to fruition, so far as I can see.)

Other trials have had more positive findings regarding benefits of open review, such as more thorough reports, but the utilitarian merits of open review remain contested. This Nature Neruoscience editorial from 1999 complained (without evidence) that open review is likely to lead to "bland" and "timid" reviews, in which technical deficiencies are identified, but no comment is made regarding the interest level of a paper. If true, this would, of course, be a problem for high-end journals like Nature Neuroscience which select on the basis of interest level.

It's nearly ten years since the BMJ's switch to open review, and, so far as I know, the the model is still restricted to just a minority of medical journals and hardly any journals in other fields. The BMJ's initial hope to publish reviewer reports seems to have been forgotten. Open peer review seems to be just another area where publishers are still looking at new technology and discussing where they want to go, when everyone on the internet is asking "are you guys coming, or what?"

This is an edited re-post of something previously posted on cotch dot net.

Fraud epidemic in China?

This is a guest post by Joe Dunckley
Last week's Nature included a news feature on scientific misconduct in China which contained the extraordinary (but not unbelievable) claim that a third of all researchers at "top institutions" in China admitted to plagiarism, falsification, or fabrication. The feature contains the extreme example of the systematic fabrication of crystal structures, but one would hope that the majority of the confessions of misconduct represent no more than the borrowing of a few paragraphs by those for whom English is not a first language (a crime, but not a hanging offence). But every publisher has its examples of photoshopped figures and impossible datasets, and it's hard to deny that certain countries pop up more often than others.
The fun part of the article, though, is the comments thread, which is full of readers' speculating about why China should have such a high-rate of misconduct. A drive for quick success and control of the sector by short-sighted bureaucrats with no actual understanding of science are suggested. A rapid expansion of science with a bottom-heavy hierarchy and insufficient supervision, or else the fierce competition and pressure in a publish-or-perish world. Perhaps it's just in the nature of communist societies? Even the impact factor is cited as a contributory cause, and the blame somehow shifted onto the publishers.
Nobody seems to consider the simple possibility that researchers from other countries might have access to better tools for disguising their fraud.

Reviewing Medical Hypotheses

This is a guest post by Joe Dunckley
Zoë Corbyn writes in the Times Higher this week that Elsevier have "started an internal review" of legendary journal Medical Hypotheses following its publication last year of two hiv/aids denialism papers (covered in Bad Science here and Respectful Insolence here). One of the offending papers, lead-authored by notorious aids denialist Peter Duesberg, took an entire two days from submission to acceptance by the peer review shunning "journal", and had already been rejected from all of the real hiv/aids journals for making such embarrassing claims as that Uganda's population increase proves that hiv can not cause aids.

It would be a shame to loose the journal that gave us Ejaculation as a potential treatment of nasal congestion in mature males and the equally entertaining response, Ejaculation as a treatment for nasal congestion in men is inconvenient, unreliable and potentially hazardous, but at the same time, we have to consider whether we are really comfortable continuing to humour the confused outbursts of Bruce Charlton.

It's interesting to note that the best defence for the journal's existence that Corbyn could find was this: "while peer review worked for 'normal science', it also had the power to suppress radical ideas." The defence comes from intelligent design creationist Steve Fuller, whose ideas I don't think even Med Hypotheses sunk as low as publishing.

Hello!

This is a guest post by Joe Dunckley
I am Joe. You might remember me from such websites as cotch dot net, friendfeed, and twitter. I work for a STM publishing company, formerly editing molecular biology journals, but now helping them develop their web publishing technology. I will be posting my ill-considered thoughts about science publishing alongside Matt's on Journalology, so that I don't have to bore the readers of my real blog with them. Do follow my vanity site for updates on all my other blogging.