A piece of peer review history

This is a guest post by Joe Dunckley
I love browsing PubMed Central. How about this, from May 1985. The paper, Exaggerated responsiveness to thyrotrophin releasing hormone: a risk factor in women with coronary artery disease, by Fowler and Dean in the BMJ, is not in itself a work of unusual historical significance. But the associated sections in the document could be: is this where open peer-review began?

In a comment posted letter published the day after I was born, Thomas Walever said:

SIR,-I congratulate you on your publication of the paper by Drs J W Dean and P W B Fowler (25 May, p 1555) complete with the referees' comments and authors' replies. This correspondence indeed illustrates some of the problems with the peer review system. It was a help to know that others have had some of the same problems that we have had as authors. While comments made by referees are often helpful, it is indeed distressing when one does not agree with the criticism of the methods. This is especially true, it seems, for statistical matters; and this was well illustrated here. It has even happened that a major criticism was that something was not done which in fact had been done and was clearly stated to be so in the manuscript.

As a suggestion, perhaps matters might be improved if we had to sign our names to our reviews. There would be many problems with this, but the comments made to the authors would probably be more careful, considerate, and constructive. Some journals already suggest this as an option; perhaps the practice should be encouraged?

Twenty-five years on, and we do not have universal open peer-review. Further experiments have been conducted, though, led by the BMJ, which now uses signed reviews (but, so far as I can tell, does not publish the reviewer reports). In 1999, a blinded trial of open review by the BMJ editors found that open review didn't really make a difference to the quality of reports, and probably lengthened the time taken for review. Editor Richard Smith, campaigner for open-science and publishing reform, stubbornly decided to adopt it anyway, citing ethical reasons: you can not be tried by an anonymous judge. (In his editorial, Smith also declares his intentions to publish reports alongside the accepted papers, and introduce live community review, neither of which have come to fruition, so far as I can see.)

Other trials have had more positive findings regarding benefits of open review, such as more thorough reports, but the utilitarian merits of open review remain contested. This Nature Neruoscience editorial from 1999 complained (without evidence) that open review is likely to lead to "bland" and "timid" reviews, in which technical deficiencies are identified, but no comment is made regarding the interest level of a paper. If true, this would, of course, be a problem for high-end journals like Nature Neuroscience which select on the basis of interest level.

It's nearly ten years since the BMJ's switch to open review, and, so far as I know, the the model is still restricted to just a minority of medical journals and hardly any journals in other fields. The BMJ's initial hope to publish reviewer reports seems to have been forgotten. Open peer review seems to be just another area where publishers are still looking at new technology and discussing where they want to go, when everyone on the internet is asking "are you guys coming, or what?"

This is an edited re-post of something previously posted on cotch dot net.