Open peer review & community peer review

There has been a lot of discussion about 'open peer review' lately - this letter to Nature is just the latest example. With all these opinions and hypotheses about peer review flying around, I think that it is useful to make some distinctions between the different types of 'open' review, so here goes.

Traditional peer review. Anonymous reports received pre-publication. Letters to the editor are considered by many journals, but especially in paper journals relatively few are published. All the BioMed Central journals accept signed comments from readers.

Open peer review. Named, pre-publication review, which is how the BMC-series medical journals work, and the BMJ too. The difference lies in that the reviews are available for readers to see in the BMC-series medical journals, but the BMJ never made this move. Comments can also be posted by readers: the BMJ's Rapid Responses should be envied by any journal. It is controversial as some reviewers don't wish to be named, and it can make finding peer reviewers harder, but to anyone who doubts the open peer review works I can point out that the BMJ has published hundreds of peer reviewed articles since it introduced open peer review, and the medical journals in the BMC series have published thousands of peer reviewed articles since they launched in 2000. Open peer review can work.

Open and permissive peer review. This is Biology Direct's approach. Articles are published if they receive reviews solicited by the author from at least 3 members of the reviewing board (aside from pseudoscience, which the editors will veto), with the comments included at the end of the article, unless the author withdraw the manuscript. More here, and I discussed their approach in a previous post. Comments can be posted by readers, as with the other BioMed Central journals.

Community peer review. The idea of community peer review is to avoid peer review being the domain of a biased subset of the scientific community, and it has a powerful philosophy that "given enough eyeballs, all bugs are shallow". It can be either anonymous or named, and still happens before formal publication, but the difference is that reviewers volunteer rather than being selected by the editors. The manuscript is public while under review, but explicitly is not 'published' at that point. This was how Nature's experiment worked (or didn't work), but it was alongside the usual anonymous editorially selected reviews, and the comments don't seem to have been treated as 'proper' reviews by the editors.

Atmospheric Chemistry and Physics uses a similar approach, apparently with much more success than Nature. The editors refuse articles that don't meet minimal scientific standards, then post the remaining articles for 8 weeks of Interactive Public Discussion (named or anonymous), then publish the final version. There doesn't appear to be any mention of rejecting articles after the initial public posting, so this permissive peer review resembles a community version of Biology Direct.

The Journal of Interactive Media in Education uses named reports, and invites review from the community. The two-step process involves private, named review by invited reviewers, followed by publication of a preliminary version that is reviewed further by the community before final, formal publication.

Permissive peer review, post-publication commentary. This is PLoS ONE's approach. They have minimal peer review, with the expectation that the scientific community will then comment on and annotate the articles. I was already a bit skeptical of the merits of minimal peer review, as are others, and now a Nature news story, among others, has attacked the publication of a study on HIV and circumcision in PLoS ONE, arguing that peer review failed in this case. Sending out an unbalanced press release written by the author seems to have compounded the problem, and wasn't very responsible. A lengthy response has been posted to the article, showing that post-publication review can work, but plenty of journals have the option to post comments, and the horse has already bolted.

No peer review, post-publication commentary. This is how Philica works, and now Nature Preceedings, part pre-print, part repository for preliminary work. I don't think that Philica is working; Nature Preceedings will probably fare better. An essential difference is that while Philica is clogged with pseudoscience, Nature Preceedings explictly won't post pseudoscience, and it has the Nature brand name to help it gather interest and comments. I found an optimistically titled Web 2.0 Peer Reviewed Science Journal, which has a website but no articles. "This page that you are reading now is a review site, and I (Philip Dorrell) am the intended reviewer. If you, as an author of a scientific paper, are interested in having me review your paper, all you have to do is publish your paper as a web page, and then send an email". Hmm... sorry Philip, but peer review involves more than just your opinion on articles. Web 2.0 requires users and content.

BioMed Central is open access, PLoS is open access, the BMJ is open access, Nature Preceedings is open access, and they are all experimenting with peer review. Matthew Falagas has commented in Open Medicine (the open access journal that arose out of the editorial dispute at the CMAJ), after spotting this pattern of a link between experimenting with peer review and open access. I think it is worth stating that despite this trend, open access and open peer review don't necessarily go together. The biology journals in the BMC-series still have anonymous review, as do the PLoS journals. The problem of access to an article is at a tangent to the problem of reviewing it - but, of course, community peer review can't work if not enough people have access to the article.

I think that if there is doubt in the integrity of peer review (and there is more and more doubt), this increases the imperative for exposing pre-publication review processes. Journals can't just be paternalistic or secretive about peer review, and readers shouldn't take it on trust that an article labelled as 'peer reviewed' has been rigorously critiqued by experts in the field. PLoS ONE is encouraging its reviewers to make their reviews public on the published article, which is a great step. Requiring reviewers to opt-out would be even stronger, but PLoS Medicine recently backed away from this policy.

If journals really want community peer review to work, we cannot just sit back and wait for comments to come in. Pre-publication peer review takes a massive effort on the part of editors to find qualified reviewers, and the chances of enough qualified reviewers stumbling across an article and feeling obliged to leave comments to make post-publication review viable and vibrant are low. Ways to solicit comments are essential, using email alerts for example. In a definite step in the right direction, PLoS ONE is organising virtual 'journal clubs'. Remember that anyone who has had a face-to-face journal club at their institute about a BioMed Central article, or a BMJ article, or a PLoS article, can and should post the results of the discussion as a comment on the article.

I think that open peer review and community peer review are the future of assessing scientific articles. It doesn't stop there - I've not even mentioned wikis!

Archivangelism - has the means become the end?

Stevan Harnad has always been insistent on the need for immediate, free access to academic research, and he sees self-archiving as the means to this end.

Now that he recognises that self-archiving may only be compatible with some publishers if there is a delay in access, Stevan (who is normally uncompromising, e.g. "OA itself is non-negotiable") seems to have accepted this fudge, which is not immediate free access: "Access to the immediate deposit can then either be set as Open Access immediately, or (in case of a publisher embargo), as Closed Access, provisionally". This is the "Immediate-Deposit & Optional-Access" (IDOS) policy. Even with a fancy name, and as Jan Velterop has noted, it's not open access.

Stevan has been adamantly against the crystal-ball-gazing that predicts a loss in subscriptions resulting from self-archiving, but his own crystal ball predicts that following a universal adoption of 'IDOS' repositories, "Embargoes will disappear very soon thereafter" [my emphasis].

Stevan has criticised advocates of open access journals who claim that open access journals will reduce costs, as he insists that it is access that is paramount and not costs, which is a fair point. Yet he now criticises open access journals and hybrid journals for the extra costs he says they impose, for example criticising CERN's decision to put up funds to pay for article processing charges as "diverting scarce research funds from research to paying publishers". It seems that if open access journals might save costs, that's a side issue; if they might cost more, we should take heed.

While there might be issues with double-payment in hybrid journals, that can be corrected by adjusting subscription costs - it isn't an insurmountable issue. Further, open access journals don't have this problem, but Stevan's criticisms of Springer Open Choice don't often allow for this distinction. How open access can be paid for is open to debate - the debate has usually focussed on the article processing charge, but as Peter Suber has noted not all open access journals charge author fees. Costs can also be met using advertising (though hardly an uncontroversial way for a medical journal to recover costs), grants from societies (this is how the Beilstein Journal of Organic Chemistry is funded), charitable and philanthropic donations, or even by cutting costs. An interesting aspect of article processing charges is that they can result in price competition on an article-by-article basis.

Rather than being a system with no barriers to access (the definition of open access as I understand it), under self-archiving each author (having signed over copyright to the publisher) needs to deposit their articles in their local repository (if one exists), then each reader needs to realise that the repository exists, find the article, and then possibly have to contact the author to get a copy, and then have the author respond and send it to them. Sadly, this seems to be laborious, incomplete and prone to failure. I had thought that the launch of Google Scholar opened up the possibility of self-archiving really being viable, but their practice of linking to all copies of an article that could be found on the web only lasted a few short months and links to free versions are now intermittent - possibly (probably), Google were sat on by publishers.

Libraries will apparently continue to subscribe to journals that their users can access at no cost, despite evidence that libraries are acutely attuned to cost, and an existing trend of university actions against high journal prices. Publishers will apparently be happy to have a business model that depends on their customers paying for a product that can be obtained for free. This business model is actually seen in shareware, though shareware is much rarer now than 10 to 20 years ago, and certainly the music industry isn't too keen on this business model. If self-archiving doesn't cause a collapse in subscriptions to closed access journals, I'd suggest that it will implicitly have failed to achieve its goal. Surely the aim is to provide immediate free access to peer reviewed academic research - if libraries and readers are unaware that they could get what they are still paying for at no cost, might that not imply that self-archiving doesn't provide universal access? Might there be those who aren't paying and who don't realise that the research is accessible, and therefore never read the article? We need to end this farcical situation of researchers not reading articles, and although I like the idea of self-archiving I don't believe that if offers a complete enough solution, or is sustainable.

Stevan's insistence on self-archiving has even extended to criticising central deposit in PubMed Central, arguing that articles must only be deposited in institutional repositories. What is BioMed Central to do; stop depositing in PubMed Central? It's hardly as though we block our authors depositing in their own local repositories.

Stevan lays claim to being the true voice of open access. In responding to an article by Ben Goldacre he argued that "It is not "two [Gold] OA publishing organisations" that have led the fight for OA, but one (Green and Gold) organisation -- the same one that first coined the term OA in 2002: the Budapest Open Access Initiative (BOAI)". Actually, it's not true that BOAI coined 'open access'.

Stevan at times appears to be entirely opposed to the idea of open access journals (despite apparently supporting the idea a decade ago), for example raising criticisms against BioMed Central as a publisher from the outset.

I can't agree with Stevan's insistence upon local institutional self-archiving to the apparent exclusion of other approaches to open access. It appears to me that with archivangelism the means has become the end.

Who pays the fiddler calls the tune?

"In our capitalist society, one of our basic tenets is who pays the fiddler calls the tune". This was a comment by Jeffrey Drazen, Editor-in-Chief of the New England Medical Journal, referring to the idea that because many open access journals charge an article processing charge rather than a subscription fee (including most of BioMed Central's), they will lower their standards to take the cash.

Peter Suber has already countered that slur criticism in detail, so I don't need to repeat the arguments here, but I think that it's worth noting that in my time at BioMed Central I've seen the acceptance rate fall by at least 10% on the BMC series. This is not exactly what you'd expect from a vanity publisher pushing its editors hard to accept articles no matter the quality, in order to take the cash. In case I've been too subtle, that's because BioMed Central is not a vanity publisher, and payment and publisher finances are kept separate from editorial decision making. Matt Cockerill doesn't stand over us with a whip, demanding more acceptances.

The reason I quoted Jeffrey Drazen above is that I think that there's quite an irony here. The NEJM raised $30 million in advertising revenues in 2005, with nearly 5000 pages of advertising. Is Dr Drazen admitting that one of the "basic tenets" of the NEJM is that the advertisers call the shots? [Shome mishtake, shurely?]

We didn't pay him, honestly

The Filter, "Your regular dose of public-interest Internet news and commentary from the Berkman Center for Internet & Society at Harvard Law School", carries a piece from David Weinberger that so gushes about BioMed Central that I would have been embarrassed to write it myself. I'm not too embarrassed to link to it though.

Mashups, mirrors, mining and open access

The Creative Commons Attribution License under which open access articles are made available by both BioMed Central and PLoS allows others to create sites that incorporate the content of these articles, so long as the original source is clearly acknowledged.

Two ways to do this are mashups and mirrors. According to Wikipedia, a mashup is a site that "combines content from more than one source into an integrated experience". A mirror is an exact copy of a website.

BioMed Central officially has four mirrors to which we feed content, at INIST in France, University of Potsdam in Germany, PubMed Central at the NIH, and the National Library of the Netherlands' e-Depot. I've come across some unofficial mirrors in specific areas like genomics and bioinformatics in the past.

PLoS ONE already has its own unofficial mirror, created by the people behind HubMed: PLoS Too. Rather than displaying the articles as they appear on the publisher's site, this is a pared-down view of the articles and it has a couple of good new features - auto-generated tags for each article, and a very quick live search box.

On the mashup side, Free Biomedical Images has made open access images available in a searchable database, mainly (entirely?) taken from BioMed Central articles, and fully attributed. Users can comment on the images, rate them, email them to a friend and jump to the published article.

A key feature of open access is that we don't hide away the full text of our articles. The entire 'corpus' of our open access research articles is available on our data mining page for anyone to download. Gerry Rubin has said that "the most important reason for Open Access is data mining".

The idea of mashups, scripts and extensions is just beginning to reach the bioinformatics community. A bioinformatics mashup by Alfonso Valencia is iHOP (Information Hyperlinked over Proteins), which links information about genes and proteins to text from PubMed. Not satisfied with just a mashup, Mark Wilkinson has created a Greasemonkey userscript called iHOPerator that enhances the iHOP website with tag clouds. You can read about in his BMC Bioinformatics article. Two other Greasemonkey userscripts link PubMed to social bookmarking sites, one to CiteULike, the other to Connotea. A third links Google Scholar to CiteULike. The iSpecies search engine pulls together information about any species you enter from disparate sources, including scores of biomedical databases and even Yahoo! Image search.

Mashups, mirrors and mining are definitely the future of science publishing.

A case study in open peer review

Last year, BMC Anesthesiology published an article by Andrew Vickers and colleagues on the use of acupuncture for the pain caused by thoracotomy; it was a pilot study to examine whether a randomized controlled trial (RCT) was feasible. Thoracotomy is surgery to the chest to allow access to the lungs. Andrew Vickers is on the editorial boards for several of our journals, a respected statistician and trial methodology expert with an interest in testing complementary medicines. It was unlikely when he submitted the work that there would be any fatal flaws. However, we don't allow submissions from our editorial boards to escape peer review - and I've seen many manuscripts from editorial board members fail to pass muster.

Who would be suitable to review the manuscript? It's about acupuncture, so acupuncturists, right? Well, partly.

If we were to ask only acupuncturists to review, there are two potential drawbacks. Acupuncturists believe in the efficacy of acupuncture, otherwise they probably wouldn't be acupuncturists. If there were flaws in the study they might be inclined to give it the benefit of the doubt, whereas someone without a vested interest in the intervention under study might raise objections. The other issue is that they might be unfamiliar with pain relief in this particular setting. On the other hand, were we to only approach those who had never used acupuncture and who were otherwise experts in pain relief we would face two potential biases. For one, they might be skeptical that acupuncture works at all and thus be too picky, raising unreasonable objections to block publication. Another consideration is that they might themselves have a vested interest in the drugs used to relieve post-surgical pain, perhaps having received speaking fees or consulted for pharmaceutical companies.

To ensure rigorous and fair review, we needed someone who was familiar with acupuncture for pain relief, preferably with an additional experience of either randomized controlled trials, systematic reviews, or anesthesia for surgical interventions. Although it is not itself an RCT, the purpose of the trial was as a pilot to see if an RCT was feasible, and those who have conducted systematic reviews will have a good knowledge of critical appraisal. Secondly, we needed someone familiar with post-thoracotomy pain relief other than acupuncture, preferably with a knowledge of randomized controlled trials. Lastly, we needed someone with a familiarity with randomized controlled trials and anesthesiology for post-surgical pain, were either of the other two not themselves familiar with it.

BMC Anesthesiology is open access, but it is unusual among journals in another way. It has open peer review, as do all the medical journals in the BMC-series. By this we mean that the reviewers consent to their names being made to be known to the authors, and for their reports to be made public if the manuscript is accepted for publication. Open peer review allows me to sweep back the curtain, and reveal the peer review process, like a magician flaunting the secrecy of Magic Circle.

Among our reviewers were two complementary and alternative medicine experts, Betsy Singh and Hugh MacPherson. Betsy Singh has published on the use of several complementary medicines, including a systematic review of acupuncture for pain relief. Hugh MacPherson has published on ways to ensure the safety and accurate reporting of acupuncture trials, and is familiar with randomized controlled trials. We also had two reviewers who are researchers of pain relief, Deniz Karakaya and Jorge Dagnino. Deniz Karakaya is a thoracic surgeon has published on the use of conventional anesthetics in various procedures, including for post-thoracotomy pain. Jorge Dagnino has also published on the use of anesthetics for post-surgical pain, including thoracotomy. We had the full house.

What criticisms or points did they raise? Dr Singh had no complaints, and it isn't surprising to see a reviewer of a manuscript by Dr Vickers say this. Dr MacPherson was well-disposed to the study, but raised several points where the authors could better report details of their work, or better justify a statement. Dr Karayaka had only two relatively minor criticisms, asking for more detail on their procedures. Dr Dagnino raised the most objections, requiring many more methodological details, and questioning some of the conclusions. It is interesting to see that the two reviewers who asked the authors to make the most corrections were an acupuncturist and a traditional anesthesiologist. Both types of researchers applied their skills of critical appraisal to help the authors improve their work. Upon re-review, Drs Karayaka and Dagnino had some remaining questions, and the editorial staff determined that the authors' response to this second round of review was satisfactory and we proceeded with publication.

Although the study is limited in its scope and conclusions, inevitable for a pilot, uncontrolled study in only 36 patients, and although it would be easy to dismiss by many journals as not interesting enough to publish, we thought it necessary and valuable to have enough qualified reviewers to assess it, and we obtained the advice of four reviewers who together were qualified to judge all the main aspects of the work. Judging soundness (technical validity) isn't easy, and is more difficult than measuring the level of interest. It isn't that uninteresting though - more than 3,000 readers have accessed the article from our website in the past 10 months, and more will have read it on PubMed Central, a level of interest for which many blog writers would willingly give up a kidney.

Response to '10 Problems with the Peer-Review Publishing Process'

Kevin Dewalt's blog on the 19th January includes 10 criticisms of peer review. I've posted a comment on his blog with my response to each of the points, but I'll copy them here as well.

Kevin's original points are italicised, and I've made a couple of additional comments since I replied on his blog that are indicated by square brackets. I hope I've corrected some misconceptions about peer review.

---
1. Unstated real or perceived conflicts of interest. Reviewers and authors can have relationships with entities that have an ulterior motive in getting material published.

True, but many journals, such as mine, require authors and reviewers to declare their competing interests - in our medical journals, these interests are published with the article. Editors are used to watching out for this.
---
2. Peer-review process advances slower than scientific progress.

Yes, but peer review doesn't stop someone first posting their article on their own web-site, discussing their work at conferences, or posting their work on a pre-print server like ArXiv. Anyway, scientific progress isn't as rapid as people believe, and without the checks and balances that peer review gives, all sorts of rubbish would be published, and scientists would have to follow even more blind alleys than they do already after reading profoundly flawed research. Peer review adds some rigour into the process of communicating scientific research. Less haste, more speed is an apt concept here.
---
3. The current process does not provide authors and reviewers with basic collaborative web tools.

That's nothing to do with peer review, just the delays in the Web 2.0 revolution getting to publishers. PLoS ONE (published by Public Library of Science, another OA publisher) does now offer reviewers and authors interactive tools to annotate articles. Many journals, like mine and the BMJ, allow any reader to comment on a published article.
---
4. Authors lose copyright privileges when publishing yet are often forced to publish to continue career advancement.

Traditional journals insist on copyright transfer. Many open access journals, including those published by BioMed Central and PLoS, allow the authors to retain copyright. The article is published under a Creative Commons Attribution License.
---
5. Peer-review networks tend to form around cliques. Those “outside the club” of a particular discipline - where often the best ideas surface - cannot get published because new ideas are rejected by the current establishment. As a result great ideas are often lost.

I don't believe that this complaint is really that valid. The complaints I've read about were by top scientists who couldn't get their idea published in Nature, Cell or Science. Well, just publish it elsewhere. There are plenty of journals that aren't as picky as those journals, and if authors had a little more self-awareness they'd recognise that they aim too high sometimes. Besides, many journals don't use established lists of reviewers, but go straight to those publishing related work and ask them. So, yes, you usually have to be a published scientist to review, but then it is called *peer* review, isn't it? I doubt that "the best ideas" surface outside academic research, the lone researcher is more likely to be a kook than a genius. There are some geniuses out there, but they are the ones you read about in the news - there's a teensy bit of a selection bias going on...
---
6. Precedence is often establish by those with the best personal contacts and not those who first introduce new theories.

I don't see the basis for this argument. Precedence does go to those who first raised a theory, so long as scientists are aware of it [this is the idea of 'priority']. Those who publish in languages other than English are at a disadvantage, admittedly, but some journals allow republication of work in English that was previously published elsewhere in another language, so that gives authors the possibility to widen their audience. Peer reviewers go out of their way to alert authors to work that first demonstrated something, and I have also insisted that authors cite certain studies. Scientists are very attuned to giving due credit for the origin of ideas or techniques.
---
7. There is no medium for wider, instant dissemination. Doctors or researchers who prepare a presentation or speech cannot “publish it” to a wider audience.

Yes, they can. ArXiv and other pre-print servers allow the publication of non-reviewed work (see e.g. Public Knowledge Project). Theses and dissertations can be published electronically (e.g. NDLTD, MIT on DSpace). This Portugese university repository, for example, allows the publication of reports, presentation etc. If a university doesn't have a repository for this kind of material, then it should do! Staff and students can take the lead, rather than waiting for journals to do it for them - journals are traditionally for peer-reviewed research, why would we necessarily expect them to post presentations? That said, Nature has recently launched Nature Protocols, so publishers are making some effort to include material that is outside their usual range.

8. Participating in the review process has little benefit for the reviewer. Performing reviews can take an enormous amount of time and the written reviews are not themselves “published”.

Reviewing takes between 2-6 hours, according a survey I read [an average of 3 hours]. I've seen reviews done in 10 minutes...
Here are a few reasons for participating in peer review:
- Allows a researcher control over what is published in their field - they are the "gatekeepers of science".
- Allows researchers to ensure that what is published accurately reflects and acknowledges their field.
- Can help a scientist get promoted and get grants, as journals often list the names of their reviewers annually.
- In the case of the medical journals in the BMC-series, published by BioMed Central, we *do* publish the reports, along with the name of the reviewer.
- Reviewers are actually paid by a small minority of journals [the BMJ pays £25], and more commonly can get other perks such as discounts on reading or publishing in the journal.
- Reviewers get the opportunity to read their competitors' work months before it will be published, and unscrupulous reviewers can deliberately block publication.
- It's interesting! They're scientists, they enjoy critiquing science!

9. Reviews and reviewers are not “reviewed”. An author who receives a biased review or one based on poor critical thinking has no recourse to publicly respond or invite others to comment.

Not true. Editors assess the reviewer reports and qualifications. Authors who receive what they perceive to be a biased review can appeal to the editor, and request a further opinion. If they are badly treated and the journal is a member of the Committee on Publication Ethics (such as BioMed Central, BMJ, Lancet) then authors can even take a case to that body [currently only editors can submit a case, but often will in cases of a dispute]. Some journals (BMJ, Lancet) have an ombudsman.

10. Journals can be prohibitively expensive for some in the developing world.

Yes - this is one of the reasons why open access is a good idea! The research is free to read for anyone with Internet access. Traditional pay-to-view journals are also members of a scheme called HINARI, a WHO project that allows some people in developing countries to read the research for free (but it does have limitations, as they need to be connected to an institution).

Does peer review work?

There are now a reasonable numbers of studies from journals such as the BMJ and JAMA on the factors affecting peer review. For example, we know due to a piece of work done by my colleagues that while author-suggested reviewers appear to return reports of equal quality to editor-suggested reviewers, they are significantly kinder to the authors in their recommendations on publication.

One of those authors, Pritt Tamber, regularly makes clear his belief that peer review doesn't work, most recently arguing in a BMJ Rapid Response that "Much of the research conducted at the BMJ [...] showed that there is little or no objective value to the process, yet journals and their editors persist with—and advocate—peer review; their only defence is that "there's nothing better," even though few have tried to find an alternative".

As Pritt notes, one alternative is the system used by Biology Direct, published by BioMed Central. The idea is that authors obtain reviews from three members of the reviewing board. If the author cannot find three members of the board to agree (or to themselves solicit an external review) the manuscript is considered to be rejected. If they can get three reports, then the manuscript will be published, no matter what the reviewers say. The twist is that the comments of the reviewers will be included at the end of the manuscript, as an integral part of the manuscript, and signed by the reviewers. The author can make revisions to the manuscript if they wish or even withdraw it, but equally they can ignore the comments and publish despite them. This is with the knowledge that readers will be able to see the reviewers' dissent. Other alternatives include the community peer review being tried by Philica, PLoS ONE and Nature (Nature's experiment appears to have been unsuccessful, but that is no reason to write-off the idea). More journals, publishers and researchers need to go out on a limb to explore new and better ways to assess and critique scientific research.

Before we go too far with condemning peer review, it is worth remembering that without an evidence base, we won't be able to work out where peer review works, where it doesn't and why, and how to improve it.

Much of the research done into the effects of peer review has been, in my opinion at least, quite superficial. Reading it has really only told me what I knew already from working as an editor.

My wish-list for studies of peer review are:

1. Creating a metric of "on-topicness" that editors can use to assess how relevant a reviewer's expertise is to a piece of research or an aspect of that work. This could be by simple similarity analyses, comparing their PubMed abstracts to the abstract of the submitted manuscript, or by more complicated semantic analyses.
2. Comparing manuscripts that were accepted to those rejected to examine the predictive factors. Some of these have been done, but the analyses always strike me as simplistic. The sample size needs to be greater, and the journals chosen need to not be so highly selective - is it really that interesting to see the factors that influence publication in journals like Nature, The Lancet or NEJM? I really want to see are the factors that affect whether a study is ever published in a reputable journal.
   
3. A side-by-side comparison of published articles with the original submitted version (before any peer review in any journal). This could be done by a paid panel who would be able to spend the time to do an in depth analysis; an alternative would be to invite journal clubs at universities worldwide to analyse manuscripts in this way (a sort-of Seti@Home for journalology). Did peer review noticeably improve the work?
   
4. Examine the fate of articles rejected by journals. Several studies of this nature have been conducted, but they mainly focus only on the journal it is eventually published in and the Impact Factor of the publishing journal. Why not examine whether any changes had been made since rejection? What about whether the rejected work is cited and read? Do a panel and journal clubs agree that the work is now sound, even if it might be uninteresting?
5. Compare the ability of different editors to assess a manuscript and select appropriate reviewers under time pressure, pitted against some of the new semi-automated tools available, like etBLAST. This would be like a peer review Olympiad.
   

It is tough to design and conduct good studies to examine peer review, but editors need to make the effort, else skeptics like Pritt will have a point. Now, just as soon as I have some spare time...

Peer review lite at PLoS ONE?

PLoS ONE, the 'Open Access 2.0' journal trumpeted by the Public Library of Science, launched late last year. Editors and reviewers often make arbitrary decisions about importance, in a chase for the Impact Factor. The idea of removing the need for journals to select the most 'important' science, and instead concentrating on publishing solid, sound science is a good one. This is a philosophy already followed by the BMC-series journals, which I'm involved with, although we do frequently reject articles on the basis that they present no advance in the field. To assess the soundness of a manuscript you usually need to find at least two experts to judge the topic, methods and statistics, and when the journal was announced I was genuinely puzzled as to how PLoS ONE would run their peer review any differently to other journals. There has been debate in the blogosphere by some who were under the impression that PLoS ONE was doing way with peer review entirely. An example I have come across gives me cause to worry that rather than focussing on conducting solid peer review, the system PLoS ONE uses will indeed sometimes scrimp on peer review.

One of the really interesting features of PLoS ONE is the annotation and discussion system. It isn't the first journal to allow readers to post comments, but it is the first to allow them to attach comments to a certain part of the published article, like a post-it note. There is a list of the Most Annotated articles, and on the day I looked one of these was A Large Specific Deterrent Effect of Arrest for Patronizing a Prostitute.

Alongside a comment discussing the use of the term "prostitute" is the Academic Editor's viewpoint. PLoS ONE may be experimental, but they don't have open peer review as standard (i.e. named, rather than anonymous reviewing, with the reports published), so this is not standard practice (they do name the Academic Editor for each article). The editor commented that "Although this manuscript was quite far from my own field of expertise, I accepted to act as academic referee for this manuscript because I felt that it was important that this type of manuscript should be published in a open access mode, and that the possibility for further discussions offered by this new journal would be very positive. Although I am reasonably confident that the scientific content and the statistics performed have been conducted appropriately, this does not mean to say that I condone all that this manuscript contains".

This brought me up short. I handle the peer review of articles on which I am not an expert, but I never make a decision to publish based only on my assessment of the manuscript. This is what peer review is for. The Academic Editor was Etienne Joly, an immunogeneticist who is also a strong supporter of open access - he is also an editorial board member for Biology Direct, another experiment in peer review published by BioMed Central.

I respect Dr Joly, but there is something worrying about an article being accepted after only being assessed by someone who is not a peer of the authors. I'm not sure that an immunologist can assess the conduct and reporting of public health/social science research. This isn't peer review, it is editorial selection. Indeed, PLoS ONE states that:
"AEs [Academic Editors] can employ a variety of methods to reach a decision in which they are confident:

Based on their own knowledge and experience;
Through discussion with other members of the editorial board;
Through the solicitation of formal reports from independent external referees".

Chris Surridge, the Managing Editor of PLoS ONE, has said that "When papers are submitted they get assigned to one of these editors based on the content of the paper and the editor’s specific areas of expertise". Now whilst the board of Academics Editors is impressive, it is still only 200 people. As PLoS ONE has ambitiously opened itself to submissions from across all of science, not just biology and medicine, it is impossible that all submissions will find an Academic Editor who will be an expert. If an Academic Editor is pressed for time (as most academics are) might they not take the easy route and attempt to assess a manuscript themselves that they are not qualified to judge, rather than embarking on the process of selecting external peer reviewers?

Etienne Joly went on to say in his viewpoint that "I have little doubt that this subject will lead to active debates. But this is exactly what PlosOne is about: Open Acess, and open discussions". PLoS ONE appears to be genuinely aiming to replace the pre-publication review process with "community-based open peer review", while at the same time not quite admitting this publicly, arguing that "the pre-publication assessment of papers is definitely ‘peer-review’". What concerns me most about the discussion of peer review surrounding the launch of PLoS ONE is the perception that 'only' assessing the technical quality of a manuscript is somehow easy. It's not. If you don't have the fall back option of claiming that something is "out of scope", "not of interest to our readers", or "more suited to a more specialized journal", then the job of assessing manuscripts actually gets harder, not easier.

Editorial selection is the process already used by the Elsevier journal Medical Hypotheses, which states boldly that "The editor sees his role as a 'chooser', not a 'changer': choosing to publish what are judged to be the best papers from those submitted". It has been said of the journal that it "exists to let people publish their craziest ideas". I would not imagine that this is a reputation that PLoS ONE hopes to emulate.
---

The most annotated article on PLoS ONE (aside from the testing 'Sandbox') now has 10 comments. Wonderful! However, they are all from an author of the article, to external links such as PubChem. Likewise, 5 of the 6 comments on the next most annotated article are links or notes added by the author. An annotation on another article is a note about the correct orientation of a figure. Is that not the sort of thing that is integral to the manuscript and needs correcting in the production process?