Peer review lite at PLoS ONE?

PLoS ONE, the 'Open Access 2.0' journal trumpeted by the Public Library of Science, launched late last year. Editors and reviewers often make arbitrary decisions about importance, in a chase for the Impact Factor. The idea of removing the need for journals to select the most 'important' science, and instead concentrating on publishing solid, sound science is a good one. This is a philosophy already followed by the BMC-series journals, which I'm involved with, although we do frequently reject articles on the basis that they present no advance in the field. To assess the soundness of a manuscript you usually need to find at least two experts to judge the topic, methods and statistics, and when the journal was announced I was genuinely puzzled as to how PLoS ONE would run their peer review any differently to other journals. There has been debate in the blogosphere by some who were under the impression that PLoS ONE was doing way with peer review entirely. An example I have come across gives me cause to worry that rather than focussing on conducting solid peer review, the system PLoS ONE uses will indeed sometimes scrimp on peer review.

One of the really interesting features of PLoS ONE is the annotation and discussion system. It isn't the first journal to allow readers to post comments, but it is the first to allow them to attach comments to a certain part of the published article, like a post-it note. There is a list of the Most Annotated articles, and on the day I looked one of these was A Large Specific Deterrent Effect of Arrest for Patronizing a Prostitute.

Alongside a comment discussing the use of the term "prostitute" is the Academic Editor's viewpoint. PLoS ONE may be experimental, but they don't have open peer review as standard (i.e. named, rather than anonymous reviewing, with the reports published), so this is not standard practice (they do name the Academic Editor for each article). The editor commented that "Although this manuscript was quite far from my own field of expertise, I accepted to act as academic referee for this manuscript because I felt that it was important that this type of manuscript should be published in a open access mode, and that the possibility for further discussions offered by this new journal would be very positive. Although I am reasonably confident that the scientific content and the statistics performed have been conducted appropriately, this does not mean to say that I condone all that this manuscript contains".

This brought me up short. I handle the peer review of articles on which I am not an expert, but I never make a decision to publish based only on my assessment of the manuscript. This is what peer review is for. The Academic Editor was Etienne Joly, an immunogeneticist who is also a strong supporter of open access - he is also an editorial board member for Biology Direct, another experiment in peer review published by BioMed Central.

I respect Dr Joly, but there is something worrying about an article being accepted after only being assessed by someone who is not a peer of the authors. I'm not sure that an immunologist can assess the conduct and reporting of public health/social science research. This isn't peer review, it is editorial selection. Indeed, PLoS ONE states that:
"AEs [Academic Editors] can employ a variety of methods to reach a decision in which they are confident:

Based on their own knowledge and experience;
Through discussion with other members of the editorial board;
Through the solicitation of formal reports from independent external referees".

Chris Surridge, the Managing Editor of PLoS ONE, has said that "When papers are submitted they get assigned to one of these editors based on the content of the paper and the editor’s specific areas of expertise". Now whilst the board of Academics Editors is impressive, it is still only 200 people. As PLoS ONE has ambitiously opened itself to submissions from across all of science, not just biology and medicine, it is impossible that all submissions will find an Academic Editor who will be an expert. If an Academic Editor is pressed for time (as most academics are) might they not take the easy route and attempt to assess a manuscript themselves that they are not qualified to judge, rather than embarking on the process of selecting external peer reviewers?

Etienne Joly went on to say in his viewpoint that "I have little doubt that this subject will lead to active debates. But this is exactly what PlosOne is about: Open Acess, and open discussions". PLoS ONE appears to be genuinely aiming to replace the pre-publication review process with "community-based open peer review", while at the same time not quite admitting this publicly, arguing that "the pre-publication assessment of papers is definitely ‘peer-review’". What concerns me most about the discussion of peer review surrounding the launch of PLoS ONE is the perception that 'only' assessing the technical quality of a manuscript is somehow easy. It's not. If you don't have the fall back option of claiming that something is "out of scope", "not of interest to our readers", or "more suited to a more specialized journal", then the job of assessing manuscripts actually gets harder, not easier.

Editorial selection is the process already used by the Elsevier journal Medical Hypotheses, which states boldly that "The editor sees his role as a 'chooser', not a 'changer': choosing to publish what are judged to be the best papers from those submitted". It has been said of the journal that it "exists to let people publish their craziest ideas". I would not imagine that this is a reputation that PLoS ONE hopes to emulate.
---

The most annotated article on PLoS ONE (aside from the testing 'Sandbox') now has 10 comments. Wonderful! However, they are all from an author of the article, to external links such as PubChem. Likewise, 5 of the 6 comments on the next most annotated article are links or notes added by the author. An annotation on another article is a note about the correct orientation of a figure. Is that not the sort of thing that is integral to the manuscript and needs correcting in the production process?