What is WebmedCentral?

Loosely following the style of Jeffrey Beall's assessment in The Charleston Advisor of various OA startups, here is an assessment of WebmedCentral, a new post-publication review biomedical journal.

DESCRIPTION
WebmedCentral is a post-publication review biomedical web portal launched in July 2010. It aims to "eliminate bias, increase transparency, empower authors, improve speed and accountability, and encourage free exchange of ideas." There is no pre-publication screening, although the instructions for authors imply some oversight for issues such as patient consent. Authors may submit revised versions. Articles can be read for free on the website, where they may be reviewed both by reviewers solicited by the authors and by readers. There is a list of "Scholarly Reviewers" on the site. Readers may also rate articles. Biomedical videos are also published. The journal has ISSN 2046-1690, but articles do not appear to have DOIs. It is not indexed in PubMed, but the articles are indexed on Google Scholar. The site aims to host other open access, open peer reviewed journals.

SUMMARY
Content: Primary scientific research, case reports, and reviews make up the bulk of the articles, alongside opinion, hypotheses, and outright fringe science. None have been peer reviewed before publication.

Usability: The site has a category listing, browse by date, featured articles, popular articles, most reviewed articles, RSS feeds, basic and advanced search, latest reviews. The PDF is only available via a Javascript link.

Cost: Free to read and publish, unless the author pays the US$50 Premium Upload fee.

Licensing: Authors retain copyright. Personal non-commercial use, digital archiving and self-archiving are allowed, though no standard license is used and details are confusing

Contact:

Address: Suite 250, 162-168 Regent Street, London W1B 5TD, United Kingdom
Phone: None given
Fax: None given
Email: contact@webmedcentral.com or http://www.webmedcentral.com/Contact_Us
URL: http://www.webmedcentral.com

COST
Free to read and publish, the journal aims to receive income from advertising and sponsorship. They offer a "premium upload service" for $50 per article that allows authors to simply email their submission to the journal. Scholarly Reviewers who post three reviews can obtain a free "premium upload".

LICENSE
The simplest formulation is that "Authors keep copyright to the article but our readers will be freely able to read, copy, save, print and privately circulate the article." However, the details are less clear. At one point they say authors "are free to publish it elsewhere" but also say elsewhere that "we require ... an exclusive license". They also say that users have a "free, irrevocable, worldwide, perpetual right of access for personal non-commercial use, subject to proper attribution of authorship and ownership of rights" but then say users may "view or download a single copy of the material on this website solely for your personal, non-commercial use". But they allow self-archiving: "WebmedCentral allows the final version of all published research articles to be placed in any digital archive immediately on publication. Authors are free to archive articles themselves." The precise freedom all this gives to users to reproduce the text is unclear, but calling WebmedCentral "open access" would be misleading.

EVALUATION
The approach of WebmedCentral is reminiscent of Google Knol, which is where PLoS Currents is hosted, or of a preprint server, except there is an active post-publication peer review system.

Open peer review and community peer review are not new ideas. A similar approach to that of WebmedCentral was tried by Philica in recent years without great success; the site rapidly filled with crank publications. Another was 'E-Biomed', which was stifled and instead became PubMed Central. Although anticipated a decade ago, biomedical publishing has been wary of preprints and other proposals to remove or reduce pre-publication peer review. BioMed Central's Genome Biology had a preprint server, but it closed in January 2006. A humanities institute is experimenting with community review on Shakespeare Quarterly, though they are using a hybrid model rather than abandoning invited pre-publication review. More generally, MediaCommons argue for community peer review in their book "Planned Obsolescence". They are far from naïve, noting that

'Too many digital publishing experiments, like Philica, have lagged due to an assumption that might be summed up as "if you build it, they will come."'

Ethics:
The journal requires appropriate ethical approval for human and animal studies and will remove studies if they find that they fail to meet ethical standards. Articles may also be removed in cases of scientific misconduct or plagiarism. They suggest that authors use statistical advice, and ask authors to adhere to reporting standards such as CONSORT. They ask authors of clinical trials to adhere to the Good Publication Practice guidelines, but do not specifically mention trial registration. They endorse the ICMJE criteria for authorship and the use of medical writers should be declared. Funding and competing interests should be declared, though there is no definition of a competing interest. They ask authors to suggest at least three reviewers and to not only pick "friendly reviewers", and say they may invite further reviewers. How these policies are enforced and who enforces them is not clear.

Technical issues:
Previous versions of an article should be linked to, but this fails. The journal allows digital archiving and digital preservation by LOCKSS members. Some test articles can be found as Word documents that are not visible via the search, which raises questions about site security. The presentations of figures is in a sidebar and sometimes without even a thumbnail, though the pop-up view is user friendly. The referencing could be improved, with clearer formatting and hyperlinks down to the references. Some of the formatting of the reviews is poor, with changes in font and font size, and several reviews are double posted.

Publication volume:
There are 366 published articles as of 30/01/2011. Submission rates appear to have peaked following publicity in August, and have since declined (see figure).

Content:
There is currently no indication on the articles that they have received no pre-publication review. As might be expected given the lack of pre-publication review, some of the articles are fringe science: aliens, homeopathy, prayer, and telepathy are all represented. There is an account of chiropractic care of a patient with fibromyalgia, an opinion article on the evidence for homeopathy in acute upper respiratory tract infections by Peter Fisher and colleagues, a study linking 'emotional quotient' and telepathy that has the obligatory mention of quantum theory, an article on the hunt for alien life that takes in the Higgs Boson, the Bermuda Triangle, and alien implants , a virtually content-free account of acupuncture in rats, and an intercessory prayer study. The latter is, thankfully, a deliberate satire.

When you get this kind of opportunity of publishing without a filter, sex always seem to come to the fore: step forward, a hypothesis on why women don't sleep with the first man they see when they ovulate, two case reports of priapism, an institutional review of Peyronie's disease, and a case report of penile fracture. As pointed out by two reviewers, it contained the unfortunate typo in the title of the corpus callosum (in the brain) rather than the corpus cavernosum, hence it was republished (demonstrating that the article version system is not working).

Many of the articles are unpublishable in any biomedical journal: a rant about academic exploitation; a review of the biological activities of a herb that the author seems to have forgotten to write; an account of a trauma registry that is confused and sketchy; a review of oral health and inequality for which the recommendations section appears to be lifted verbatim from Nunn et al. 2008, who are not cited. How many more of the articles will contain plagiarism would be interesting to see.

On the more positive side, there are a series of interesting articles by three authors: Leonid Perlovsky has published a series of mainly hypothetical papers, e.g. on language and cognition; William Maloney, a New York dentist, has published a series of overviews and historical accounts, e.g. the medical legacy of Babe Ruth; Uner Tan has published a series of articles of his observations and theories of quadrapedal locomotion in humans, e.g. these two cases.

Other interesting reads are a survey of the role of hairdressers and bartenders as informal emotional support following the 9/11 attacks and their responses to this role, a study by Robert Dellavelle on how journals don't require ethics approval for meeting abstracts, and a series of witty anecdotes by an Israeli psychiatrist of cases of "curing demons" in his patients.

Around a quarter of the articles are case reports. The insatiable demand of hospital doctors to publish case reports has clashed with a reluctance of medical journals to publish what are often "me too" publications offering little generalisable insights, and which are often poorly presented and incomplete. The recent trend of open access case report journals - BMJ Case Reports, Cases Journal, Journal of Medical Case Reports, Clinical medicine insights. Case reports, Case Reports in Ophthalmology etc. from Karger, Case reports in Medicine from Hindawi and the American Journal of Case Reports (free, not OA) - doesn't appear to be matching demand.

There are also 58 reviews, 31 opinion articles, and at least 15 of the "original articles" are not research articles; less than half of the articles on WebmedCentral are primary research.

Reviews:
Some of the reviewers are published researchers, but they usually have only a handful of publications and they would be unlikely to be selected as peer reviewers by a mainstream biomedical journal editor – this could be seen as a positive or a negative. There are pages listing reviewer details, but the reviews by a single reviewer are not listed.

Relatively few articles have received an insightful review or comment. Around 55% (201 articles) have received a review of some kind, and the most any article has received is six reviews (see right hand panel of the figure). 138 reviews were unsolicited and 211 were solicited by the authors. The quality of the reviews is usually low. Just over half of both solicited and unsolicited reviews contain critical analysis, i.e. at least some mention of improvements the authors could make to their article, meaning that probably less than 25% of all articles receive any degree of critical analysis. Many reviews are sycophantic, for example one case report is said to be "the best ever article publishe[sic] so far". Many merely state what the articles is about - one author-invited reviewer spends 358 words reiterating what the article says and telling us that it is a "must read" - or give the views of the reviewer on the subject rather than the article - another reviewer devotes a mere 23 words of a 430 word review to even mentioning the paper. Most reviews are very short: the average is only ~115 words for both author-suggested and unsolicited reviewers; the longest is just over 1500 words (see left hand panel of the figure for the length distribution). Comments with critical analysis are much longer (~175 words) than those without (~50 words). If I were to see reviews like most of those on WebmedCentral during standard peer review, I would never use that reviewer again.

Some of the reviews include comments such as "this is suitable for publication" or "I hope it is accepted", which indicate a lack of awareness of the publishing model. One author has even reviewed his own paper. An article I consider unpublishable received the reviews, and I quote them in full, "good" and "No comment".

There are some examples where robust review has taken place. The concerns raised by the reviewers on this paper, including a lack of mention of ethics or consent, would lead most editors to reject such a paper – but WebmedCentral has no routine mechanism for doing this. Authors responded to reviews only on a handful of papers. A lively debate developed around a physician's self-case report, but this was a rare exception. I found one example of what appears to be functional peer review, with the authors revising their work and the reviewer stating that they are happy with the revisions.

In Bambi, Thumper's parents taught him that "If you can't say something nice... don't say nothing at all", but I think that the opposite applies in peer review. If you can't come up with critical comments about a paper, you're probably missing something: every paper has something wrong with it. The sycophantic nature of many of the reviews in WebmedCentral might be inherent to open (named) peer review, but in my experience and according to published studies, open peer review increases the length of reviews and makes them more polite, but has no effect on review quality. Another factor may be that many of the authors and reviewers of WebmedCentral are from India: R. A. Mashelkar argued in Science that "India must free itself from a traditional attitude that condemns irreverence", and Nikhil Kumar and Shirish Ranade argued in Current Science that "it is a preponderance of obsequious reverence and sycophancy that has placed the science in the country on a downhill slope." Are we seeing this unwillingness to criticise in action?

Overall assessment:
This is an interesting experiment in post-publication peer review, which both indicates the possibilities – instant publication, open community review – and the perils – unsound science, unbalanced opinion, and substandard writing being presented as part of the scientific literature.

Building a functioning publishing platform from scratch is no easy matter, and several hundred publications in seven months is an impressive figure. There has been a noticeable engagement from the community, with over 365 submissions and a total of nearly 350 reviews in seven months, 40% of them by reviewers not suggested by the authors. However, the submission rate is declining and the coverage and quality of reviews is not nearly high enough to functionally replace pre-publication review.

The onus is on the authors to obtain reviews: the journal states that it will obtain reviews, but this is not in evidence - just under half of the papers have no reviews, and 30% have only one review. More effort needs to be put into gaining reviews from qualified experts.

Reviews are essentially worthless if nobody pays any attention to them, be that an editor, the authors or the readers. Pre-publication peer review is not merely a filter, but it also acts to improve articles. On WebmedCentral there is no pressure for articles to be revised in accordance with any critical reviews, perhaps other than author embarrassment. As reviewers see a lack of response to their comments, they may lose enthusiasm.

Without a clear indication that reviewers have criticised an article and no indication that the articles are not peer reviewed, readers may view the work uncritically. If reviewers state for instance that the work is not sound, this should be clearly flagged up to readers near the top of the page, and articles should be sortable based on the answers given in the review from and the rating given by reviewers and readers. Another layer should be added, allowing articles to be promoted by agreement from their 'Scholarly Reviewers' to a "publication standard" level, giving authors an incentive to revise their work. "Featured articles" do exist, but the criteria used are not revealed. WebmedCentral are forming an "Advisory Board" of "eminent scientists"; perhaps this board will increase the rigour of the site.

Without the oversight of an editor choosing diverse reviewers and because most scientists are unaware of the site, it may become a closed community of the same authors positively reviewing each others' work – the precise opposite of the aim of the journal. Unless the process is reformed, WebmedCentral is likely to remain a "Cargo Cult science" journal, which in the main publishes articles that only superficially resemble the peer-reviewed literature, and that are reviewed in a manner that is only a pale imitation of pre-publication peer review.

Other commentary on WebmedCentral:

WebmedCentral and ginger pee on Pleion, a discussion on Ramy Aziz's FriendFeed, a blog review, a response to receiving an email from them, a brief welcome, a mention by Jenny Rohn in a Guardian comment thread, a comment by an author, an assessment in a sports science magazine, a mention in Lab Times, a critical appraisal of an article by PJ of Pyjamas in Bananas, a blog reaction in French, and a few Twitter comments.

But who runs this site?
"We are a group of medical and management professionals with no affiliation to any major biomedical publishing group" is all they say, but who runs the site shall be revealed in my next post: "Who are WebmedCentral?".

A piece of peer review history

This is a guest post by Joe Dunckley
I love browsing PubMed Central. How about this, from May 1985. The paper, Exaggerated responsiveness to thyrotrophin releasing hormone: a risk factor in women with coronary artery disease, by Fowler and Dean in the BMJ, is not in itself a work of unusual historical significance. But the associated sections in the document could be: is this where open peer-review began?

In a comment posted letter published the day after I was born, Thomas Walever said:

SIR,-I congratulate you on your publication of the paper by Drs J W Dean and P W B Fowler (25 May, p 1555) complete with the referees' comments and authors' replies. This correspondence indeed illustrates some of the problems with the peer review system. It was a help to know that others have had some of the same problems that we have had as authors. While comments made by referees are often helpful, it is indeed distressing when one does not agree with the criticism of the methods. This is especially true, it seems, for statistical matters; and this was well illustrated here. It has even happened that a major criticism was that something was not done which in fact had been done and was clearly stated to be so in the manuscript.

As a suggestion, perhaps matters might be improved if we had to sign our names to our reviews. There would be many problems with this, but the comments made to the authors would probably be more careful, considerate, and constructive. Some journals already suggest this as an option; perhaps the practice should be encouraged?

Twenty-five years on, and we do not have universal open peer-review. Further experiments have been conducted, though, led by the BMJ, which now uses signed reviews (but, so far as I can tell, does not publish the reviewer reports). In 1999, a blinded trial of open review by the BMJ editors found that open review didn't really make a difference to the quality of reports, and probably lengthened the time taken for review. Editor Richard Smith, campaigner for open-science and publishing reform, stubbornly decided to adopt it anyway, citing ethical reasons: you can not be tried by an anonymous judge. (In his editorial, Smith also declares his intentions to publish reports alongside the accepted papers, and introduce live community review, neither of which have come to fruition, so far as I can see.)

Other trials have had more positive findings regarding benefits of open review, such as more thorough reports, but the utilitarian merits of open review remain contested. This Nature Neruoscience editorial from 1999 complained (without evidence) that open review is likely to lead to "bland" and "timid" reviews, in which technical deficiencies are identified, but no comment is made regarding the interest level of a paper. If true, this would, of course, be a problem for high-end journals like Nature Neuroscience which select on the basis of interest level.

It's nearly ten years since the BMJ's switch to open review, and, so far as I know, the the model is still restricted to just a minority of medical journals and hardly any journals in other fields. The BMJ's initial hope to publish reviewer reports seems to have been forgotten. Open peer review seems to be just another area where publishers are still looking at new technology and discussing where they want to go, when everyone on the internet is asking "are you guys coming, or what?"

This is an edited re-post of something previously posted on cotch dot net.

Open peer review & community peer review

There has been a lot of discussion about 'open peer review' lately - this letter to Nature is just the latest example. With all these opinions and hypotheses about peer review flying around, I think that it is useful to make some distinctions between the different types of 'open' review, so here goes.

Traditional peer review. Anonymous reports received pre-publication. Letters to the editor are considered by many journals, but especially in paper journals relatively few are published. All the BioMed Central journals accept signed comments from readers.

Open peer review. Named, pre-publication review, which is how the BMC-series medical journals work, and the BMJ too. The difference lies in that the reviews are available for readers to see in the BMC-series medical journals, but the BMJ never made this move. Comments can also be posted by readers: the BMJ's Rapid Responses should be envied by any journal. It is controversial as some reviewers don't wish to be named, and it can make finding peer reviewers harder, but to anyone who doubts the open peer review works I can point out that the BMJ has published hundreds of peer reviewed articles since it introduced open peer review, and the medical journals in the BMC series have published thousands of peer reviewed articles since they launched in 2000. Open peer review can work.

Open and permissive peer review. This is Biology Direct's approach. Articles are published if they receive reviews solicited by the author from at least 3 members of the reviewing board (aside from pseudoscience, which the editors will veto), with the comments included at the end of the article, unless the author withdraw the manuscript. More here, and I discussed their approach in a previous post. Comments can be posted by readers, as with the other BioMed Central journals.

Community peer review. The idea of community peer review is to avoid peer review being the domain of a biased subset of the scientific community, and it has a powerful philosophy that "given enough eyeballs, all bugs are shallow". It can be either anonymous or named, and still happens before formal publication, but the difference is that reviewers volunteer rather than being selected by the editors. The manuscript is public while under review, but explicitly is not 'published' at that point. This was how Nature's experiment worked (or didn't work), but it was alongside the usual anonymous editorially selected reviews, and the comments don't seem to have been treated as 'proper' reviews by the editors.

Atmospheric Chemistry and Physics uses a similar approach, apparently with much more success than Nature. The editors refuse articles that don't meet minimal scientific standards, then post the remaining articles for 8 weeks of Interactive Public Discussion (named or anonymous), then publish the final version. There doesn't appear to be any mention of rejecting articles after the initial public posting, so this permissive peer review resembles a community version of Biology Direct.

The Journal of Interactive Media in Education uses named reports, and invites review from the community. The two-step process involves private, named review by invited reviewers, followed by publication of a preliminary version that is reviewed further by the community before final, formal publication.

Permissive peer review, post-publication commentary. This is PLoS ONE's approach. They have minimal peer review, with the expectation that the scientific community will then comment on and annotate the articles. I was already a bit skeptical of the merits of minimal peer review, as are others, and now a Nature news story, among others, has attacked the publication of a study on HIV and circumcision in PLoS ONE, arguing that peer review failed in this case. Sending out an unbalanced press release written by the author seems to have compounded the problem, and wasn't very responsible. A lengthy response has been posted to the article, showing that post-publication review can work, but plenty of journals have the option to post comments, and the horse has already bolted.

No peer review, post-publication commentary. This is how Philica works, and now Nature Preceedings, part pre-print, part repository for preliminary work. I don't think that Philica is working; Nature Preceedings will probably fare better. An essential difference is that while Philica is clogged with pseudoscience, Nature Preceedings explictly won't post pseudoscience, and it has the Nature brand name to help it gather interest and comments. I found an optimistically titled Web 2.0 Peer Reviewed Science Journal, which has a website but no articles. "This page that you are reading now is a review site, and I (Philip Dorrell) am the intended reviewer. If you, as an author of a scientific paper, are interested in having me review your paper, all you have to do is publish your paper as a web page, and then send an email". Hmm... sorry Philip, but peer review involves more than just your opinion on articles. Web 2.0 requires users and content.

BioMed Central is open access, PLoS is open access, the BMJ is open access, Nature Preceedings is open access, and they are all experimenting with peer review. Matthew Falagas has commented in Open Medicine (the open access journal that arose out of the editorial dispute at the CMAJ), after spotting this pattern of a link between experimenting with peer review and open access. I think it is worth stating that despite this trend, open access and open peer review don't necessarily go together. The biology journals in the BMC-series still have anonymous review, as do the PLoS journals. The problem of access to an article is at a tangent to the problem of reviewing it - but, of course, community peer review can't work if not enough people have access to the article.

I think that if there is doubt in the integrity of peer review (and there is more and more doubt), this increases the imperative for exposing pre-publication review processes. Journals can't just be paternalistic or secretive about peer review, and readers shouldn't take it on trust that an article labelled as 'peer reviewed' has been rigorously critiqued by experts in the field. PLoS ONE is encouraging its reviewers to make their reviews public on the published article, which is a great step. Requiring reviewers to opt-out would be even stronger, but PLoS Medicine recently backed away from this policy.

If journals really want community peer review to work, we cannot just sit back and wait for comments to come in. Pre-publication peer review takes a massive effort on the part of editors to find qualified reviewers, and the chances of enough qualified reviewers stumbling across an article and feeling obliged to leave comments to make post-publication review viable and vibrant are low. Ways to solicit comments are essential, using email alerts for example. In a definite step in the right direction, PLoS ONE is organising virtual 'journal clubs'. Remember that anyone who has had a face-to-face journal club at their institute about a BioMed Central article, or a BMJ article, or a PLoS article, can and should post the results of the discussion as a comment on the article.

I think that open peer review and community peer review are the future of assessing scientific articles. It doesn't stop there - I've not even mentioned wikis!

A case study in open peer review

Last year, BMC Anesthesiology published an article by Andrew Vickers and colleagues on the use of acupuncture for the pain caused by thoracotomy; it was a pilot study to examine whether a randomized controlled trial (RCT) was feasible. Thoracotomy is surgery to the chest to allow access to the lungs. Andrew Vickers is on the editorial boards for several of our journals, a respected statistician and trial methodology expert with an interest in testing complementary medicines. It was unlikely when he submitted the work that there would be any fatal flaws. However, we don't allow submissions from our editorial boards to escape peer review - and I've seen many manuscripts from editorial board members fail to pass muster.

Who would be suitable to review the manuscript? It's about acupuncture, so acupuncturists, right? Well, partly.

If we were to ask only acupuncturists to review, there are two potential drawbacks. Acupuncturists believe in the efficacy of acupuncture, otherwise they probably wouldn't be acupuncturists. If there were flaws in the study they might be inclined to give it the benefit of the doubt, whereas someone without a vested interest in the intervention under study might raise objections. The other issue is that they might be unfamiliar with pain relief in this particular setting. On the other hand, were we to only approach those who had never used acupuncture and who were otherwise experts in pain relief we would face two potential biases. For one, they might be skeptical that acupuncture works at all and thus be too picky, raising unreasonable objections to block publication. Another consideration is that they might themselves have a vested interest in the drugs used to relieve post-surgical pain, perhaps having received speaking fees or consulted for pharmaceutical companies.

To ensure rigorous and fair review, we needed someone who was familiar with acupuncture for pain relief, preferably with an additional experience of either randomized controlled trials, systematic reviews, or anesthesia for surgical interventions. Although it is not itself an RCT, the purpose of the trial was as a pilot to see if an RCT was feasible, and those who have conducted systematic reviews will have a good knowledge of critical appraisal. Secondly, we needed someone familiar with post-thoracotomy pain relief other than acupuncture, preferably with a knowledge of randomized controlled trials. Lastly, we needed someone with a familiarity with randomized controlled trials and anesthesiology for post-surgical pain, were either of the other two not themselves familiar with it.

BMC Anesthesiology is open access, but it is unusual among journals in another way. It has open peer review, as do all the medical journals in the BMC-series. By this we mean that the reviewers consent to their names being made to be known to the authors, and for their reports to be made public if the manuscript is accepted for publication. Open peer review allows me to sweep back the curtain, and reveal the peer review process, like a magician flaunting the secrecy of Magic Circle.

Among our reviewers were two complementary and alternative medicine experts, Betsy Singh and Hugh MacPherson. Betsy Singh has published on the use of several complementary medicines, including a systematic review of acupuncture for pain relief. Hugh MacPherson has published on ways to ensure the safety and accurate reporting of acupuncture trials, and is familiar with randomized controlled trials. We also had two reviewers who are researchers of pain relief, Deniz Karakaya and Jorge Dagnino. Deniz Karakaya is a thoracic surgeon has published on the use of conventional anesthetics in various procedures, including for post-thoracotomy pain. Jorge Dagnino has also published on the use of anesthetics for post-surgical pain, including thoracotomy. We had the full house.

What criticisms or points did they raise? Dr Singh had no complaints, and it isn't surprising to see a reviewer of a manuscript by Dr Vickers say this. Dr MacPherson was well-disposed to the study, but raised several points where the authors could better report details of their work, or better justify a statement. Dr Karayaka had only two relatively minor criticisms, asking for more detail on their procedures. Dr Dagnino raised the most objections, requiring many more methodological details, and questioning some of the conclusions. It is interesting to see that the two reviewers who asked the authors to make the most corrections were an acupuncturist and a traditional anesthesiologist. Both types of researchers applied their skills of critical appraisal to help the authors improve their work. Upon re-review, Drs Karayaka and Dagnino had some remaining questions, and the editorial staff determined that the authors' response to this second round of review was satisfactory and we proceeded with publication.

Although the study is limited in its scope and conclusions, inevitable for a pilot, uncontrolled study in only 36 patients, and although it would be easy to dismiss by many journals as not interesting enough to publish, we thought it necessary and valuable to have enough qualified reviewers to assess it, and we obtained the advice of four reviewers who together were qualified to judge all the main aspects of the work. Judging soundness (technical validity) isn't easy, and is more difficult than measuring the level of interest. It isn't that uninteresting though - more than 3,000 readers have accessed the article from our website in the past 10 months, and more will have read it on PubMed Central, a level of interest for which many blog writers would willingly give up a kidney.

Does peer review work?

There are now a reasonable numbers of studies from journals such as the BMJ and JAMA on the factors affecting peer review. For example, we know due to a piece of work done by my colleagues that while author-suggested reviewers appear to return reports of equal quality to editor-suggested reviewers, they are significantly kinder to the authors in their recommendations on publication.

One of those authors, Pritt Tamber, regularly makes clear his belief that peer review doesn't work, most recently arguing in a BMJ Rapid Response that "Much of the research conducted at the BMJ [...] showed that there is little or no objective value to the process, yet journals and their editors persist with—and advocate—peer review; their only defence is that "there's nothing better," even though few have tried to find an alternative".

As Pritt notes, one alternative is the system used by Biology Direct, published by BioMed Central. The idea is that authors obtain reviews from three members of the reviewing board. If the author cannot find three members of the board to agree (or to themselves solicit an external review) the manuscript is considered to be rejected. If they can get three reports, then the manuscript will be published, no matter what the reviewers say. The twist is that the comments of the reviewers will be included at the end of the manuscript, as an integral part of the manuscript, and signed by the reviewers. The author can make revisions to the manuscript if they wish or even withdraw it, but equally they can ignore the comments and publish despite them. This is with the knowledge that readers will be able to see the reviewers' dissent. Other alternatives include the community peer review being tried by Philica, PLoS ONE and Nature (Nature's experiment appears to have been unsuccessful, but that is no reason to write-off the idea). More journals, publishers and researchers need to go out on a limb to explore new and better ways to assess and critique scientific research.

Before we go too far with condemning peer review, it is worth remembering that without an evidence base, we won't be able to work out where peer review works, where it doesn't and why, and how to improve it.

Much of the research done into the effects of peer review has been, in my opinion at least, quite superficial. Reading it has really only told me what I knew already from working as an editor.

My wish-list for studies of peer review are:

1. Creating a metric of "on-topicness" that editors can use to assess how relevant a reviewer's expertise is to a piece of research or an aspect of that work. This could be by simple similarity analyses, comparing their PubMed abstracts to the abstract of the submitted manuscript, or by more complicated semantic analyses.
2. Comparing manuscripts that were accepted to those rejected to examine the predictive factors. Some of these have been done, but the analyses always strike me as simplistic. The sample size needs to be greater, and the journals chosen need to not be so highly selective - is it really that interesting to see the factors that influence publication in journals like Nature, The Lancet or NEJM? I really want to see are the factors that affect whether a study is ever published in a reputable journal.
   
3. A side-by-side comparison of published articles with the original submitted version (before any peer review in any journal). This could be done by a paid panel who would be able to spend the time to do an in depth analysis; an alternative would be to invite journal clubs at universities worldwide to analyse manuscripts in this way (a sort-of Seti@Home for journalology). Did peer review noticeably improve the work?
   
4. Examine the fate of articles rejected by journals. Several studies of this nature have been conducted, but they mainly focus only on the journal it is eventually published in and the Impact Factor of the publishing journal. Why not examine whether any changes had been made since rejection? What about whether the rejected work is cited and read? Do a panel and journal clubs agree that the work is now sound, even if it might be uninteresting?
5. Compare the ability of different editors to assess a manuscript and select appropriate reviewers under time pressure, pitted against some of the new semi-automated tools available, like etBLAST. This would be like a peer review Olympiad.
   

It is tough to design and conduct good studies to examine peer review, but editors need to make the effort, else skeptics like Pritt will have a point. Now, just as soon as I have some spare time...