28 Apr 2007

What to do about late peer reviewers?

Editors and authors are left in the lurch when reviewers are late in returning their reports or even fail entirely to return a report. Although reviewers are usually volunteers, they have made a promise to the journal and their scientific colleagues, and the failure to return a report can greatly lengthen and complicate the review process.

Marc Hauser and Ernst Fehr writing in PLoS Biology have an idea of how to remedy this. "Reviewers that turn in their reviews late are punished, whereas those that arrive on time are rewarded". They suggest that "
for every day since receipt of the manuscript for review plus the number of days past the deadline, the reviewer's next personal submission to the journal will be held in editorial limbo for twice as long before it is sent for review" and "for every manuscript that a reviewer refuses to review, we add on a one-week delay to reviewing their own next submission".

I hate when reviewers are late, and it would be immensely satisfying to take revenge on them by snarling up their own submitted manuscripts, but I'm not sure that this is a workable system. It is true that journals can track the timeliness and helpfulness of reviewers - we do this at BioMed Central, so technical feasibility is not my objection.

Publishing is not a game; the aim is to get research checked and, if sound, published as quickly as possible. Deliberately adding in delays and checks both adds costs and impedes science. A publisher that started imposing such sanctions might lose those reviewers as both reviewers and authors, and introduce even more antagonism into what can already be a fraught process. This system would hit senior and well-known researchers the hardest, as they are most often asked to review and they simply can't agree to review every article sent to them. Reasons for declining would need to be distinguished - is someone who suggests a qualified and keen colleague deserving of sanction? What if they were inappropriate, or didn't even receive the email as it landed in their spam filter? What if they had a genuine reason that they were unable to return the report, such as the several reviewers in New Orleans at the time of Hurricane Katrina who were somewhat understandably late with their reports, or those suddenly falling ill or with a family emergency? What if they needed more time to complete a thorough reanalysis, as one of our editorial board members did recently?

What can we do instead? We already have a reviewer discount, such that reviewers who return their reports on time to a journal within the BMC series are entitled to a 20% discount on the article processing charge the next time (within a year) that they are the submitting author of a manuscript submitted to a journal in the series (i.e. BMC Bioinformatics, BMC Cancer etc.).

Editors can be ruthless. If an agreed reviewer is late, we may well make a decision without them if we already have reports in from other reviewers - if a reviewer doesn't want their time in reading the manuscript wasted and their opinions ignored, they should get the report in on time.

Fostering a good relationship with reviewers and authors helps. Authors who receive timely reviews will feel inclined to review quickly themselves. Authors who don't may well refuse to review until they have received a decision, the flip side of Hauser and Fehr's proposal. Equally, authors and reviewers need to remember that editors are human too. We sometimes receive a level of bad tempered abuse from authors that if we dished back out we'd be fired. A way to remind reviewers that we're human is to phone them - email can give the false impression that journals are run by robots, although we do find that email reminders can be very effective. A look at the statistics for the times that reviews are returned shows that most are returned within a day of us sending a reminder email letting the reviewer know that their report is due within three days.

If we respect each other and agree with the aim of efficiently and effectively assessing scientific research, we're all better off. I'm not sure that penalties are the best way to achieve this.


Bill Hooker said...

The first journal to threaten me with such "punishments" will receive by return post a letter requesting that they never ask me to review another manuscript and advising that they should expect never to get a ms sub from me again.

I get my reviews in on time anyway, but I'm a volunteer dammit. I can pretty much guarantee horrible backlash if Hauser and Fehr's proposal is ever tried. It's offensive.

Matt Hodgkinson said...

The new Seven Stones blog is also not a fan of this proposal. They suggest some sort of financial incentive, or a prize to the best reviewers, as well as publishing a list of reviewers. Pedro Beltrao, commenting on the post, also suggests an aggregated record of peer reviews across publishers - much like the new PublicationsList, but for reviewers.