Literature hacks: PubMed searches by RSS

This is a guest post by Joe Dunckley
There are all sorts of ways you could find out about new articles that you might want to read. There's that big room across campus that's full of old writings on paper, but that's too far away and they have some silly rule about not eating your lunch near their writings on paper, and anyway you're not sure you still have the card that lets you in. You can't trust your colleagues to point out an article that isn't crushingly mediocre, unless it's because it concerns a species or a disease whose name sounds mildly amusingly puerile, but those ones are never actually remotely related to your work. You subscribe to electronic tables of contents, but these days everyone's publishing in PLoS ONE, and you're not wading through their contents every week in the hope of finding the occasional thing that's relevant. You could regularly search PubMed, but that means typing in keywords over and over, and wading through the results asking yourself, "have I seen this paper already, or do I just feel like I've seen this paper already?"

So you could subscribe to email alerts for your PubMed searches, but my god, man, what the hell do you think you're doing? What, you haven't got enough email already? Make you feel special, having your phone stop you every five minutes with unimportant impersonal notifications? If it's not private, not time-critical, and does not require a reply, it should not be pestering you with an email. That article has taken ten years to get from concept to publication, it can wait a little longer for you to read it -- not that you even read more than one in every twenty of the articles you're alerted to.

Which is why it should be obvious to any of our readers why they should be using HubMed's RSS feeds of PubMed searches, with their Google Reader, to keep up with the literature. New articles will accumulate and be available to scroll through in the sophisticated and cleanly laid out environs of the Google Reader, when it's convenient for you to read them. Reader will tick off items that you've seen and present to you items that you haven't yet seen, without ever screaming "look at me, look at me right now!"

Update: Since I originally wrote this, PubMed released their major update, introducing their own implementation of RSS saved searches, which looks at least as good as that of HubMed, and takes less effort to set up -- just click the RSS button next to the search box on the search results page.

Literature hack: context search

This is a guest post by Joe Dunckley
Ryan Gregory has just started a new blog: Hackademe. It's a Lifehacker for academics, sharing his tips for scientists who are struggling to cope with all the shiny distractions around them. Here at Journalology towers, we have a whole bunch of these hacks lying around, and I hope we won't be treading on Ryan's toes if we share some of our literature searching and managing tips. (Though Ryan has already discussed reference manager software on his day old blog, so perhaps he has this one covered...)

A very simple one to get us started, then: install context search. Firefox comes with a built-in right-click tool for easy searching of Google for the text on the page that you have highlighted. Context search replaces that search Google tool with a search any-number-of-search-engines tool.

You can add all your own favourite search engines to the menu: just go to the search engine website and use the drop menu in the toolbar search box.

Now you can run through PubMed every unfamiliar gene, disease or researcher you stumble upon while reading, with three easy clicks and no typing. (Warning: novices may find themselves up at two in the morning having followed a long chain of context searches from cell signalling pathways to YouTube videos of snow ploughs on speeding trains.)

Dudes, I don't know how you coped in the olden days of typing your search terms into PubMed, and yet people seriously try to tell me that mankind once worked with "card catalogues" and "interlibrary loan". I'm not buying it, you guys.

Tools to search the literature, and PubReMiner plugin

Recently I came across PubMed PubReMiner, created by Jan Koster. I've been very struck by this tool, which I think is pretty much the best way to search PubMed.

I have previously tried a number of different tools (see my list of Tools to search the literature in the sidebar), and Google Scholar by far outstrips a standard PubMed search due to the use of the PageRank algorithm to pull the most prestigious work to the top of the results. The PageRank algorithm doesn't just look at citations, rather it weights them by how often that referring articles has itself been cited. A citation from a source that is itself heavily cited counts more than one from a source that nobody has ever cited.

I've tried out Kfinder, which takes an abstract or other text as the input, and suggests keywords based on the frequency of occurrence of improbable words. You select keywords, and it returns researchers who match that search in Medline at least twice. Kfinder is quite slow and limited to Medline, but it is intuitive, and a good start to selecting keywords if you haven't had much practice.

PubNet from the Gerstein lab looks really promising. It visualizes the network resulting from a query to Medline. The network to the left, focussed around Howard Ochman and Emmanuel Lerat, clearly shows a network of collaborating colleagues, but I only ran that search because I knew of the network already. It could be useful, but I've not found the time to devote to exploring its possibilities, and it takes a while to generate the visualization at times. If it were quicker and easier to navigate the results, I might use it.

I've only had a quick play with Authoratory, and while the concept is excellent (automatically mining information from the results of PubMed searches), the delivery is lacking. When Deborah Saltman, our Editorial Director for Medicine, tried it she found that she was missing, and the keyword search doesn't take Boolean searches yet. A definite work-in-progress.

e-Biosci is clever in that it accepts any text as input (an abstract, or even a whole manuscript, although it was quite sluggish!) and calculates the concepts contained within. You can add and remove concepts to refine your search, and weight how important they are, and then search using these concepts in Medline abstracts and some full text, including BioMed Central's. The advantage of this approach is that you never need to think about appropriate keywords or search terms; the disadvantage is that some concepts are quite diverse. A good example is that an abstract about physician uncertainty in medical decision-making returned some physics articles near the top! I find that it can return items that you probably wouldn't have found otherwise, and can be very accurate at times.

eTBLAST is one of the big hitters in the field. It runs searches against Medline automatically when given an input of text, much as e-Biosci does, and returns a list of related articles. You can then get list of experts in the field, journals to submit to, the history of publishing in this field and several more features. eTBLAST does all the thinking for you, but it does take its precious time. It can take minutes for the results to be returned, which makes me think that the option to have the results emailed is the only way it will get routinely used.

But, as I said at the top, PubMed PubReMiner is my current favourite. Why? Well, it takes standard PubMed queries, which makes it very easy to start using. It is quick and unfussy, and returns the results in easy-to-read columns: a list of the most common journals in the results, a list of the authors who appear most often, and a list of words that most commonly appear in the abstracts, as well as MeSH terms, affiliations and the publications by year. It is simple, but highly effective.

I liked it so much, that I made a Firefox search plugin for it. After vainly following a tutorial, I found that searchplugins.net has a plugins generator, which I've used to create one for PubReMiner, complete with a logo. It is set to the default of a 1000 abstract limit. You can view the source code, and search for it under PubMed or PubReMiner. You can also install it now.

Does peer review work?

There are now a reasonable numbers of studies from journals such as the BMJ and JAMA on the factors affecting peer review. For example, we know due to a piece of work done by my colleagues that while author-suggested reviewers appear to return reports of equal quality to editor-suggested reviewers, they are significantly kinder to the authors in their recommendations on publication.

One of those authors, Pritt Tamber, regularly makes clear his belief that peer review doesn't work, most recently arguing in a BMJ Rapid Response that "Much of the research conducted at the BMJ [...] showed that there is little or no objective value to the process, yet journals and their editors persist with—and advocate—peer review; their only defence is that "there's nothing better," even though few have tried to find an alternative".

As Pritt notes, one alternative is the system used by Biology Direct, published by BioMed Central. The idea is that authors obtain reviews from three members of the reviewing board. If the author cannot find three members of the board to agree (or to themselves solicit an external review) the manuscript is considered to be rejected. If they can get three reports, then the manuscript will be published, no matter what the reviewers say. The twist is that the comments of the reviewers will be included at the end of the manuscript, as an integral part of the manuscript, and signed by the reviewers. The author can make revisions to the manuscript if they wish or even withdraw it, but equally they can ignore the comments and publish despite them. This is with the knowledge that readers will be able to see the reviewers' dissent. Other alternatives include the community peer review being tried by Philica, PLoS ONE and Nature (Nature's experiment appears to have been unsuccessful, but that is no reason to write-off the idea). More journals, publishers and researchers need to go out on a limb to explore new and better ways to assess and critique scientific research.

Before we go too far with condemning peer review, it is worth remembering that without an evidence base, we won't be able to work out where peer review works, where it doesn't and why, and how to improve it.

Much of the research done into the effects of peer review has been, in my opinion at least, quite superficial. Reading it has really only told me what I knew already from working as an editor.

My wish-list for studies of peer review are:

1. Creating a metric of "on-topicness" that editors can use to assess how relevant a reviewer's expertise is to a piece of research or an aspect of that work. This could be by simple similarity analyses, comparing their PubMed abstracts to the abstract of the submitted manuscript, or by more complicated semantic analyses.
2. Comparing manuscripts that were accepted to those rejected to examine the predictive factors. Some of these have been done, but the analyses always strike me as simplistic. The sample size needs to be greater, and the journals chosen need to not be so highly selective - is it really that interesting to see the factors that influence publication in journals like Nature, The Lancet or NEJM? I really want to see are the factors that affect whether a study is ever published in a reputable journal.
   
3. A side-by-side comparison of published articles with the original submitted version (before any peer review in any journal). This could be done by a paid panel who would be able to spend the time to do an in depth analysis; an alternative would be to invite journal clubs at universities worldwide to analyse manuscripts in this way (a sort-of Seti@Home for journalology). Did peer review noticeably improve the work?
   
4. Examine the fate of articles rejected by journals. Several studies of this nature have been conducted, but they mainly focus only on the journal it is eventually published in and the Impact Factor of the publishing journal. Why not examine whether any changes had been made since rejection? What about whether the rejected work is cited and read? Do a panel and journal clubs agree that the work is now sound, even if it might be uninteresting?
5. Compare the ability of different editors to assess a manuscript and select appropriate reviewers under time pressure, pitted against some of the new semi-automated tools available, like etBLAST. This would be like a peer review Olympiad.
   

It is tough to design and conduct good studies to examine peer review, but editors need to make the effort, else skeptics like Pritt will have a point. Now, just as soon as I have some spare time...