Ranking biomedical retractions

This past week I was in Washington, DC for the 30,000+ Society for Neuroscience conference. Recently I wrote about (my interpretations of) the purpose of peer review. Well this prompted a group of us to get together to discuss open science and the future of scientific publishing.

Obviously science and the process of science have been on my mind.

My fellow neuroscientists Jason Snyder and Björn Brembs and I got together for lunch to talk about... lots of stuff, actually... but I came away with a few questions that seemed to have empirical answers.

Before I jump in though, if you haven't seen Björn's talk What's wrong with scholarly publishing today? check out the slides at the end of this post. They're packed with some mind boggling data about the business of peer-review.

At some point during our lunch, Retraction Watch (which is an amazing site), came up, and ultimately inspired two questions:

• Which journals have the most retractions?
• Which biomedical fields have the most retractions?

I did a quick-and-dirty check of this using PubMed's API, because they have a nice "search by retraction" option:

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&field=word&term=hasretractionin&retmax=10000&usehistory=y

There was one issue: every article--regardless of scientific field--for the general science journals (Science, Nature, PNAS) are indexed in PubMed. So if an article (or a dozen) about semiconductors (see: Jan Hendrik Schön) was retracted from Science, it would still show up in this analysis. The result was inflated biomedical retraction counts for those journals, so I had to manually adjust counts down by removing non-biomedical retractions (just to put everything on par, since PubMed doesn't index non-biomedical peer-review journals).

Here are the results for the 1922 retractions across 796 journals:

PNAS (59 retractions) and Science (52) lead the pack, followed by J Biol Chem (40), J Immunol (33), and Nature (31).

Next I counted words that appeared in the titles of the retracted articles to get a feel for what kinds of papers are being retracted. Here's all words that appear at least 50 times in paper titles:

• cells (189)
• activity (154)
• effects (152)
• human (148)
• patients (136)
• protein (108)
• factor (104)
• gene (103)
• expression (102)
• receptor (96)
• study (81)
• cancer (70)
• treatment (57)
• surgery (54)
• disease (54)
• DNA (53)
• virus (50)

(Similar words were grouped using TagCrowd)

At first blush, it looks like cell/molecular/micro biology represents a big chunk of the retractions (cells, protein, factor, gene, expression, receptor, DNA, virus), but human patient research isn't much better off... (human, patients, surgery).

I've heard the argument before (sorry, can't remember where) that fields where the data is more difficult to collect and replicate are more prone to shady research practices... I'm not sure if that's exactly being reflected here, but the exercise was an interesting one.

Thoughts?

What's wrong with scholarly publishing today? II

View more presentations from Björn Brembs