darb.ketyov.com

Caveat lector: This blog is where I try out new ideas. I will often be wrong, but that's the point.

Home | Personal | Entertainment | Professional | Publications | Blog

Search Archive

18.11.11

Ranking biomedical retractions

This past week I was in Washington, DC for the 30,000+ Society for Neuroscience conference. Recently I wrote about (my interpretations of) the purpose of peer review. Well this prompted a group of us to get together to discuss open science and the future of scientific publishing.

Obviously science and the process of science have been on my mind.

My fellow neuroscientists Jason Snyder and Björn Brembs and I got together for lunch to talk about... lots of stuff, actually... but I came away with a few questions that seemed to have empirical answers.

Before I jump in though, if you haven't seen Björn's talk What's wrong with scholarly publishing today? check out the slides at the end of this post. They're packed with some mind boggling data about the business of peer-review.

At some point during our lunch, Retraction Watch (which is an amazing site), came up, and ultimately inspired two questions:
  • Which journals have the most retractions?
  • Which biomedical fields have the most retractions?
I did a quick-and-dirty check of this using PubMed's API, because they have a nice "search by retraction" option:

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&field=word&term=hasretractionin&retmax=10000&usehistory=y

There was one issue: every article--regardless of scientific field--for the general science journals (Science, Nature, PNAS) are indexed in PubMed. So if an article (or a dozen) about semiconductors (see: Jan Hendrik Schön) was retracted from Science, it would still show up in this analysis. The result was inflated biomedical retraction counts for those journals, so I had to manually adjust counts down by removing non-biomedical retractions (just to put everything on par, since PubMed doesn't index non-biomedical peer-review journals).

Here are the results for the 1922 retractions across 796 journals:
Bradley Voytek retraction counts Science Nature PNAS

PNAS (59 retractions) and Science (52) lead the pack, followed by J Biol Chem (40), J Immunol (33), and Nature (31).

Next I counted words that appeared in the titles of the retracted articles to get a feel for what kinds of papers are being retracted. Here's all words that appear at least 50 times in paper titles:
  • cells (189)
  • activity (154)
  • effects (152)
  • human (148)
  • patients (136)
  • protein (108)
  • factor (104)
  • gene (103)
  • expression (102)
  • receptor (96)
  • study (81)
  • cancer (70)
  • treatment (57)
  • surgery (54)
  • disease (54)
  • DNA (53)
  • virus (50)
(Similar words were grouped using TagCrowd)

At first blush, it looks like cell/molecular/micro biology represents a big chunk of the retractions (cells, protein, factor, gene, expression, receptor, DNA, virus), but human patient research isn't much better off... (human, patients, surgery).

I've heard the argument before (sorry, can't remember where) that fields where the data is more difficult to collect and replicate are more prone to shady research practices... I'm not sure if that's exactly being reflected here, but the exercise was an interesting one.

Thoughts?

12 comments:

  1. Could you try weighting by the total number of papers published by each journal over the same period? The biggest journals seem to have the most retractions, but this could be just because they publish more articles.

    ReplyDelete
  2. Of course you're correct. Sadly what counts as a "citable publication" by which to weight is quite nebulous and can be "adjusted" by negotiation. (See the Brembs talk I embedded.)

    ReplyDelete
  3. I'll go out on a limb there and suggest that "high impact" journals ("glamour mags", some say) are also subject to the "more eyeballs" effect, i.e. more people read it and therefore errors are more likely to be picked up. Conversely the Obscure Journal of Extremely Specialized Area of Science, although it might be perfectly respectable, will receive less post-publication scrutiny.
    Disclaimer: I work as a minion (subeditor) at Nature, but opinions and more or less accurate facts and errors are all mine.

    ReplyDelete
  4. Nico: Definitely! But there's another factor in there: those more eyeballs translate into more prestige which translates into better jobs which leads to looser ethics when publishing due to intense competition within academia for jobs.

    ReplyDelete
  5. Great analysis! It confirms previously published results:

    http://bjoern.brembs.net/comment-n766.html

    In other words: journal rank is a better predictor of retractions than actual citations. No surprise there.

    ReplyDelete
  6. I was told recently that 80% of papers published in Nature and Science have never been replicated. Of course, that could just be a scientific urban legend... will see if I can find a source. Anyway, I wonder if the number of times a given article has been replicated can be used as an index of "reliability" as well?

    ReplyDelete
  7. Thanks Björn! Wanted to take a it *little* further by removing non bio-medical articles and adding the textual analysis.

    T2: How you would measure replication, if not by citation count though?

    ReplyDelete
  8. Yeah, that's the hard part. It'd probably be a labor intensive process going through the forward citations by hand from ISI or something (because just citation count is inflated by people using the finding to justify something tangential or people citing them because they couldn't replicate the finding). Yeah, now that I'm thinking about it that's a bitch to automatically estimate.

    ReplyDelete
  9. Very interesting analysis, Bradley. Notable that PNAS appears to have considerably more retractions than Science or Nature, which wouldn't fit a simple "journal rank" account.

    Any idea what this might be due to? Some ideas from Steve Caplan: http://bit.ly/qXsAHA

    ReplyDelete
  10. It's hard to argue with Steve's logic... any time you can get around any aspect of peer-review, the more likely it is for shoddy science to get published.

    ReplyDelete
  11. Bradley, you also need to look at when these retractions were made. You'll find that the smaller journals are the ones adding to the recent retractions boom in the second half of the last decade. See data here: http://www.nature.com/news/2011/111005/full/478026a/box/2.html

    ReplyDelete
  12. Richard: Great link and excellent points. Interesting there... what happened with Acta Crystallographica E?!

    ReplyDelete