Caveat lector: This blog is where I try out new ideas. I will often be wrong, but that's the point.

Home | Personal | Entertainment | Professional | Publications | Blog

Search Archive


Something ghoti with science citations

Science has a lot of problems. Or rather, scientometrics has a lot of problems. Scientific careers are built off the publish or perish foundation of citation counts. Journals are ranked by impact factors. There are serious problems with this system, and many ideas have been offered on how to change it, but so far little has actually been affected. Many journals, including the PLoS and Frontiers series, are making efforts to bring about change, but they are mostly taking a social tactic: ranking and commenting on articles.

I believe these methods are treating the symptom, not the problem.

Bradley Voytek drunk ghoti

Publish or perish reigns because our work needs to be cited for we scientists to gain recognition. Impact factors are based on these citation counts. Professorships are given and tenure awarded to this who publish in high-ranking journals. However citations are biased, and critical citations are often simply ignored.

Bear with me here for a minute. How do you spell "fish"? g-h-o-t-i: "g-h" sounds like "f", as in "laugh". "o" sounds like "i", as in "women". "t-i" sounds like "sh", as in "scientific citations". This little linguistic quirk is often (incorrectly) attributed to George Bernard Shaw; it's used to highlight the strange and inconsistent pronunciations found in English. English spelling is selective. You can find many spelling examples that look strange, but support your spelling argument.

Just like scientific citations.

Bradley Voytek scientometrics

There are a lot of strange things in the peer-reviewed scientific literature. Currently, PubMed contains more than 18 million peer-reviewed articles with approximately 40,000-50,000 more added monthly. Navigating this literature is a crazy mess. When we created brainSCANr, our goal was to simplify complex neuroscience data. But now we want to shoot for more.

At best, as scientists we have to be highly selective about what studies we cite in our papers because many journals limit our bibliographies to 30-50 references. At worst, we're very biased and selectively myopic. On the flip side, across these 18+ million PubMed articles, a scientist can probably find at least one peer-reviewed manuscript that supports any given statement no matter how ridiculous. Don't believe me? Here's my first whack at a questionable series of statements supported by peer-reviewed literature:

Human vision extends into the ultraviolet frequency range1, possibly mediated by an endogenous violet receptor2.


The effects of retroactive prayer are well-described in improving patient outcomes1. Herein we examine the hypothesis that such retroactive healing is mediated by an innate human ability for "psi"; that is, for distance healing mediated by well known quantum effects2.

What we need is a way to quickly assess the strength of support of a statement, not an authors' biased account of the literature. By changing the way we cite support for our statements within our manuscripts, we can begin to address problems with impact factors, publish or perish, and other scientometric downfalls.

brainSCANr is but a first step in what we hope will be a larger project to address what we believe is the core issue with scientific publishing: manuscript citation methods.

We argue that, by extending the methods we present in brainSCANr to find relationships between topics, we can adopt an entirely new citation method. Rather than citing only a few articles to support any given statement made in a manuscript, we can create a link to the entire corpus of scientific research that supports that statement. Instead of a superscript number indicating a specific citation within a manuscript, any statement requiring support would be associated with a superscript number that represents the strength of support that statement has based upon the entire literature.

For example, "working memory processes are supported by the prefrontal cortex"0.00674, gets strong support, and a link to PubMed showing those articles that support that statement. Another statement, "prefrontal cortex supports breathing"0.00033, also gets a link, but notice how much smaller that number is? It has far less scientific support. (The method for extracting these numbers uses a simple co-occurrence algorithm outlined in the brainSCANr paper).

My citation method removes citation biases. It provides the reader a quick indication of how well-supported an argument is. If I'm reading this paper and I see a large number, I might not bother to look it up as the scientific consensus is relatively strong. But if I see an author make a statement with a low number--that is, a weak scientific consensus--then I might want to be a bit more skeptical about what follows.

We live in a world where the entirety of scientific knowledge is easily available to us. Why aren't we leveraging these data in our effort to uncover truth? Why are we limiting ourselves to a method of citations that has not substantially changed since the invention of the book? My method may have flaws, but it much harder to game than the current citation biases that only give us the narrowest slice of scientific support. My citation method entirely shifts the endeavor of science from numbers and rankings of journals and authors (a weak system for science, to say the least!) to a system wherein research is about making statements about truth. Which is what science should be.


. (2006). The Impact Factor Game PLoS Medicine, 3 (6) DOI: 10.1371/journal.pmed.0030291
(2010). How to improve the use of metrics Nature, 465 (7300), 870-872 DOI: 10.1038/465870a
Robinson KA, & Goodman SN (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of Internal Medicine, 154 (1), 50-5 PMID: 21200038


  1. Thank you for this post Bradley! Very interesting. This might not replace citation but can be an important indicator as to how trendy the topic is or how many studies are done on the subject.

    On a related note, I created a new site that lets you search for on-going clinical trials by location or by keyword: http://clismap.appspot.com

    I'm now thinking of making another site that would show the impact of a completed trial. It would be based on PubMed article that are linked to a clinical trial and how many citations did it get. Please contact me if you are interested in collaborating with me on this project. This idea came to me after I read this article in Nature Medicine: Cancer drugs should add months, not weeks, say experts.

  2. Anonymous14:02

    Where are your 0.05 and 0.00001 coming from? If the whole idea is to remove bias, then your numbers would have to be unbiased. I'm not sure how you're going to get an unbiased number from a meta-analysis of the vaccine/autism link.

  3. Sarkis: That's very cool! And thanks for the offer, but seriously I'm spread *so* thin right now with the research that's paying my bills that I don't have time for any other collaborations right now. :/

  4. Anonymous: I've made an edit. That's totally going to detract from my point... of course that link is spurious, which is what I was trying to say. I've updated the post with a more simple example, as well as a link to the method. Thanks.

  5. Thanks for this interesting post. You are right that there's a lot of arbitrariness about citations at present. And we are all familiar with the reviewer who trys to get you to cite their tangentially relevant paper. I've even had a journal editor tell me that it is their policy that authors must cite at least N papers from that journal: I was so shocked by this, I can't remember which journal. Certainly one desperate to get its impact factor up.
    Your solution is a creative one but I don't think it'll work. As Sarkis points out, it would be more a measure of trendiness than solidity of results. So, I can see you could get articles that were about PFC and working memory, but how could an automated system check whether they had confirmed the link in a methodologically sound way? I'd be interested to see how it would work in my area. Take this example. The question of whether there's a link between auditory difficulties and dyslexia is much researched and very controversial; some people convinced, others think it's a red herring. I have had papers to review where the author starts out with a statement to the effect that we all know dyslexia is caused by auditory problems, and they follow with a sheaf of supporitng references. They ignore the numerous papers that question or moderate this conclusion. Would your method be able to fix that? Or would it just pick up that there were loads of papers on the topic? It's complicated because most papers don't have black and white answers: they may find a weak association, an association with just one measure, or just in a subset etc.
    As a reviewer, my recommendation in this case is to point the authors to an authoratitive review by someone who has evaluated rather than just counted the papers, and suggest that they cite that, and moderate what they say to be more balanced.

  6. Dr. Bishop: Thank you for your comments! Sadly, I agree, there are many problems with the method I've discussed, but I'm mainly just trying to think about other ways of addressing what I believe is a pretty serious problem for modern science. The fact that the journal wanted you to cite their papers specifically is ridiculous. That's a horrible indicator of where we are at.

    I also agree that, in a lot of ways, authoritative and thorough reviews are the best indicator of the status of a given field or scientific argument. However they could are often biased.

    The motivation behind my "solution" was to remove specific citations and allow for linking directly to an entire scientific corpus. In the process, it would be possible, perhaps, to also get a numerical judge of the amount of knowledge within a domain.

    So, for the example you gave above, there's not a consensus by any means, but there is a lot of knowledge about the relationship between audition and dyslexia.

  7. Perhaps it would be more effective to combine the automated brainSCANr value with a crowd-sourced peer review score coming from a number of validated reviewers. Reviewers would attain validation by referral, recommendation, and by getting good reviewer reviews (think YELP for peer review).

    One idea in progress open for improvement:

    I wonder how brainSCANr would do with a novel finding. Tomorrow Brad makes a marvelous discovery that stimulation of the PFC with a 10Hz pulse gives subjects the ability to walk through walls (work with me here). PFC and walking through walls is not well linked or supported, but the data are sound and the study is well received. People begin walking through walls a lot but brainSCANr gives Brad's paper a low review score simply because of the novelty of it's brilliance.

    In many cases that's not how science progresses (unless you're Ramachandran) but it would cool for the algorithm to pull many disparate links together and suggest areas of research based on the intersection of different ideas that together form a breakthrough via their gestalt.

  8. Bryan: I guess for my groundbreaking PFC-stimulation phasing work, the low priority would still flag others to be cautious about the truth of the relationship. If the relationship hadn't yet been replicated, we should still doubt it.

  9. What I've been arguing for is for annotation of citations themselves with a certain character, for example, A extends the conclusions of B or B provides a counterexample to A. This is really what's needed for any automated approaches to be able to do more than just present a popularity metric, but I do think that reputation (to weight popularity) is possible now, given the availability of attention metrics as well as citation metrics.

    1. William: I don't quite understand how the system you're advocating would work. This a long-term mission of yours over at Mendeley? :D