darb.ketyov.com

Caveat lector: This blog is where I try out new ideas. I will often be wrong, but that's the point.

Home | Personal | Entertainment | Professional | Publications | Blog

Search Archive

22.9.11

"Peer-review" does not equal "publisher-owned journal"

Yesterday was my first real day working at UCSF. Most of it was spent filling out paperwork and completing all of the regulatory training.

For those of you unfamiliar with the regulations required for human research, let me just say that they are legion, for they are many. In order to work with human subjects you have to (among other requirements) complete several online courses and questionnaires.

This is usually all well and good if not a bit annoying... (Yes, I know I shouldn't inject my subjects with radioactive spider venom without their consent. No, I didn't consider that radioactive spider venom would be an Investigational New Drug. Yes, I disclosed my consultancy income from OSCORP.)

Anyway, many universities such as UCSF, Berkeley, etc. require researchers who work with human subjects to complete their online training on a specific website: the Collaborative Institutional Training Initiative.

For example I had to complete the "CITI Good Clinical Practice", "Human Subjects Research", and the "Responsible Conduct of Research" curricula. While answering questions in the latter, I encountered the following (which I got "wrong"):
(Someone needs to "peer-review" the inconsistant capitalization in the headers and that superfluous comma in that "Comment".)

This question struck me as especially odd. "Why," I asked, "is this unscientific, unsubstantiated question here? Are they afraid that researchers will stop publishing their research in peer-review and just BLOG everything?"

Does "those who have an interest in the work" not include peers? Have the authors not heard of arXiv.org?

Of course "BLOGS" aren't a replacement for peer-review. Blogs can be peer-reviewed, though, and "peer-review" is not equivalent to a publisher-owned journal. Where does the line between "blog" end and an online journal with commenting begin?

As Bora said over at the SciAm blogs:
Blog is software. Blog is primarily a platform. It is a piece of software that makes publishing cheap, fast and easy. What one does with that platform is up to each individual person or organization.

He points out the open science approach by Rosie Redfield as an example of peer-review that can be moved onto a blog to some extent.

Blogs may not replace peer-review, but they are certainly part of it already.

As was pointed out in a Nature News piece:
To many researchers, such rapid response is all to the good, because it weeds out sloppy work faster. "When some of these things sit around in the scientific literature for a long time, they can do damage: they can influence what people work on, they can influence whole fields," says [David] Goldstein.

Personally, I use my blog for many things. I talk about my own research a lot.

But I also post some fun, off-the-cuff ideas and analyses that, were I more motivated and had infinite time, I could probably polish and write up for peer-review.

It doesn't make those ideas less valuable, just more rough. And I hope someone takes some of them and runs with them. But of course, a major issue here is one of idea attribution.

As Ben Goldacre said in his Correspondence to Nature:
The growth of blogs, Twitter and free online access have caused a welcome explosion in scientific content. But this is atomized and interconnected by a hotchpotch of linking and referencing conventions. If we are going to harness its true value, we shall need dedicated librarians and information scientists to find ways of automating the process of linking content together again. That in itself would be a transgressive scientific innovation.

The current academic reward structures don't give me anything for this blog. So in the meantime I'll continue doing research with people and publishing it in "real" peer review while rolling my eyes at the occasional awkwardness with which academia approaches technology.

ResearchBlogging.orgGoldacre, B. (2011). Harnessing value of dispersed critiques Nature, 470 (7333), 175-175 DOI: 10.1038/470175b
Mandavilli, A. (2011). Peer review: Trial by Twitter Nature, 469 (7330), 286-287 DOI: 10.1038/469286a

9 comments:

  1. I was struck by the title of this post because of a recent piece I read in which two big journal publishers - Wiley and Elsevier - defended their business model: http://theconversation.edu.au/open-access-and-academic-journals-the-publishers-respond-2804

    What was so odd about their defense is that they kept conflating peer-review with publisher-owned. Open access publishers like PLOS and Frontiers follow the same rigorous peer-review system, but they let the authors retain the copyright... Well, the authors pay to retain the copyright. In some ways, they represent a middle ground between informal blogs and publisher-owned/expensive-to-access journal articles.

    ReplyDelete
  2. I find the "fair and rigorous" statement in the CITI test particularly interesting. I feel like the concept of "peer review" isn't just conflated with "publisher owned" but also with the notion that the status quo is flawless. We both know that there's a strong asymmetrical bias in our field, with the "peer review" being abused by people to either protect their own work by torpedoing competitors or to hijack the topic of the research being evaluated in order to accomodate the reviewer's own personal biases.

    While I'm not saying that the comment section of "BLOGS" (why must we yell it?) is necessarily a better alternative, it's been my experience that unless both sides are completely anonymous the current form of the peer review system will continue to be flawed and inefficient. I think the model of PLoS or Frontiers is a good start toward improving the "fair" part while maintaining the "rigorous" part.

    Side Note: I think that the CITI tests are like putting lipstick on a pig to deal with the issue of ethics in science... makes the administrators feel better but we don't really learn anything we didn't already know. But then again that's just me.

    ReplyDelete
  3. I think their question is basically a symptom of current closed "academic publishing" system - which slows down the progress. I can give many examples I experienced why closing data and scientific reports are big obstacles over progress in science. Especially in neuroscience... All that "valuable" data are hindering progress just because the data recorders want to be authors in a peer reviewed journal!

    Do these guys know that a CMU math professor wrote a well-established paper by relying on thousands of blog comments? Here is that paper? http://www.cs.cmu.edu/~odonnell/papers/dhj.pdf

    The author's name in this paper was a pseudoname (Dr. Polymath) representing the collective intellect (many math minded internet users)...

    Here is a nice video by Michael Nielsen talking about open science project - I had heard all about it from that video...

    And please please keep blogging for gods sakes! :)

    ReplyDelete
  4. Dan: Exactly! It's a strange conflation, and I'm beginning to believe it's not an accidental one.

    Tim: Nice catch. That didn't even register with me. And yes... the CITI tests are generally busy work. BUT, that said, I'd rather live in a society where human subjects testing restrictions are such that we have to jump through these bureaucratic hoops than one where they're less strict.

    It's cool that both Dan and Tim highlighted PLoS and Frontiers.

    togliatti: I'd totally forgotten about that paper! Very cool. And yes, I will (almost certainly) keep blogging. If nothing else, I think it helps my writing and keeps me interested in my field.

    ReplyDelete
  5. It's not nearly as serious a threat, and I still prefer to read things on paper, but I feel like many published journals are showing the same stubbornness as newspapers. It'll be interesting what the next ten years does to them (published journals), especially with the sheer availability of information that online databases have been able to achieve. What will be interesting is if, say, blogs that are concerned with peer-reviewed research start popping up on those databases, or on Google scholar.
    Heck, discussing your research on your blog prior to attempting publishing can even void your chances of final approval, even though you aren't even supporting a competing journal, but just that they are denied exclusive hold over that data.

    ReplyDelete
  6. Wow, considering the time lag between completing a study, submitting a manuscript for publication, and that manuscript actually being published in the journal is so great, it seems encouraging more rapid communication, such as blogging, is in the best interest of science. Some journals seem to view blogging as kind of the same thing - I can't remember which journal this was, but in their Author Instructions, they mention that posting results on a blog could be considered prior publication that would render the manuscript ineligible for their journal.

    It seems at the very least, this question is up for debate, and shouldn't be considered an objective question for CITI to include (though this isn't the first poor question I've seen on the CITI course - and it likely won't be the last).

    ReplyDelete
  7. Austin: Yeah, I've wondered about whether or not discussing my work on my blog prior to publication would violate the policies in place at certain journals about prior publication...

    Sara: Well there you go. That addresses that question. And I agree, the role of blogs in the scientific process is certainly open for debate, and not anywhere near as cut-and-dry as the CITI question portrays it.

    ReplyDelete
  8. I think that question is very inappropriate; smacks of old scientists brainwashing the young into supporting the status quo! Really irritated me, so to work it out I blogged it: http://wp.me/ph4jF-hn

    ReplyDelete
  9. Exelent information Mr. Bradley, Thank you for Posting..

    ReplyDelete