Neuroscience

Anonymous Peer Review Means Never Having to Say You're Sorry

The Neurocritic has been feeling reproved this past week, so it's time to post an oldie but goodie about the arbitrary nature of peer review, published in Brain (full-text available free).

Might as well take up gambling.

Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000 123:1964-9.

We aimed to determine the reproducibility of assessments made by independent reviewers of papers submitted for publication to clinical neuroscience journals and abstracts submitted for presentation at clinical neuroscience conferences. We studied two journals in which manuscripts were routinely assessed by two reviewers, and two conferences in which abstracts were routinely scored by multiple reviewers. Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance [kappa = 0.08, 95% confidence interval (CI) -0.04 to -0.20] for 179 consecutive papers submitted to Journal A, and was poor (kappa = 0.28, 0.12 to 0. 40) for 116 papers submitted to Journal B. However, editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection (Journal A, odds ratio = 73, 95% CI = 27 to 200; Journal B, 51, 17 to 155). There was little or no agreement between the reviewers as to the priority (low, medium, or high) for publication (Journal A, kappa = -0.12, 95% CI -0.30 to -0.11; Journal B, kappa = 0.27, 0.01 to 0.53). Abstracts submitted for presentation at the conferences were given a score of 1 (poor) to 6 (excellent) by multiple independent reviewers. For each conference, analysis of variance of the scores given to abstracts revealed that differences between individual abstracts accounted for only 10-20% of the total variance of the scores. Thus, although recommendations made by reviewers have considerable influence on the fate of both papers submitted to journals and abstracts submitted to conferences, agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.

Or one could take up lobbying for open peer review. The new journal, Biology Direct, has already done so.

Koonin EV, Landweber LF, Lipman DJ. A community experiment with fully open and published peer review. Biol Direct. 2006 Jan 31;1(1):1.

. . .

The general view of the current system of peer review of scientific work boils down, more or less, to the tired Churchill quote on democracy: it is the worst system imaginable except for all others. Yet, all publishing scientists are painfully aware of the growing problems of the peer-review system. The crucial feature of the present peer review approach is that it is, predominantly, anonymous, and a reviewer, safe behind the veil of anonymity, has virtually absolute power over the helpless author. And absolute power we know only too well can corrupt absolutely. Hence the biased reviews, the inexplicably delayed perfunctory reviews, the pedantic reviews making a huge deal of minor quibbles, and other kinds of unfair and upsetting reviews that we all dread reading but, unfortunately, may even find ourselves writing. Of course, it is not some sinister villains who produce these obnoxious reviews, it is we peers, we distinguished members of the active scientific community. Indeed it would not be much of an exaggeration to state that all of us, on one occasion and another, have been on the receiving end of peer review abuses, and (almost) any frequent referee has probably been an abuser as well, whether accidentally, intentionally, or at least in the eye of the author.

The Neurocritic is happy to provide a new form of anonymous peer review, free of charge.

- The Lovely Dr. Arina K. Bones, Phd Strikes Again!
First, we had the groundbreaking Implicit Association Tests to identify the alien and the dead among us. Now, Dr. ARINA K. BONES, PhD has presented her latest masterwork Do Social Psychologists Cause Priming Research, or Does Priming Research Cause Social...

- The Voodoo Of Peer Review
Are blogs good or bad for the enterprise of scientific peer review? At present, the system relies on anonymous referees to provide "unbiased" opinions of a paper's (or grant's) merits. For today, the discussion will focus on peer review of papers...

- This Message Has Been Sent To You Because Of Your Eminence In The Field.
More Obsequious Science Spam... This one's from Bentham Open. Maybe you or one of your friends have gotten a similar pitch. First, the story circa early 2007. The Open Neuroscience Journal Dear Scientist. Bentham Science Publishers have gained...

- Peer Review Trial And Debate At Nature
Omni Brain links to a new feature at Nature: a 3 month trial period of "open" peer review. For those who opt to submit their papers under this track, authors can choose to have their submissions posted on a preprint server for open comments, in parallel...

- Peer Review: Solemn Duty Or Merely Extra Work?
The other day on Twitter I saw Rolf Zwaan tweeting about Collabra's policy of rewarding reviewers with credit that can be traded in for credit for organisational article processing fees (APC) budgets, into Collabra's fee waiver account, or for...

Neuroscience