Prestigious scientists can play this role even with anonymous review. They can publicize their support of a new and radical idea through talks at conferences, word of mouth, and other informal channels (like blogging). This will increase attention to their new idea, especially since many scientists like to keep track of what the top researchers in their field are doing. And if other scientists give extra weight to the views of their high status colleagues, they will also raise their probability estimates for the hypothesis. So the new idea can receive a boost before any paper is published. Then when papers supporting the radical new hypothesis are submitted for anonymous review, they will have a better chance of being published whether or not the prestigious scientist is an author.
This is an interesting and important point - the sort of thing where once you hear it, you realize all the evidence pointing in that direction that your mind just ignored because of the cached proverb "old scientists resist progress".
I wonder if there's some selection bias inherent in the studies presented here. Assuming that it has been established that older scientists are more willing to accept new controversial hypotheses than younger scientists, has it also been established that they differentially accept good new controversial hypotheses? What I see here is that they tended to embrace the big paradigm shifts relatively early, but it doesn't say anything about older scientists' tendencies to embrace controversial hypotheses that ended up later being discredited. Specifically, Linus Pauling's obsession with Vitamin C megadosing later in life springs to mind.
Reviewers are likely to have a hard time shooting down the work of anyone they know personally, and in specialized sciences, the probability that a paper was written by a friend or former colleague of the reviewer is high enough to be a problem. On the other hand, credibility does matter; if an unestablished author observes something strange, it's likely that he's made a mistake, but an old hand making the same observation should not be ignored. Perhaps instead of a name, reviewers should be given an abstract indication of the author's credibility, such as the author's faculty title, or the number of times they've published before.
Reviewers are likely to have a hard time shooting down the work of anyone they know personally, and in specialized sciences, the probability that a paper was written by a friend or former colleague of the reviewer is high enough to be a problem.
On the other hand, people have different, and often recognizable writing styles. If you're in a specialized science with a small enough circle, then you may be able to recognize a paper written by your friend (or your foe!) even though it was anonymized.
Or maybe you may even know that your friend (or foe?) is working on a paper on a specific topic just from reading their blog/Facebook/Twitter/etc.
I'd be interested in how exactly something like that would work, let's be careful not to make the object of science publishing papers.
I'm not sure how helpful this sort of meta level second guessing of the minds of others is. Reviewers should do their best to review the merits of the paper, regardless of what they know or guess about author. Any epistemic benefit you could derive from considering the author's identity is washed out by the huge risk that you're simply using that as an excuse to make the conclusions you wanted to make in the first place.
This is why we disdain ad hominem arguments. Not because they are invalid per se, but because we know that all too often they only serve as an excuse not to think, not to consider all the evidence.
On the other hand, maybe if papers weren't anonymous, those with prestige could easily be given too much deference. How much Protection From Editors do they actually deserve?
Exactly. It seems unlikely that prestigious researchers will be unable to publish their brilliant but unconventional idea because they can't fully utilize their fame to sway editors. In fact, prestigious researchers have exactly what is needed to ensure their idea will take hold if it has merit: job security. They have plenty of time to nurture and develop their idea until it is accepted.
The title of the post is "Does blind review slow down science?", not "Does blind review stop science?". The prestigious researchers may have the time, but there are plenty of members of humanity that don't. Science is slow enough as it is. We would be well advised to consider any factors that may speed up progress.
There seems to be to assumptions that need to be correct for blind review to be detrimental: Both A) Older established scientists are more likely to be correct when they are positing an anti-establishment thinking explanantion than a younger, less established scientist, and B) those scientists are nonetheless no more capable of marshaling the required set of arguments to do so when faced with blind review than that younger scientist.
I have no issue with A), but B) seems to me to be supremely unlikely - the very factors of an established pattern of rigor that make it more likely that an older scientist may be a safer bet when he breaks from the establishment than I am, also would appear to make it more likely that he or she can establish the case without relying upon reputation.
I might be wrong, but I wouldn't have to fake surprise at learning I was.
I think your argument, on the other hand, relies on the assumption that the reviewers are objective and rational. In an ideal world, they would be. In practice, however, just seeing a conclusion they don't agree with may be enough that they won't evaluate the argument wholely on its merits.
As a somewhat over-the-top example: suppose you were a reviewer in a popular biology journal which, due to its popularity, also got lots of low-quality submissions. Now you are told to review an article whose author you don't know, but which argues that our common conception of how evolution works is actually really flawed. You know that lots of deluded creationists bombard the journal with submissions arguing points like these, and you have had the misfortune of personally reviewing several before this. (They were all completely worthless.)
Then an alternate universe where the situation is exactly the same, but the journal hasn't implemented a policy for double-blind reviews. You are told the name of the author, which is Richard Dawkins.
In which scenario do you think you'll give the submission a more fair hearing?
First I should state that I disagree with anonymous review for the same reasons that I disagree with an unaccountable judiciary - the negative effects on responsibility.
However, there are several problems with the theory in this essay - the most important being that the editors know who the writer or researcher is and can decide to go ahead and publish on that score no matter what the reviewers say. The editors have a strong incentive to advance novel but true theories in that it will advance the reputation of the journal.
If reviewers anonymity is abolished, how many would dare to submit a negative review, knowing that eventually their own work might be refereed by the author?
This is a bit off-topic so I'm not going into any detail here, but you might check out this book by Max Boot, "Out of Order: Arrogance, Corruption, & Incompetence on the Bench", a large proportion of the problems he wrote about arose from judges not being personally responible for their actions on the bench.
Also, more generally, I am a libertarian largely because I believe that everyone is totally and completely responsible for their own actions. Even if someone is holding a gun to your head, you decide what you do in response (and are responsible for letting yourself get in that position). Or if you are drunk or drugged, you are responsible for putting yourself in that position and therefore for what you do while that way.
I mean that people should bear some part of the forseeable costs of their actions. I say "some part" because the actions of others also influence costs, and stress "foreseeable" because in any complex system things interact to such an extent that only very direct results can actually be attributed reliably to any one party. Most attributions of "fault" in complex systems is scapegoating or motivated by interpersonal status games.
I don't understand your objection to anonymous review on the basis of accountability. Doesn't "anonymous review" in this context just mean that the reviewers don't know the authors and affiliations of the papers they're reviewing? In that case, what is there to be accountable for? The reviewers themselves aren't any more anonymous in "anonymous review" than in standard review, are they?
Doesn't "anonymous review" in this context just mean that the reviewers don't know the authors and affiliations of the papers they're reviewing?
In this context, yes, that's the only thing it means.
Maybe I was wrong about that, but I also understood it to mean that the reviewer was also unknown to the author, even after the review. I have heard several stories (can't remember the sources; possibly only urban-scientific legends) of reviewers giving poor reviews of work that could have pre-empted things they were currently working on. And similar self-serving tactics.
Previously, Robin Hanson pointed out that even if implementing anonymous peer review has an effect on the acceptance rate of different papers, this doesn't necessarily tell us the previous practice was biased. Yesterday, I ran across an interesting passage suggesting one way that anonymous review might actually be harmful:
(From Richard A. Posner, "Aging and Old Age")
If this hypothesis holds (and Posner admits it hasn't been tested, at least at the time of writing), then blind review may actually slow down the acceptance of theories which are radical but true. Looking up the Peter Messeri reference gave me the article "Age Differences in the Reception of New Scientific Theories: The Case of Plate Tectonics Theory". It notes:
In this light, an older scientist's acceptance of a new, radical hypothesis should tell us to give the new hypothesis extra weight. It might even be appropriate to apply a heavier "reputation weighting" on controversial theories than established ones - we'd expect the established scientist to write papers supporting the established theories, but not controversial ones. However, blind review makes this impossible. The reviewers may even mistake the established scientist as an undiscriminating free thinker, who endorses any controversial theories simply because they're unpopular. This will slow down the acceptance of hypotheses which are, in fact, correct.
A possible objection to this could be that old scientists are unlikely to change their minds, and therefore old scientists not getting enough credit for their achievements won't have much of an effect in the spread of new hypotheses. (After all, if no old scientist endorses controversial hypotheses, then it doesn't matter if those endorsements aren't properly weighted in review.) Not so. Messeri finds that science does actually progress a lot faster than "one funeral at a time" (as Max Planck put it), and old scientists are ready to adopt new theories given sufficient evidence. While the last holdouts for outdated theories do tend to be the old, their increased security also makes them the first who are willing to publicly support new theories: