Anonymous Review Matters
An April 2006 JAMA reported big effects of switching to anonymous peer review for presenting at the AHA annual meeting. Science News reported:
The survey focused on some 67,000 research abstracts submitted to the American Heart Association (AHA) between 2000 and 2004. … Beginning in 2002, AHA changed its review process so that authors’ names and affiliations were stripped from abstracts before they were sent out for peer review. … For instance, during 2000 and 2001, abstracts from U.S. authors were 80 percent more likely to be accepted … After blinding, the U.S.-based papers were only 41 percent more likely to be accepted, … Similarly, the share of abstracts from faculty at highly regarded U.S. research universities dropped by about 20 percent, after blinding. For authors in government agencies, the acceptance rate fell by 30 percent.
Clearly reviewers were using author names and affiliations as clues about quality, since their choices changed when they no longer had access to such clues. But this study says nothing about whether this practice was biased, since it didn’t look at any other indicators of paper quality. Hat tip to Rafe Furst.
Norman, I think the main point is that the paper doesn't offer much evidence one way or the other on the main claim of interest. The likelihood of the data observed seems about the same if author info biases evaluations, and if it improves evaluations. So updating our priors on this evidence just gives us our priors back.
Let's get away from what the paper itself said, and whether Robin was or was not criticizing, and as the underlying question. Robin has proposed a definition of paper quality as resource compensated posterity review. I accept this definition. If our goal is to maximize paper quality, so defined, should referees use the authors' names? I take it that we all accept that the JAMA paper shows that using the authors' names affects acceptance and the dispute is whether the effect increases or decreases quality.
The main objection raised by Robin and Douglas is that the JAMA paper hasn't shown that the effect decreases quality. I don't see why they should have to. As I understand it, there is no data currently which established paper quality as defined by the posterity review standard. This means that one's conclusion will turn on one's prior / null hypothesis / presumption. If we start with the presumption that using the author's name does NOT decrease quality, then it is true, as Robin and Douglas point out, that the JAMA study does not prove the contrary. If we start with the presumption that using the author's name DOES decrease quality, then the JAMA study establishing that there is an effect, and if we combine this with our presumption that the effect is negative, then we can conclude that authors' name should not be used.
Why should our presumption be that using the authors' name reduces quality? I have given theoretical reasons to thing that (1) using the authors' name may reduce reduce quality, and that (2) blind reviewing is likely to be well correlated with quality as defined posterity review. No one has explained why my theoretical argument is flawed on either point. Unless someone can undermine my argument, or at least propose a counter-argument as to why using the authors' name is likely to improve quality, I think that it follows that the correct null hypothesis is that the effect of using the authors' name, if any, is to reduce quality. The JAMA study shows that it does have an effect, thereby eliminating the need for the 'if any' caveat. I think my null hypothesis is also the common sense null hypothesis, which is why the JAMA study apparently didn't make it clear.