This computer science conference tested single vs double-blind review for reviewer bias. Here’s what they found.
Forests have been felled in the pursuit of documenting the pros and cons of single vs double-blind peer review. Which method better upholds the quality of published research remains an active area of debate, with many high-ranking conferences and journals falling on opposing sides of the argument.
Whereas those who argue the case for single-blind peer review say that, in practice, double-blind often doesn’t really mean double-blind. That it’s frequently apparent who an author is based on citation, subject, or writing style.
What happened when a computer science conference weighed up single vs double-blind
The Web Search and Data Mining conference (WSDM) had historically used single-blind conditions in its rigorous review process, which sees each paper assigned to four specialist reviewers and results in an acceptance rate of just 15.6%.
Two of the co-chairs of the 2017 conference, Andrew Tomkins of Google and Min Zhang from Tsinghua University, were asked to consider switching to double-blind review. Curious about whether double-blind peer review has the power to reduce reviewer bias within their field, they reviewed the literature and discovered that no experiment had been done on single-blind vs double-blind in computer science.
So, with the help of William D. Heavlin (Google), they decided to carry one out.
Examining single vs double-blind peer review
The researchers wanted to examine whether double-blind review had an impact on implicit reviewer bias with respect to an author’s gender, country, prestige and affiliation. So they split their conference reviewers into two cohorts:
- Those who were given access to author information
- Those who weren’t
Both groups of reviewers then bid on papers to signal their interest in reviewing them. Once bidding was complete, the organising committee assigned each WSDM submission two reviewers from each cohort, and the review groups set to work.
When the organisers analysed the review data, here’s what they discovered…
Reviewer bias in single vs double-blind conditions
There were three significant differences in the behaviour of the conference’s single vs double-blind review groups.
1. The reviewers from the single-blind cohort bid for papers less prolifically, bidding on 22% fewer, on average, than those in the double-blind cohort.
2. The bids entered by single-blind reviewers were weighted towards submissions from top universities and top companies, compared with the bids of double-blind reviewers.
3. The single-blind reviewers were relatively more likely to submit a positive review for submissions that had prestigious authors or that came from high-quality organisations than were their double-blind counterparts.
Conferences should “seriously consider” advantages of double-blind review
The experiment found that under single-blind review conditions, reviewers make use of information about authors and their affiliations. And this disadvantages some authors.
In the field of computer science, fewer bids make it more difficult to allocate submissions to the reviewers most suited to review them. So single-blind review (with the resulting fewer bids) may reduce the overall quality of a conference’s review process.
Submissions from researchers at prominent organisations may also benefit from single-blind conditions because an increased number of bids makes them more likely to be allocated to the most appropriate reviewers.
The experiment’s findings also suggest that, in single-blind peer review, a submission written by an eminent author from a prestigious organisation might get a more positive review than the same paper written by an early-career researcher from a relatively unknown organisation.
As a side note, the influence of gender on reviewers’ behaviour under single vs double-blind conditions was not found to be statistically significant in this study. However, the authors performed a meta-review of their findings combined with other reported studies on the effect of gender in peer review. Based on this, they found that “the overall effect against female authors can be considered statistically significant.”
“Clearly, our understanding of the implications of reviewing methodologies remains nascent,” the authors conclude by saying. “Nonetheless, we feel that program and general chairs of conferences should seriously consider the advantages of using double-blind reviewing.”
Read the full paper published in the Proceedings of the National Academy of Sciences of the United States of America.