Peer review is critical to maintaining research quality at conferences. But what are the implications of using single blind vs double blind review?
Peer review is the process of subjecting an author’s research to the scrutiny of experts. Using a peer review process (single blind vs double blind) at your conference allows you to easily evaluate every paper submitted to your event, and accept only the most credible for publication. It also gives researchers the opportunity to improve the quality of their papers before publication, as reviewers will often give recommendations for improvement along with their assessment.
While peer review remains the dominant method for research validation, there are many ways you can facilitate your review process. The two main conditions of peer review are single-blind and double-blind. Forests have been felled in the pursuit of documenting the pros and cons of single blind and double blind peer review. Which method better upholds the quality of published research remains an active area of debate, with many high-ranking conferences and journals falling on opposing sides of the argument.
In this article, I’ll compare both reviewing types and outline their advantages and disadvantages, so you can make up your mind for yourself. I’ll also look at what happened when a computer science conference tested single vs double-blind peer review for reviewer bias.
Ready? Let’s jump in.
Single blind review
The “blind” in single blind review refers to what information authors can see. In a single blind peer review, reviewers’ identities are kept hidden from authors. This is the traditional form of peer review, and it’s still the type that’s most common.
Advantages
The advantage of single blind reviewing is it allows your reviewers to critique papers without any influence being exerted by the authors. If a reviewer knows their name won’t be visible to the author (or to the public), it gives them the freedom to give an honest critique – even when it’s not favourable.
Disadvantages
Even though it’s the most widely used form of peer review, single blind isn’t always looked upon in a positive light in the academic community. For example, reviewers could use the fact that they are anonymous as an opportunity to be unusually critical of authors.
For example, a survey by the Publishing Research Consortium found that whilst 85% of respondents had experienced single-blind review, only 52% described it as effective (and it was the preferred option for only 25%).
Double blind review
In a double blind peer review, the identities of both the authors and reviewers are kept hidden. If you’re collecting submissions for double blind review, you’ll need to ensure they’ve been prepared in a way that hides authors’ identities fully i.e. no repeated citations of their own or colleagues work throughout the submission. So it’s a good idea to make sure that the peer review software you’re using for your conference can facilitate both single and double blind review.
Advantages
While double blind has all the benefits of single blind, it’s got some added ones to boot. Because both authors and reviewers are “blind”, this guards against reviewers being influenced by an author’s prestige. This means that even submissions from the stars in your field are considered on their individual merit rather than on the author’s reputation.
Another benefit of double blind is it reduces the possibility of reviewer bias. Bias can take many forms, for example an author’s gender, country of origin, academic status or previous publication history.
Disadvantages
In practice, however, double blind often doesn’t really mean double blind. It’s likely that your research field is relatively small, so it will often be apparent to reviewers who the author is based on citation, subject, or writing style.
The survey above was more positive in its assessment of double blind reviews (although only 45% of respondents had directly experienced it). Despite this, 71% of respondents described it as effective, and it was the preferred option for 56%.
Still unsure which method is best for your event’s peer review process? Read on to learn about the leading solution in the research conference space.
What happened when a computer science conference tried both
The Web Search and Data Mining conference (WSDM) conference tested single vs double-blind review for reviewer bias. Here’s what they found.
WSDM had historically used single blind conditions in its rigorous review process which results in an acceptance rate of just 15.6%.
Two of the 2017 conference’s co-chairs, Andrew Tomkins of Google and Min Zhang from Tsinghua University, were asked to consider switching to double blind review. Curious about whether review conditions can reduce reviewer bias, they discovered that no experiment had been done on single vs double blind in computer science.
So, with the help of William D. Heavlin, also from Google, they decided to carry one out.
Setting up the experiment
The researchers wanted to examine whether double blind review had an impact on implicit reviewer bias with respect to an author’s gender, country, prestige and affiliation. So they split their conference reviewers into two groups:
Group 1. Those who could access author information
Group 2. Those who had couldn’t
Both groups then bid on papers to signal their interest in reviewing them. Once bidding was complete, the organising committee assigned each submission two reviewers from each cohort.
When the organisers analysed the review data, here’s what they discovered…
Reviewer bias in single vs double blind conditions
There were three significant differences in the behaviour of the conference’s single vs double-blind review groups.
1. The reviewers from the single-blind cohort bid for papers less prolifically, bidding on 22% fewer, on average, than those in the double-blind cohort.
2. The bids entered by single-blind reviewers were weighted towards submissions from top universities and companies, compared with the bids of double-blind reviewers.
3. The single-blind reviewers were relatively more likely to submit a positive review for submissions from prestigious authors or high-quality organisations than were their double-blind counterparts.
Conferences should “seriously consider” advantages of double-blind review
The experiment found that under single blind review conditions, reviewers make use of information about authors and their affiliations. And this disadvantages some authors.
Submissions from researchers at prominent organisations may also benefit from single blind conditions. This is because an increased number of bids makes them likely to be allocated to the most appropriate reviewers.
The findings also suggest that, under single blind conditions, a submission written by an eminent author from a prestigious organisation might get a more positive review than the same paper written by an early-career researcher from an unknown organisation.
“Clearly, our understanding of the implications of reviewing methodologies remains nascent,” the authors said. “Nonetheless, we feel that program and general chairs of conferences should seriously consider the advantages of using double-blind reviewing.”
Read the full paper published in the Proceedings of the National Academy of Sciences of the United States of America.
Conclusion
Peer review is like democracy, the saying goes, despite its flaws, it’s the best system we have.
Whether you’re using single or double blind, make sure your review software can allow you to add additional measures like preventing nepotism based on email domain and organisation. This can help prevent a conflict of interest between the reviewer and author, and when combined with double blind review, can make your conference’s review very strong.
To see how this works in practice, check out our online conference platform.