How to find the right peer review system

7 minute read

Peer review dates back to 1731 and, unsurprisingly, it’s evolved a lot since then. Here’s how to make sure your review system is up to the job.

Since peer review emerged on the medical scene way back in the 18th century (Medical Essays and Observations from the Royal Society of Edinburgh is widely recognised as the world’s first peer-reviewed publication), the system has had its fair share of criticism. But peer review remains widely considered an integral form of scientific evaluation.

A proper peer review system interrogates the academic work submitted to a conference or journal, and ensures that the quality of published research remains high.  It also gives researches an extra opportunity to network with the research community.

If you’re organising a research conference, sourcing a system to help you manage authors, abstracts and reviewers through the peer review process will be a big part of your remit. While there’ll be lots of other event tasks to manage – from promoting your conference to building your programme –  it’s the peer review phase that often takes the greatest effort and attention to detail.

Because reviewing can come in so many different forms, it’s important you source a peer review system that’s up to the task at hand. But first, let’s look at some of the different types of peer review out there.

Common forms of peer review

The role of reviewing is to help conference chairs decide whether to accept or reject papers or abstracts. Up till fairly recently, the most common forms of reviewing were single or double-blind review which, in both cases, mean that authors don’t know who the reviewers are. However, the idea of revealing who the reviewers are (open peer review system) has grown in recent years.

Let’s look at the three styles of review in a little more detail.

Single-blind review

Single-blind review keeps the names of the reviewers anonymous from authors, so it’s strongly dependent upon the goodwill of reviewers. While the reviewers are hidden, the authors’ name and affiliation are on their abstracts or papers. This was the traditional form of peer review and it’s still the most common type we encounter on Ex Ordo.  The validity of research is key during the peer review process and reviewers shouldn’t be influenced by authors, keeping them anonymous means authors can’t contact them. It also means that many reviewers feel more comfortable giving critical reviewers where necessary.

There’s an ongoing discussion around the ethics of single-blind peer review. While reviewers’ identities are kept anonymous, they can still see who the authors are. Reviewers may use their anonymity as justification for being unnecessarily harsh. And single-blind peer review doesn’t protect authors against gender bias, or any sort of bias for that matter.

Like all types of reviews, a case can be made both for and against single-blind. There’s a good reason it’s still the most common reviewing format.

Double-blind review

This model is set up to keep anonymity throughout the peer review process, as both the reviewer and the author are anonymous. Author anonymity limits reviewer bias, whether this is towards an author’s gender, country of origin, academic status or publication history. It also ensures that articles written by prestigious or renowned authors are considered on the basis of their individual merit, rather than their author’s reputation. It is argued that double-blind review performs no worse than single-blind and that it maintains the perception of fairness and equality in the peer review process for conferences and journals.

The biggest argument against double-blind peer review is that it’s not really blind after all. In relatively small fields of research, it can be easy to make an educated guess of who a given author is. Reviewers can often identify authors through their writing style and subject matter, and it’s exceedingly difficult to guarantee total author anonymity. And it’s also not uncommon for authors to self-reference their previous published work.

Ex Ordo website

Open review

Traditional peer review keeps reviewers anonymous. However, in the past 30 years, conference and journal publishers have been testing alternative approaches to bring more transparency to the peer review process. Open peer review is one such approach, and it means that both authors and reviewers know who the other party is.

One of the better aspects of an open review system is it highlights who gives the not-so-constructive reviews. By publicly linking reviewers with their critiques, it may encourage them more thorough and constructive reviews. This benefits authors (especially early-career ones) who may gain a better understanding of how to improve their work.

Having said that, the open reviewing system has been criticised as leading to very tame reviews. Reviewers may not feel free to be as critical as they would be if the review were blind. Similarly, a reviewer may be inclined to give a better review when assessing a colleague’s paper to avoid tension in the workplace. Because of this, reviews can have a positive bias towards researchers who are known to reviewers. (Though if your submission system has a conflict checker it will help you avoid matching reviewers with authors from the same organisation or country.)

Interestingly, BMJ carried out a study which discovered that the quality of reviews didn’t diminish whether an abstract was reviewed openly or blind. What changed was the time taken to complete the review. On average, reviewers took 25 minutes longer to critique an abstract in open peer review than when it was in blind review. The study also found that 55% of reviewers declined to take part in the open review system. So perhaps the question for your conference is: will your reviewers be happy with their identities being public?

Features of a peer review system

Now, to the software that’ll help get through your peer review process in one piece.

At Ex Ordo, we count former conference organisers and published researchers among our ranks: we get that each conference is unique and has different needs for peer review. Having said that, if you’re expecting to receive more than 50 abstracts or papers, I recommend you invest in a decent software package to help you manage them. It doesn’t need to be Ex Ordo, but it does need to be designed for the complexities of peer review.

Easy allocation of submissions to reviewers

Gathering submissions and allocating them to the right reviewers can be a difficult task if you’re using an old clunky abstract management system. Not all systems enable conference organisers to allocate submissions to reviewers easily, so make sure to look for one that does.

The system should be smart enough to match submissions to reviewers using the author’s and reviewer’s selected topics or themes. This prevents reviewers from being allocated a submission within a topic they’re not confident reviewing. It should also incorporate the number of required reviews per submission and enable single or double-blind reviews.

A good peer review system will also allow you to tweak allocations if you need to, before you send submissions and marking schemes to your reviewers.

Flexible marking scheme

The ability to adapt your review process to your conference’s needs is vital. Any peer review system on your shortlist should allow you to create your own marking scheme and allow reviewers to give feedback to authors or chairs.

But an often-overlooked feature is how easy it is for reviewers to review a submission. An over-complicated process is a great way to confuse and annoy reviewers. The best peer review systems are easy for reviewers to follow and, as a result, cut down on the time it takes them to complete each review.

Screenshot of marking scheme on Ex Ordo

Ex Ordo’s one-click marking feature is easy and quick to use.

We’re firmly in the digital age, but don’t be surprised if some of your reviewers prefer to download submissions and mark them offline before uploading them again. And your tool should facilitate this.

Multi-stage reviews

The world of research is rapidly evolving and peer review is too. Multi-stage review has emerged in recent times. If your conference is considering multiple stages of peer review, make sure you’re looking at systems that support this feature. If your conference will need authors to submit corrected versions of their papers or abstracts for a second round of review, you’ll need a system that can support this. Not every peer review system can, so do your homework.

Visibility of progress

Reviewers are busy people who may not be great at answering emails. To help keep you in the loop, your software should give you an overview of each submission’s and reviewer’s progress. This will let you can keep track of what’s where without spamming all your reviewers. And, crucially, it’ll allow you to send a few pointed reminders when you need to.

Screenshot of reviewers progress on Ex Ordo peer review software

Re-allocating submissions

Reviewers are busy people who give up their time for the greater good of the research community. So, brace yourself… It can happen from time-to-time that a reviewer can no longer commit to peer reviewing a paper at your conference.

This kind of speed bump is an easy fix when you have a system that supports re-allocation of submissions. A feature like this partnered with a communication centre can keep you organised by helping you reallocate the submissions and inform affected reviewers of the additional work they’ll be carrying out.

Easy communication

Communication is crucial throughout the peer review process. If you’re an organiser you’ll need to send your call for papers, remind your authors to register and your reviewers to review. A built-in communication tool is a really handy feature in any abstract management system. But particularly so during the reviewing phase when it’s vital you can easily manage and communicate with your reviewers.

Gif of sending a call for abstracts on Ex Ordo

The communication centre in Ex Ordo

Some systems, like Ex Ordo, have a built-in communication centre allowing you to communicate with members of your conference throughout the many stages of the abstract management process. An intuitive feature like this allows for easy management and progress during the review phase. For example, you can send tardy reviewers personalised reminder emails within with a few clicks.  Or you can easily identify if a reviewer hasn’t accepted their invitation to review a paper and reassign it to another reviewer.

Balancing the workload

Speaking of the additional work reviewers will be carrying out, another source of annoyance for reviewers is being over-burdened with submissions. Reseachers review in their spare time (often after their own full-time job), and overloaded reviewers tend to withdraw their offer to review or go silent. When they do, you’ll be left scrambling to replace them.

Thankfully, any peer review system worth its salt will allow you to set the max number of submissions that each reviewer should be allocated.  It should also have a conflict checker to help you avoid matching reviewers with authors from the same organisation or country.

Happy reviewing.

Ex Ordo website

Brian Campbell

Brian helps researchers find conferences on PaperCrowd. He enjoys entertaining his baby nephew, playing an insane amount of sport, and being wildly competitive at party games. (He’s a pass-the-parcel champion.) Previously, he worked as the admin of an international entrepreneurship research conference.