Organiser Time Traps #2 – Peer Review

3 minutes read

Picture the scene – you’re a conference chair/organiser and you’ve just closed off your call for papers. You’ve received 300 submissions, even more than you anticipated. You decided early on that each abstract would be reviewed by 3 people, so you have now 900 reviews to be carried out.  So how do you allocate 900 abstracts to your reviewers?

Manual Allocation                                             

If the conference is run manually, you probably recorded the abstract details (titles, abstract, author info etc) in an Excel file. You’ll need a second Excel file to manage your reviewers. When you email your reviewers asking them to participate you’ll need to monitor and record their response so you know how many reviewers you will have. You should also ask them to indicate their area of expertise so that you can factor this into the allocation.

One way to do the allocation manually is to print off all the abstracts and lay them out on the floor of your office (if it’s large enough!). If you’ve decided that each reviewer should receive a maximum of 10 reviews, write each reviewer’s name on 10 individual sticky notes. Then scan through each abstract and using the sticky notes attach the 3 most appropriate reviewers to each one. Sounds like a lot of work? It becomes even more complicated when you have to:

  • factor in each reviewer’s area of expertise so that you can allocate them the most appropriate abstracts.
  • spread the abstract submissions evenly among your reviewers.
  • prevent reviewers scoring abstracts from their own university.

Stress of peer review

 Image Courtesy of: Wealth Artisan

you’re happy with your allocation you’ll need to email the reviewers their allocated abstracts (making sure that author details are not included). Reviewers will need a scoring system, so you’ll probably create this in Word/Excel and attach it to the email. As there is no way to monitor the review process, you’ll have to check in with the reviewers they are on track. This can take up a lot of time and can sometimes agitate reviewers.  As each reviewer completes his/her reviews and emails them back, you’ll have to record each one in the reviewer Excel file.

So, how much time will all this take? Printing off 300 abstracts, laying them out and writing out your sticky notes will take a couple of hours. Scanning each abstract, finding 3 appropriate reviewers for it, emailing the abstract to the reviewers, monitoring progress and recording their review will take 3-5 minutes per abstract. Say on average that each abstract takes 4 minutes of admin work, which works out at 1200 minutes or 20 hours for the 300 abstracts.

Admin Time: 4 mins x 300 abstracts = 1,200 minutes.

That’s 20 hours of mundane, uninspiring admin work in Peer Review stage.

Online Allocation

So how does an online system fare in comparison? You can get a detailed comparison of time savings with the Ex Ordo Interactive Calculator. With an online system, you should be able to send a personalised invitation to your reviewers in 10 minutes. Building an online marking scheme will take about 20 minutes. An online system can do a best-fit match of reviewers to abstracts in a few seconds. You then can scan the allocation and make any changes you need – so allocation will take approx 30 minutes. In total, setup should take about 1 hour.

Everything else should be automated. The reviewer confirms participation and nominates their topics of expertise online, abstract submissions are matched to reviewers with one click of a button, and the grades are submitted online.  In effect, an online system will save you about 2 days in peer review aspect alone. Plus it’s friendlier to the environment as you eliminate the need to print off hundreds of abstracts and sticky notes.

Between peer review and abstract submission, we’ve shown how a chair/organiser can save nearly 50 hours of unnecessary admin time.

In the next post, we will look at time trap #3 – deciding which abstracts to accept and reject and notifying the authors.