Since 2013, we’ve been attending IEEE’s Panel of Conference Organizers (POCO) as a preferred partner for IEEE, participating in panels on peer review.
At this year’s POCO in Glasgow, we had the opportunity to lead a group session with IEEE volunteers.
We asked the participants what would be the characteristics of the perfect Peer Review System. Here’s a summary of their vision.
The peer review process is at the heart of the development of a technical programme. A good peer review process ensures the suitability, the quality and the originality of the content being presented at the conference.
At the roots of an efficient peer review process is a relevant allocation of papers to reviewers. The Ex Ordo system, for example, uses a powerful algorithm that ensures submissions are automatically allocated to the most appropriate reviewer. For the organiser, it means no more countless hours spent trying to match papers and reviewers manually.
In addition, the tool should allow Chairs to control the number of papers each reviewer gets in order not to overload them.
But beyond that, a good peer review system must be intuitive and offer an excellent experience to the reviewers. Reviewers should be able to read and score the submissions both online and offline. Abstracts and papers should be printable, along with the marking scheme and IEEE reviewers want to be able to add comments online.
It has been suggested at POCO that a mobile app for reviewers could be a useful tool, and some innovative features have been mentioned, such as speech to text via Google Voice or the ability to mark up the document while reading.
Guaranteeing an amazing experience to reviewers through an easy-to-use interface will help IEEE conference organisers retain reviewers – which is a significant bonus since finding reviewers can be a big challenge.
2. Tailored to IEEE’s workflow
IEEE prides itself on sponsoring high quality conferences. To achieve such a standard of excellence across all the societies, the organisation put in place a set of guidelines and processes that organisers have to follow.
IEEE provides a number of tools to support the entire peer review workflow. A good peer review system must therefore integrate with IEEE’s ecosystem of tools, in particular with:
• PDF eXpress to make sure all the format of each paper complies with the standards
• eCopyright to release author’s copyright to IEEE
• CrossCheck to avoid plagiarism
The peer review system should also work in conjunction with the registration system. That way, the registration system could “talk” to the peer review system and pull the number of accepted papers an author has for example, to allow for an automated calculation of the paper fees.
With a registration system aligned with IEEE concentrated banking, societies would also have the possibility to get an overview and easily monitor the budgets of their conferences across the organisation.
3. Supporting IEEE’s quality standards
There are a few ways a peer review system can support the quality standards of IEEE’s conferences.
First, the keystone of a good content is to collect quality papers. This can be achieved by sending out the Call for Papers to the right persons and by integrating with tools such as Crosscheck to avoid plagiarism and ensure the originality of the content.
Second, these submissions have to be reviewed by appropriate experts. An idea that was discussed at POCO is to have a central repository of reviewers to help conference organisers find reviewers.
The tool should also allow conference organisers to match the peer review process with IEEE’s best practices. For example, they should be able to set up the system so that every submission is reviewed by a minimum of two reviewers, preferably three.
4. In line with IEEE’s vision
At Ex Ordo, we know IEEE is constantly reflecting on the future of peer review and how to ensure the good reputation of IEEE conferences lives on. This means experimenting with innovative initiatives to keep improving the peer review process.
During the group session, IEEE volunteers mentioned some inventive ideas.
One of them was to create a space for anonymous discussions where reviewers could discuss papers, post additional clarification and see other reviewers’ comments.
Another area for innovation is the recognition of reviewers. Over the years, we’ve appreciated how important the work of reviewers is for research and that more often than not, this work is performed anonymously. We believe there might be some solutions to recognise the importance of this work, and IEEE volunteers seemed to agree. They suggested two initiatives.
The first one is for authors to be able to score their reviewers’ comments. These scores could then be used to identify “good reviewers”. Obviously, the peer review system should provide the features to support this process. The second suggestion was to distribute badges recognizing the reviewers’ work, although it’s been highlighted that it will require critical mass before they are respected.
Over the last few years, IEEE conferences have also become more and more international. Despite being a great thing for research, this globalization has also raised certain challenges that will certainly impact paper submission and peer review practices.
Some of these characteristics of the perfect peer review system are decidedly innovative. Including them in a peer review system would first require for IEEE to widely adopt them.
In any case, at Ex Ordo, we believe the perfect peer review system always starts with a killer user experience, flexibility and solid support.
What about you? What’s your opinion on the future of peer review?