OpenUP Pilot1: Open Peer Review for Conferences

An important aspect of the process of doing science is peer reviewing. It is one of the most important tools for quality assurance of the scientific method. The traditional approach to peer review is single- or double blinded reviewing done by a small group of appointed reviewers.

Despite its relevance, traditional peer reviewing has three major drawbacks:

1. It does not scale well. The number of submitted papers at both journals and conferences is steadily rising while the number of expert reviewers is not increasing proportionally.
2. Double-blindness is often compromised as reviewers have a very good overview of the participating research groups and can often infer authorship by language and topics.
3. Compromised double-blindness or single-blindness can introduce a lot of bias in the evaluation of research. Additionally, unfair evaluation can lead to an enforcement of predominant ideas and hinder progress.

Open peer reviewing (OPR) is an alternative, completely transparent approach, which can help solving these challenges. There are multiple ways to open up the review process:

• Open Identity: Authors and reviewers are aware of each other‘s identity
• Open Participation: A larger community is involved in the reviews
• Open pre-review: early versions of material are public before the review
• Open Report: Review report is published alongside the publication
• Open final-version comments: commenting online possible after the verdict

In OpenUP, we are testing the practicability and impact of OPR at conferences. The first venue for Pilot 1 was the Second European Machine Vision Forum 2017 (EMVA 2017). The conference organisers agreed to test the four OPR principles “Open Identity”, “Open Participation”, “Open Report” and “Open final-version comments” in the poster submission system.

The most important changes to the traditional workflow of submission, reviewing and voting include:

• All participants can see all submissions after the submission deadline and can discuss the submission with the authors
• All identities connected to comments and submissions are visible to all participants
• For the final voting, all participants have four votes; project committee members get 10 votes.
• The final result based on the sum of all votes from all members was accepted as a final ruling
• All discussions can continue in the CMS interface after the conference has finished

The software to manage the submission, reviewing and voting of potential contributions is called a conference management software (CMS). We had to find a CMS solution that supports the specific mix of new OPR features needed for Pilot 1.

As a first step, the existing landscape of open source CMS was catalogued and the best fit for adapting to the Pilot 1 specifications selected: HotCRP. After this, an adapted CMS based on HotCRP was developed and released as open source (see in GitHub).

The updated CMS was maintained and monitored by the Pilot 1 team to help with the transition to an OPR system. The support for all phases of the review process (submission, review, voting, conference, post-conference) worked fine. In total 78 users registered to the CMS: all persons who submitted a paper/poster through the system (13), the program committee members (8), the PC (1), and additional 56 people who registered after the voting was already over. However, only the submitting authors, the program committee and PC (22 in total) used the CMS during the peer review and voting phase. In general, the stakes were not high for this OPR test phase: In total, there were 13 submissions of which 10 were chosen to be presentations. The remaining 3 still participated as posters.

A dedicated experience survey was handed out to the audience to evaluate the acceptance and impact of OPR. In most of the questions the participants were asked to judge their view on a scale from 0 (Strongly Disagree) to 5 (Strongly Agree). The Pilot 1 team collected 19 completed experience surveys. Only 10% of the survey respondents were women.

In addition, the team gathered personal feedback from the CMS users. Many indicated that they liked the reduced effort and inclusion of all authors into the voting process.

The results on the feedback from 19 of the 22 participants showed a very positive response:

• Clearness of execution and implementation/CMS interface: 94% strongly agree
• Acceptance of OPR (review phase): 100% strongly agree
• Acceptance of OPR (voting phase): 88% strongly agree
• Open identity skewed feedback towards too much positivity: 69% agreed
• Overall acceptance of OPR approach, would support again: 94% strongly agree
• Greatest fear associated with OPR included: biased/whitewashed reviews due to non-anonymity; backlash for bad reviewing (maybe over other channels/private email etc.)

We will continue to collect insights into OPR at a second venue: the conference eHealth 2018 held in Vienna. Here you can read more about the approach we are going to apply soon:  http://www.ehealth2018.at/author-infos/

Stay tuned for more results!