Peer review at the crossroads

Organisers: OpenUP

Speakers: Edit Gorogh (OpenUP project), Michael Markie (F1000), Tony Ross-Hellauer (OpenAIRE project), Jadranka Stojanovski (University of Zadar), Jon Tennant (Communications Director for ScienceOpen)
More information (incl. agenda & slides): Please refer to the workshop page

The workshop builds on the results of the OpenUp landscape scan and the OpenAIRE report on open peer review. It has multiple purposes including:

1. Assessing existing and evolving methods and functions of alternative peer review mechanisms
2. Breaking down peer review into the basic processes to identify the benefits and challenges
3. Identifying questions and issues that need further investigation.

Tony Ross-Hellauer conducted an extensive research on the various notions of Open Peer Review available in the scholarly discourse.

Relying on this definition of open peer review, research on the available alternative review tools and services was conducted within the OpenUP project. After analyzing the peer review tools according to the defined seven traits, four major groups were identified which not only make the classification of alternative tools and services easier, but also illustrate the current trends in open access publishing (e.g. preprint publishing, open platforms with public and peer commenting, etc.). The introduction also shed light onto the researchers’ attitude on transparency and openness in the peer review process. Both OpenAIRE (2016) and OpenUP (2017) surveys have come to the conclusions that:

1. There is an increased demand toward transparency in the scholarly publishing system, and therefore OPR is already becoming a mainstream solution, and although
2. Researchers show positive reactions to most OPR traits (esp. open interaction, reports, participation),
3. There is a surprisingly strong rejection of open identities.

Connected to our research and survey results, the invited experts were asked to share their experiences about the following topics:

• Recognition of review work, considering peer review as research output (Michael Markie),
• Barriers to open peer review, researchers’ attitude (Jon Tennant),
• Survey results on open access publishing in Croatia and presenting PEERE activities (Jadranka Stojanovski).

Jon Tennant started his presentation with introducing the main points of debate in the review discourse:

1. How can referees receive credit or recognition for their work, and what form should this take?
2. Should referee reports be published alongside the manuscripts?
3. Should referees remain anonymous or have their identities disclosed?
4. Should peer review occur prior or subsequent to the publication process (i.e., publish thenfilter)?

The primary obstacle in front of the uptake of open access review and publishing solutions is fear. Fear governs researchers’ decisions about participating in the review process. Fear on part of the researchers to be blamed for the refusal to perform peer review, for avoiding confrontational in public, for accepting abuse of power dynamics and intimidation, for the empowerment of bad actors, for marginalization, and for less honesty and criticism. Jon suggested several different solutions to this problem, such as:

• Solutions Built-in accountability processes.
• Transparency by default to level the playing field.
• Using standard ethnic codes (e.g., COPE).
• Increase quality through accountability and identification.
• Attach CC-BY license and DOIs to peer reviews – make them citable.
• Make peer reviews valued and part of permanent academic records.

Michael Markie opened his presentation by declaring the need for a fairer and more efficient peer review. Among the many benefits of open peer review, he emphasized that within open peer review researchers can receive credit for the work involved in conducting the review. Review work should be considered as a genuine research output, since it is an integral part of the scientific endeavour, improves quality and upholds standards, and allows others to build upon the knowledge base. Michael presented Publons as a good example of a review service, which offers help in keeping track of one’s review work. Publons is partnering with publishers that utilize the review report previously submitted. Open peer review implies that (1) adding name and affiliation means reviews can be attributed, (2) publishing the review means it can be given a permanent identifier, and (3) publishing a review enables usage metrics to be added. The question of rewarding review work was also discussed: how review work can be acknowledged, what kind of reward is effective to motivate researchers to participate in the review process.

Jadranka Stojanovski introduced the activities of the PEERE network. The main objectives of this collaboration are to analyze peer review in different scientific areas by integrating quantitative and qualitative research, (2) to evaluate implications of different models of peer review and to explore new incentive structures, rules and measures, and (3) to involve science stakeholders in data sharing and testing initiatives, and to define collaboratively a joint research agenda that points to an evidence-based peer review reform. PEERE is collaborating with big publishers, the result of this cooperative work is e.g. the PEERE Data Sharing Protocol, signed by Elsevier, Springer and Wiley representatives, regulates data sharing on peer review in all PEERE activities.

Jadranka also talked about the results of a Croatian survey in which 441 OA journals were asked to participate (129 answers – 78 SSH, 51 STM). The results reinforce a very traditional view of the peer review discourse in Croatia. The participating journals do not use crowdsourced review, OPR, or cascade peer review

Discussion

1. Key factors: open identities and report. Is there reluctance to share identities because researchers do not know what is best?

A major factor in this regard is getting the right person to do the review work, if it happens identity is not an issue. Furthermore, recognition for review can instigate a more collaborative process (Michael) A difference between big publishers and small society publishers should be taken into consideration. Survey results show that smaller publishers are afraid of loosing reviewers by opening up the process. For small communities openness might help collaborative work. Moreover, language barrier might support closed identity for review (Jadranka).

2. Traditional journals offering closed review
They offer more options now, more and more they have different journal titles with various peer review options, future might see the publishing the IDs of reviewers. Although ORCID is a profit organization. It is here to stay for the long run, becoming the norm in publishing (Michael).

3. Discussion on open identities includes issues such as people will not be so honest, if they are they will suffer the consequences? How can we increase training or dialogue in this subject?
Training is important, since there is also no education in review work at the moment. Regarding honesty, should be looking at it in a very small scale. Science  pen , F1000 had no deconstructive reviewers. Regarding the blind review process, anonymous reviewers can be identified easily by the authors. Publishers should share their data (Jon). However, we should not force reviewers to share their identities, we might loose good reviewers. Furthermore, some reviewers do not want to get credit for their work (Jadranka). So until this fear is not dealt with in academia, we should give the opt-out option for reviewers (Jon).

4. General bias in the process: does open peer review combat these human biases, preventing
these bias to occur?
OPR can change that. We looked into the gender disparity of our reviewers, finding only 30% female (after recommendations of the authors). We should look into these biases (Michael). Transparency can show the trends in research communication development (Michela Vignoli). Studies on gender bias show that it is not particularly a problem of peer review, but it is a problem of academia in general (Jon).

5. Using algorithms to find reviewers, counting reputation.
Algorithms help to find reviewers. Machines are not replacing selection, just help in the process.

6. Motivation to review: there is a problem of finding reviewers.
Open peer review and getting recognition might provide incentive to participate. Educating authors about the citations and credit they can get from it. In the metric driven world these incentives resonate (Michael). Reviewers should not be paid for review work. When money is introduced altruism is lost (Jadranka). By paying for it, somebody would come and just publish rubbish reviews for getting paid (Jon).

7. What happens if review is decoupled completely from journals?
Science Open offers collections of articles and open invitation system to attract reviewers. Community organized peer review is a possible solution (Jon). However, still need governing body to oversee the process despite the democratization of the process (Michael).

8. People do not comment usually
If you are rewarded comments will come, without reward it is a struggle to get comments (Michael). There is a blurred difference between review and commenting, there is definite overlap there (Jon, Jadranka). Comments to final products makes no sense since it is disassociated from the article. F1000 has an open-ended review with version control, in this case commenting works and makes sense. (Michael)

9. How can we accelerate the process of funding agencies pushing openness?
Funders will ask what your impact is, and peer review will be part of this evaluation (Michael). Services, such as Publons, incentivize and award review work (Jon). And there are examples in EU (Croatia) where review works should be reported, besides other academic results, as well (Jadranka).

10. Is OPR changing the nature of peer review? How is OPR related to the wider changes we see in the scholarly communication system?
OPR is more part of the open science discourse, it goes hand in hand with open data. It will become a natural phenomenon (Michael). It also exposes the misnomer that it has to be associated with an article, OPR is exposing the process, decoupling it from journals. Journals do not help open science (Jon).

--

Invited speakers: Jon Tennant (Communications Director for ScienceOpen), Michael Markie (Publisher for F1000 Platforms), Jadranka Stojanovski (PEERE, University of Zadar)