Report from the Peer Review Workgroup

Peer review is the worst form of evaluation except all those other forms that have been tried from time to time.
with apologies to Winston Churchill

Introduction

Peer review is a much-discussed topic. The peer review workgroup at OSI2016 was well aware that there is already considerable material available on peer review: a little formal research; a great deal of thoughtful and insightful commentary; organizations such as PEERE,¹ which aims to improve the transparency, efficiency and accountability of peer review; and a range of experiments and innovations already taking place. Much of the debate is about the different flavors of traditional peer review in the context of the editorial assessment that leads to publication in a scholarly journal or monograph—double-blind or single-blind versus open—and which flavor is best suited to minimize the acknowledged limitations of traditional peer review.

Conscious of the sheer quantity of thinking and innovation already taking place in peer review, the workgroup agreed that within our timeframe of two days we would be unlikely to reinvent peer review, and that it would not be a good use of time to go over ground that had been well covered already. We therefore agreed to focus on peer review in the context of open scholarship and to identify immediate priorities for further work.

Overview of recommendations

After an evaluation of the entire scholarly process from idea to post-publication commenting, and after conducting an analysis of the strengths, weaknesses, opportunities, and threats of traditional peer review (see Appendix: SWOT), our workgroup identified goals that could be implemented to move peer review toward a more transparent and inclusive process. Our recommendations are as follows:

• Move toward greater transparency to improve accountability and minimize bias.
• Move toward greater inclusiveness by encouraging wider participation in formal and informal peer review at different stages in the development of research outputs.
• Explore separating the evaluation of rigor and soundness before publication (e.g., through technical checks) from the evaluation of interest, impact, and originality after publication in order to speed time to publication.
• Identify new approaches that lessen rather than increase the burden of reviewing and decrease the waste of reviewer’s time.
• Conduct more evidence-based analyses of different forms of peer review.
• Address incentives and motivations to participate in peer review, not only in the context of rewards or credits for individuals but also in terms of the importance of peer review for promotion and tenure.

We recognize the following challenges:

• There are differences among disciplines, publishing models, generations, platforms, and so on that affect the practicalities of a move to complete transparency.
• Incentivizing a cultural shift toward openness is required not only for “traditional” peer review but also (and even more so) for some of the innovations that we discussed.

Underpinning all of our discussions was the question of motivation and incentives. The value of peer review needs to be acknowledged, whether through formal credit systems or in simple rewards for taking part, but also this work needs to be valued by the employers and funders of those doing peer review and to count meaningfully towards career progression.

What is peer review?

Peer review was defined for the purpose of this discussion as follows:

• Peer review is the critical evaluation of scholarship by relevant, non-biased experts who give constructive feedback.
• Peer review provides validation of scholarship with the goal of its improvement.
• Peer review may include (and traditionally does include) judgment of impact on and interest to a field.

According to this definition, peer review serves more than a gatekeeping or filtering function in the scholarly dissemination process: it can and should contribute to and enhance the literature.

For the sake of convenience, this report uses “publication” in order to refer to the traditional, formal publication of results in a journal or monograph. We acknowledge, however, that publication can be formal or informal, can include making research outputs publicly available throughout their development, and that these outputs can potentially benefit from some form of peer review at any stage.

The categories of peer review we defined and discussed were:

• “Pre-publication” peer review refers to any form of review by one’s peers prior to formal publication (i.e., as a journal article or monograph).
• “Traditional” peer review refers to the formal review process typically undertaken as a contingent of formal publication.
• “Post-publication” peer review refers to any form of evaluation following formal publication.

Recommendations for pre-publication peer review

Our group recommends the following actions with regard to pre-publication peer review:

• We encourage the use of preprint servers so that work in draft is openly available, which enables wider feedback and commentary, leading to greater readiness for formal review in the context of a publication. Peer review aside, we believe that preprint servers in themselves are beneficial, establishing priority and increasing the speed at which information is disseminated, while also encouraging collaboration through feedback and commentary.
• In this context, we also encourage the facilitation of a flexible, nonlinear process of peer review outside of and supplementing journal-based peer review that facilitates many kinds of scholarly engagement and collaboration (e.g., through new tools and broader cultural acceptance). The type of review conducted will depend on the nature of the output, the stage in development of the output, and the timing of the output.

Questions:

1. Is there a role for preprint servers in the publication of scholarly monographs and could they benefit from a form of peer review in this context?
2. How could participation by a wide spectrum of commenters be encouraged, so that all research benefits?
3. Is the option of anonymous review necessary to encourage participation, but with credits only available for signed reviews?
4. Should reviewers be acknowledged in the formal publication?
5. To encourage participation in review at this early stage, should there be a closed (facilitated) route—for example, overseen by learned societies?

Recommendations for traditional peer review

Our group recommends the following actions with regard to traditional peer review:

• We recommend that all disciplines work toward a culture of openness in peer review.
• We encourage the exploration of and deliberation on the problems, real and perceived, regarding transparency in peer review.

We acknowledge that we need to hear from a broad spectrum of authors as important stakeholders, in order to better understand the difference in acceptance and enthusiasm for more open peer review in different disciplines. Evidencing these differences:

• A number of journals already operate fully open peer review, without a blind review process, with signed reviews that are published alongside the final article (e.g., many BioMed Central journals, BMJ Open).
• Frontiers journals publish the name of the reviewers on the articles.
• eLife publishes the editor’s decision letter, including a summary of the reviewers’ reports as well as the authors’ response. The reviewers and editors collaborate on the key points that need to be addressed by the authors.
• The Journal of Design and Science from the MIT Media Lab will offer open review on a collaborative publication platform, encouraging community collaboration rather than formal peer review.
• The EMBO Journal publishes the unsigned reviewer reports, editor decision and author responses with the published article.
• PeerJ reviewers may choose to sign their reports.

The benefits of fully open peer review are generally accepted to include the following:

• less likelihood of bias, or transparency if a reviewer has a potential competing interest
• accountability of reviewers
• reviews may be more constructive²
• if published, the review becomes part of the published record and the reviewer can be credited with the work.

The workgroup recognized that not all disciplines are ready to make the leap to a fully open and transparent process of peer-review, and also that there are areas where full openness might not be appropriate—for example, for ethical or security reasons. Furthermore, the group was aware that some journals choose to use double-blind peer review in the belief that it tackles some of the issues that open peer review also addresses, or for broadly cultural reasons (e.g., fear of retribution, etc.). More evidence-based research on the different forms of peer review would be useful.

Meanwhile, we encourage preliminary steps that encourage the cultural shift needed on the road toward fully open peer review. Such steps could include decoupling the elements of fully open peer review by

• making the signing of reviews possible but optional, encouraging reviewers to identify themselves to the authors if they so wish;
• allowing reviewers to choose to have their names published along with the paper; and
• publishing unsigned reviews with the paper.

Third-party tools may be used to facilitate the disclosure and publication of reviews. A few options include:

• Peer Review Evaluation (PRE)³ provides third-party validation of the review conducted on a journal’s published papers, through a widget showing the level at which the paper was reviewed. This widget also gives publishers the ability to include the reviews alongside published papers if they lack the technology to implement this themselves.
• Publons⁴ allows individuals to create verified reviewer profiles for tracking their review activity. In addition, publishers can partner with Publons to encourage reviewers to create profiles and publish reviews on the Publons platform.

Recommendations for post-publication peer review

Our group recommends the following actions with regard to post-publication peer review:

• We recommend the facilitation of post-publication review of traditionally reviewed publications.
• We believe this is more likely to work if it is formalized and facilitated.
• We recommend experiments with crowd systems that incentivize broad, representative participation—for example, with a currency, rating, or credit system.
• As with review at other stages of the development of the research output, any credits or ratings should be acknowledged by employers or funders of those doing the reviews as valid metrics in career progression.

We acknowledge that the benefits to research of facilitated post-publication peer review (detection of errors and misconduct, continuous improvement of the literature) must outweigh the complexities that this might introduce into scholarly communication, such as the sheer amount of information that could become available; the technology required to underpin such a system; and the processes required to facilitate useful discussion, review, and response.

Questions:

1. Should comments on and discussions about articles be appended as part of the published record, or should the articles themselves be updated on the basis of post-publication review, just as they are with pre-publication and during-publication review?
2. What would be the implications of a multiplication of versions for indexing and citation, and for the concept of the version of record?

A number of forms of post-publication commentary are already available, both on publication platforms and on third-party platforms. The comments made available on some of these platforms are anonymous and include:

• informal unsolicited discussion or commentary after full peer review (e.g., in the comments section of the publication platform itself) and third-party options such as PubPeer,⁵ Pub-Med Commons⁶ and ScienceOpen⁷.
• publication after a technical check followed by invited full peer review (e.g., with F1000 Research, the article appears in PubMed only after two peer reviews have been received⁸).

Potential issues to consider

Our workgroup included members who have published research, but no members whose primary role was conducting (or reviewing) research, nor employing or funding researchers. We cannot design new models of peer review without the input of these stakeholders.⁹

Further, the recommendations above led to several questions that should be assessed further:

1. Would encouraging the extensive use of preprint servers for all scholarship and encouraging the formal publication of reviewer reports contribute to information overload?
2. Are there standards for validating reviews?
3. Are we technologically ready for preprint servers?
4. What impact will peer-review outside formal publication have on journals? What will the role of the journals be? Societies vs. publishers? Are journals the “filters”?
5. What is the sweet spot in the timeline for the open review? Is it field-dependent?
6. Would costs be shifted rather than reduced? Publication of multiple versions of scholarly outputs may incur additional costs.
7. Can we kill the concept of the “version of record,” allowing updates and re-review?
8. How can fears—real or perceived— regarding possible retribution in an open review culture be addressed? How will potential cases of retribution be addressed?

Conclusion

On our first day we produced a SWOT analysis of traditional peer review (see Appendix: SWOT). The weaknesses can be grouped under these broad headings:

• possibility of bias
• not designed to identify misconduct
• delays publication
• may conceal conflicts of interest
• lack of incentive to participate (e.g., credit).

We agreed that more open peer review should, by its very openness, address issues of bias and conflicts of interest. Extending peer review outside the formal publication process and opening it up to a wider cohort of reviewers might increase chances of identifying misconduct. Prepublication review in the context of preprint servers should mean that papers are submitted when ready for publication, reducing rounds of re-review by the journal. That said, by formalizing peer review during multiple steps in a publication process, attention must be paid to the burden on reviewers. Expecting a more formal review process at multiple stages of scholarship could exacerbate reviewer burnout. Providing credit may incentivize reviewers but it not yet known whether credit would in fact encourage more people to participate as reviewers.

The workgroup concluded that open review at all stages of the research process would include the benefits of more democracy, more people involved, more likely to spur collaborations, and more sharing of knowledge at an earlier stage (prior to formal publication).

The following actions are recommended. Over the short term, we should:

• collate a library of literature on peer review in the context of open scholarship, including published research, good commentaries, and blogs;
• encourage peer review for grant funding and for outputs other than research paper (e.g., data sets and software);
• develop a better understanding of how peer review is conducted in the humanities and social sciences, especially for scholarly monographs, in order to assess how it could be more open; and
• collate an overview of current, innovative peer review initiatives, again with a focus on open scholarship.

Over the longer-term, we should:

• carry out pilot studies or focus groups in order to gain a greater understanding of different perspectives of researchers with respect to open peer review in different disciplines (e.g., genomics, engineering, psychology, history); and
• standardize the use of DOI for peer reviews and for the identification of different versions (i.e. pre-publication, post-publication), working with Cross-ref, NISO, and others.

OSI2016 Peer Review Workgroup

Bev Acreman, Commercial Director, F1000

Peter Berkery, Executive Director, American Association of University Presses (AAUP)

Caroline Black, Editorial Director, BioMed Central (SpringerNature)

Chris Bourg, Director, MIT Libraries

Becky Brasington Clark, Director of Publishing, Library of Congress

Angela Cochran, Director of Journals, American Society of Civil Engineers (ASCE)

Kevin Davies, Vice President for Business Development, American Chemical Society, and Publisher, C&EN

Rachel Dresbeck, President, National Organization of Research Development Professionals (NORDP) and Director of Research Development and Communications, Oregon Health and Science University

Catriona MacCallum, Advocacy Director, PLOS

Paul Peters, CEO, Hindawi Publishing

Bobby Schnabel, CEO, Association of Computing Machinery

Francisco Valdés Ugalde, Director General, Latin American Faculty of Social Sciences (FLACSO) in Mexico

Appendix: SWOT of traditional peer review

Strengths

Weaknesses

Relies on trust/goodwill Trusted by researchers Peer review adds value Provides some level of validation by experts Mostly trusted by most research communities Imperfect, but the best system we have to date It does work within its limits Voluntary/free Encourages care and rigor Filters for a target audience Expert scrutiny Often leads to improvements or discovers flaws Adds credibility to published works Can sometimes spot flaws Improves papers when it works properly Improves science and stimulates thinking Sets criteria for acceptance, thereby motivating authors to improve quality10 Favors discussion and feedback Tried and tested Careful reading is a benefit

Lack of openness hides bias Biased with regard to gender, affiliation, country, discipline, which interfere with objectivity and empower certain views and/or paradigms Not transparent – biases go uncovered Susceptible to conflicts of interest (amongst reviewers, editors) Single-blind peer review allows reviewers to veil criticism behind anonymity11 Not ‘blind’ enough Unintentionally promotes conservatism (especially grants, but that’s a different conference perhaps…) Doesn’t promote innovation Negative/inconclusive papers not published Dependent on trust and goodwill, which is eroding Perceived credibility No credit for reviewing Not designed to identify (and doesn’t protect from) fraud and misconduct Data in supplementary material often overlooked Complex methods in multidisciplinary papers Review of only one research object (article) at one time period Little training for peer reviewers Increasingly difficult to find reviewers; open access journals may not attract quality reviewers Reviewers review for journals and editors, not for their peers Element of chance — only 2 or 3 reviewers out of many potential opinions No independent scrutiny and analysis Too few eyes The longest part of the publication process — can be time-consuming, slow — which delays publication.12 This might mean that important data is withheld from public/researchers. Reviewers at some journals delay publication by imposing burdensome/non-critical demands on authors Scooping Unwieldy system for managing is cost-and resource-intensive Peer review stops on publication Doesn’t add value

---

Opportunities

Threats

Pre-and/or post-publication review could be a new model Fully transparent post-publication review for journals Fully transparent pre-publication review for books Becoming more public Open, post-publication peer review Credit/recognition for reviewers an essential part of scholarly ecosystem Cascade review can reduce inefficiency13 Automation/de-skilling of some elements-leave it to people to judge results Quality/science/impact Better tools for matching qualified reviewers to content In an online environment it is possible to make peer review more of an ongoing process Open review promotes transparency14 Portable peer review Remove shackles of print/mail and develop existing system for digital world

Peer review is an attention portal that adds value, so changing it could be threatening It is unclear whether researchers will continue devoting time to peer review if they are not incentivized to do so If not done by the journal where does that leave the journal? Does it matter? “Managing peer review” becomes commercial product People thinking it’s fixed Novel ideas and emerging subjects disadvantaged Throw everything online and hope for the best leads to lots of shoddy information Flawed research still gets published (e.g., STAP, Benveniste, etc.) Closing the scientific mind Gaming/fraud/cheating Bias Corruption Time (waste of extensive amount of time finding reviewers)

Notes: