Improving Peer Review: A Study on Structure

Journal peer review is a process that helps decide if a research study should be published. Editors get papers from researchers and send them to other experts (Reviewers) to examine. Reviewers check if the study's methods, data, and conclusions are solid before giving their recommendation. They can recommend that the paper be published as is, ask for changes, or reject it outright.

#Problems with Peer Review

Research has shown that the peer review system has many issues. Reviewers sometimes miss important problems with the research methods. They may not interpret results accurately, misuse references, or fail to report necessary details that would allow others to replicate the study. There can also be a lack of information needed to judge the study's quality or bias.

Many studies have found that different reviewers often do not agree with each other. For instance, one analysis found a low Agreement rate of only 34%. Even more recent data showed that in one round of reviews across many journals, only 30% of reviewers agreed with each other.

A part of this disagreement can come from the lack of clear Questions that reviewers are supposed to answer. In other words, if they do not have specific guidelines, it can lead to different opinions on a paper's quality.

#The Elsevier Pilot Study

In August 2022, a trial was run by Elsevier to test a set of structured questions that reviewers could use when evaluating submissions. This pilot study involved 220 journals from different fields and impact factor levels. The questions were meant to guide reviewers but were not mandatory, meaning they could choose to skip any questions they did not want to answer.

The goals of this pilot study included checking how reviewers used the new questions, looking at how much agreement there was between reviewers, comparing agreement rates to those journals' past performance, and improving the question set.

#Review Process Overview

Reviewers were asked to answer nine specific questions related to the research being reviewed. These questions covered things like if the study's objectives were clear, if the methods were detailed, and if the results were interpreted correctly. Some questions allowed for yes, no, or not applicable answers, while others were open-ended, allowing reviewers to explain their thoughts. After these questions, reviewers could also leave additional comments for both the authors and the editors.

Reviewers were also expected to give a final recommendation, such as whether the paper should be accepted, revised, or rejected.

#Analyzing Reviewer Responses

The study analyzed how reviewers answered the structured questions and counted the number of words in their responses. This was done by looking at how many reviewers answered the questions properly or directed answers to the comments section.

Comment sections showed varied responses. Some reviewers used the space to add on to their answers, while others copied complete review reports they had prepared earlier.

#Comments from Reviewers

Out of the reviewers, a large majority utilized the comments section to add their thoughts. Many crafted traditional review reports, which often contained summaries and detailed feedback. The word counts of these comments varied significantly, with traditional reports being longer than those using the structured questions alone.

When looking at how well these comment sections covered the structured questions, most traditional reviews addressed four or five out of the nine questions, usually focusing on methods and interpretations rather than strengths or limitations.

Notably, reviewers who filled out comments for the authors were also more likely to leave notes for the editors.

#Reviewing Agreement Among Reviewers

The study looked closely at how much agreement there was between reviewers on the structured questions. The highest agreement occurred when evaluating the paper's organization, while the lowest was on interpreting results and methods.

When it came to the final Recommendations, about 41% of reviewers agreed with each other, which is a significant improvement compared to past peer reviews that showed only 30% agreement. However, no notable differences emerged in agreement based on the field of study or the journal's impact factor.

#Refining the Review Questions

After analyzing the results of the pilot study, suggestions were made to improve the structured questions. Initially, some questions were worded in a way that made it unclear what a "yes" or "no" answer meant. Therefore, the questions were rephrased to have consistent language.

New questions were also created, targeting standard sections of a paper. The goal was to have a clear and simplified way for reviewers to assess papers, with a better mix of yes/no and open-text options to allow for more constructive feedback.

#Conclusion on Structured Peer Review

The pilot study showed that using structured peer review questions was well received by reviewers. They responded to the questions adequately, and the rates of agreement among them were better than in traditional peer reviews.

It's recommended that journals using structured peer review inform their reviewers about the questions early on, so they can keep them in mind when assessing papers. This approach could create a more consistent and efficient review process, ultimately leading to better decisions by editors and improved papers from authors.

Encouraging authors to use the same questions for self-evaluation may also help them enhance their submissions before sending them for review.

Overall, structured peer review could provide a clearer framework, leading to fairer assessments and improvements in research quality.