Valuing Scholarship by Manuscript Reviewers: A Call to Action

Authors and reviewers are intrinsically related; they represent 2 sides of the same coin of synthesizing and advancing knowledge. Each has a crucial role in developing work that advances a field. Authors create and then transform their scholarly work into manuscripts. Peers in the field assess the quality of the work and the degree to which it advances what is already known. These judges offer critical appraisals—of importance, study methods, logic of author arguments, how the work is situated within other advances in the field—as well as suggestions for improving the clarity and organization of a paper.⁵  These recommendations may extend to several pages of detailed organized comments, including suggested wording for rewrites and additional references to bolster or refute arguments. Reviews also inform editor decisions regarding the relevance of the work to the journal's mission and audience.⁵  When published, authors may receive honors and accolades—and financial (merit) incentives—for their work; reviewers, who may have improved the article substantially during 1, 2, or even more revisions, usually do not.

There are many benefits to reviewing, such as learning about new developments in the field, research methods, article organization, graphics strategies, and how to write clearly.¹  In addition, reviewers deserve credit for their scholarship. Although specific requirements vary among specialties, the Accreditation Council for Graduate Medical Education (ACGME) considers “serving as a journal reviewer” a scholarly activity for faculty.⁶  At least one professional society includes serving as a journal reviewer in their point system for determining eligibility for fellow designation.⁷  Some academic institutions have begun to acknowledge reviewer contributions as evidence of scholarship (Lalena M. Yarris, MD, MCR, written communication, March 12, 2021). For others, faculty may demonstrate the quality of their work by submitting sample reviews as part of their annual performance review for promotion (Box 1).

Some journals send letters to reviewers' supervisors—such as department chairs or division chiefs—describing exceptional reviewer performance. Other journals, including the Journal of Graduate Medical Education (JGME), annually send a certificate to reviewers ranked in the top 10% in review quality or recognize reviewers who have completed more than a certain number of reviews per year. Yet journals can do more to support reviewer recognition (Box 2). A transparent system to evaluate the quality of reviews, beyond metrics such as timeliness and number of reviews per year, would be a critical element of this process.

For promotions and other committees to factor in review performance, either journals need to provide credible measures of review quality or committees must assess actual written reviews, similar to their assessment of an individual's original publications. Currently, many journals do rate review quality and have metrics such as how many reviews have been done annually and review timeliness. Providing this information to reviewers seems a minimal return for reviewer efforts, yet may require additional staff and editor time, always in short supply. In addition, the reliability of ratings is often unknown, and journals use different, although usually similar, systems.

For JGME, reviewers' metrics include: the number of requested reviews per year; number of declined reviews; timeliness of completed reviews; manuscripts' numbers, titles, and final dispositions; and an average of all review ratings, assessed by the editors. These data are created in real time and often go back to the journal's inception, for most journals. These metrics reveal information about professionalism, engagement, and review quality in comparison with other reviewers.

At JGME, each review is rated by the handling editor on a score from 50 to 100 on the quality, accuracy, and clarity of the review. The Editor-in-Chief may adjust these ratings according to editors' tendencies to score reviewers either high or low, to homogenize the “tough” vs “easy” raters. Although subjective, editors' review ratings tend to converge after they have served on the editorial board for 1 to 2 years (ie, different editors tend to rate the same review similarly). These ratings may have consequences, as reviewers with low average ratings, based on several papers, may not be asked to review again. A more rigorous, while still efficient, rating system would be preferred. Nonetheless, existing metrics could be used, by faculty members seeking promotion, as evidence of high performance akin to assessments of faculty teaching and mentoring (Box 3).

Promotions or merit review committees should add reviewer performance to their criteria. Journals do collect metrics on review performance that could be helpful. Just as in other forms of scholarship (eg, publications, research grants, presentations), a single review would rarely be considered sufficient evidence of sustained productivity. Review quality and quantity over time is essential. We hope that that this editorial adds impetus to the conversations around reviewing and leads to increased recognition for reviewers' essential work in advancing scholarship.