Editors have a love-hate relationship with medical education review articles: they are wonderful in concept yet can prove gnarly in execution. We often feel like excavators digging for the “gold” within. For researchers and educators, a well-done review article can save considerable time when searching for best practices, methods, and next research steps. Yet good reviews are difficult to carry out in the medical education realm. Are there useful strategies for first-time authors of review papers? How does one decide which approach to use, such as narrative, systematic, realist, or scoping? This editorial summarizes best practices from the perspectives of editors receiving review manuscripts, with a secondary aim of stimulating readers to consider a review article as a first step, before their next educational or quality improvement project.

First and foremost, a well-conducted review is often the initial resource for educators and researchers: review articles are read.1  A review examines and evaluates prior published—and sometimes unpublished—work on a particular topic, such as effective feedback methods or assessment tools for communication skills. Reviews identify, evaluate, and interpret key articles on a subject.

A scholarly approach requires that educators and researchers first examine current knowledge about a topic, before planning new curriculum, faculty development, or assessment projects. A recent review that summarizes the topic well is an invaluable resource. In the current dynamic world of medical education, updates to prior reviews are essential as well, because knowledge and evidence expand over time. When using precious discretionary time to research and read the literature, why not take the extra steps to use a rigorous approach to facilitate dissemination of your findings? Papers that effectively summarize the literature are welcome in a variety of venues, including local newsletters, blogs, print and online journals, and grand rounds or symposia presentations.

Choosing the review approach depends on the question you are asking and the type of evidence that exists. No type of review is “best.” Experts often view the various review approaches on a spectrum from quantitative approaches, such as systematic reviews with or without meta-analysis, to qualitative methods, such as narrative reviews. Realist and scoping reviews employ “mixed methods” that combine aspects of quantitative and qualitative concepts.2  Researchers should choose the method that fits the question(s) and sources of evidence (see table 1).

table 1

Commonly Used Medical Education Review Approaches

Commonly Used Medical Education Review Approaches
Commonly Used Medical Education Review Approaches

In the past, narrative reviews—sometimes called traditional, qualitative, or nonsystematic—reflected the opinions formed by the authors, with reference to articles considered seminal by the author-experts. The best examples of this type of review included a representation of influential articles on a topic, but did not use a comprehensive search strategy or review the quality of the evidence. This approach is subject to bias: articles can be selected without a transparent process and the quality of the original work is not always apparent to readers. For current narrative reviews, readers expect that authors will make their values, preferences, and assumptions clear.1 

Currently, narrative reviews may borrow strategies from systematic review processes, but remain flexible in methods and creative in synthesizing evidence.3  Narrative reviews can examine topics that are particularly diverse in subjects, methods, or outcomes; topics for which little original research exists; or broad overviews of a topic without a specific, focused research question. A qualitative approach is helpful to organize and synthesize various writings about an educational concept, particularly when few research articles exist or results cannot be easily pooled.

Recent narrative reviews resemble systematic reviews in that they often include a well-defined search of specified literature or other sources of evidence, with explicit selection and inclusion criteria, in addition to authors' expert opinions.1,4  Flexibility in format allows the authors to tell the review “story” in a more compelling or interesting manner than the typical systematic review. This format may attract a larger audience of readers as well. When possible, these reviews should address key issues, controversies, and questions of the moment. One example of a narrative review is the annual literature review presented at national meetings and/or published in specialty and medical education journals.5  Publications are selected using explicit criteria for quality, and themes are summarized, often without additional synthesis of the findings. Book chapters and case-based reviews are other examples of narrative reviews.

With an evidence-based medicine approach dominating clinical decision-making, it is unsurprising that the systematic review—a systematic collection, analysis, and aggregation of evidence supporting clinical practice—has become a leading method for medical education. The Best Evidence Medical Education (BEME) initiative was established in 1999 and emulated the evidence-based approach of the Cochrane Collaboration in clinical medicine. BEME led the development of standardized methods for medical education systematic reviews.6  These methods include a comprehensive search for all relevant studies and explicit, transparent criteria for selection: a recipe that others can reproduce exactly (box 1).7  For systematic reviews of interventions, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines specify the required study elements, yet these guidelines are less helpful for medical education interventions.8  In our experience, following PRISMA may not improve the clarity of a medical education review. Although BEME and other resources explicitly describe steps to conduct a medical education systematic review, this approach requires at least 2 investigators and inexperienced authors will benefit from having an experienced team that includes a medical librarian.

box 1 Key Elements for Systematic Reviews
  • Focused, often narrow research question

  • Explicit search strategy and inclusion and exclusion criteria

  • Protocol for abstracting, reviewing, summarizing article findings

  • More than 1 data abstracter and a process to adjudicate disagreements

  • Quality assessment of articles, using an accepted or transparent new quality index

  • Synthesis of findings

  • Quality of evidence used in determining strength of conclusions and recommendations

  • Discussion of how limitations may have affected findings

A systematic review with a meta-analysis is a powerful tool to amplify findings from quantitative studies, to increase the power to detect differences, and to enhance generalizability. In this type of review, combining data from smaller studies may allow calculation of an effect size (the magnitude of differences) and overcome the limitation of small samples in individual studies. However, in medical education, the subjects, interventions, outcome measures, and research designs are usually disparate and complex, which prevents aggregating data. In many situations contextual factors are part of the intervention and must be considered in a nuanced analysis. Other factors to consider include variable research designs and contamination issues, such as the unintended sharing of information with nonblinded subjects and researchers. As a result, meta-analysis is used relatively infrequently in medical education reviews.

A realist review, also referred to as a realist synthesis, uses evidence to further implementation of interventions and policy in real-world contexts.9  Realist review questions focus on what works, for whom, in what context, and why and might be considered a subset of systematic reviews. The realist inquiry seeks to understand why interventions work in some contexts and not in others, and derives from a realist philosophy that sits between positivism (reality can be observed, measured, and known) and constructivism (reality is experienced through human senses and cannot be objectively known). Realist reviews seek insights related to the social, cultural, and environmental contexts in which educational interventions operate, such as under what circumstances will learning from an intervention be likely to occur. These reviews seek to elucidate factors and mechanisms to improve the likelihood of success when transporting an intervention from 1 context to another. This type of synthesis is relevant in medical education, as studies using similar interventions can produce opposite outcomes. Novice reviewers will benefit from working with a team with prior experience in qualitative research or realist reviews.

A novel type of review, the scoping review, uses a broad approach to summarize and disseminate activity in a field by describing the extent of current knowledge and exposing gaps, which is particularly valuable in emerging fields.1012  These reviews cast a wide net, with evidence from editorials or opinion pieces as well as research studies. The approach is systematic but also flexible and iterative (with all steps clearly described), to capture a broad range of data and opinions. A scoping review may be undertaken to determine if a systematic review is feasible: BEME recommends this approach. With a variety of sources, along with variability in subjects, interventions, settings, and methods, scoping reviews analyze findings through quantitative and qualitative methods, with frequent author-team discussions. By providing an outline or map, a scoping review can be a valuable first step to guide the direction of next research efforts.11  A notable distinction of scoping reviews is that stakeholders, outside of the review team, may be consulted during the review process to ensure the findings will be helpful to potential end-users of the results. Scoping reviews are well suited to many medical education questions, but are not easy to do: a team with diverse skills in identifying and interpreting information is recommended.

The approach or type of review depends on the research question; thus, determining the research question is an essential first step. Much has been written about the importance of developing a great research question for original research; this is just as important for a review. When developing the research question(s), it is critical to determine how the review will build on past work and fill critical gaps: Why is this review needed? If a review has been published recently, how will your work enhance understanding of the topic? Use of different identification and selection methods (such as expanding the search to other repositories or other disciplines such as sociology or business) or use of a different review approach may provide useful new information and understanding.

A written stepwise process for identifying and selecting evidence (papers, presentations, policies), abstracting key information, and data analysis is important for a scholarly approach and indispensable for future publication. Reference to review guidelines is helpful, although this can be overwhelming due to the level of detail. Taking small sips at the guideline well, over time, will increase your understanding and insights regarding guideline application. No review can adhere to every guideline recommendation; however, being clear about where the approach did not follow the guidelines, and why, will increase the value of your work. Reading prior reviews, if they exist, will inform decisions regarding the question, approach, and processes.

Another essential ingredient to reduce bias and human error is to involve more than 1 person in the review process. This will require methods to adjudicate disagreements for each step of the review process. Collaboration is essential for each decision, which includes reviewing the quality of the evidence or materials selected for review. Randomized double-blinded, controlled trials do not abound in medical education and authors likely will need inclusion criteria that encompass other research designs. The quality of studies will vary, from threats to internal validity (results are credible, true) or external validity (generalizability of results) or both. It is essential to review the quality of the evidence, with your own clearly stated, relevant criteria or with a previously used quality index, such as Medical Education Research Quality Instrument, Newcastle-Ottawa Scale–Education, and QUESTS (BEME) criteria for quantitative studies, or Côté and Turgeon for qualitative work.6,1315  As no criteria will exactly fit a given paper, reviewers use judgment in their application—with these decision steps included in the methods section for transparency.

At this point many readers may be thinking: this is too much work—time to stop reading. Please do not! The Journal of Graduate Medical Education (JGME) has received valuable review papers from new as well as experienced author-investigators. Actually, what comes next in the process is arguably more difficult yet also more creative and satisfying: synthesizing findings from the review process.

Organizing, interpreting, synthesizing, and summarizing disparate findings (different learners, interventions, outcomes, timelines) into a coherent story for readers or listeners is challenging. If you chose a narrow topic, it may be straightforward—but in this case few articles or evidence may have been discovered, which may lead to few new insights. Organizing the various articles (evidence) by similarities in the intervention or methods to reveal new understanding requires thoughtful discussions.

Be creative in organizing the information for the readers. Pay particular attention to the visual presentation of data. Often authors work hard to amass and annotate a large amount of evidence, but devote less effort to synthesizing and summarizing the findings. Analysis must go beyond listing the characteristics of identified articles and evidence, with use of quantitative or qualitative methods to produce new insights. Often the most interesting analysis looks at why different types of studies have variable findings; an in-depth review of the individual study methods and bias sources should provide some answers or at least hypotheses. It is particularly important to look at results from the highest-quality studies rather than lumping high- and low-quality work together.

Remember to discuss, in a concise limitations section, how key decisions in the review process or the authors' values may have affected the analyses and findings. Finally, stating “more research is needed” is not helpful for readers or listeners. More research is always needed, but any research recommendations should be specific and derived from the findings.

Medical education reviews are difficult, but helpful resources for novice as well as experienced authors exist (see table 2). A team effort yields the best results while diffusing the workload. JGME editors are interested in receiving new review papers for consideration. We respectfully ask potential authors to consider our “editor pet peeves” list before submission (see box 2).

table 2

Resources for Reviews

Resources for Reviews
Resources for Reviews
box 2 Editor Pet Peeves Regarding Review Papers
Abstract
  • Omits reason review needed at this time

  • Omits how articles or resources are chosen

  • Omits brief assessment of the quality of the evidence

  • Omits actual findings

  • Lacks 1 or 2 best practices, recommendations, or next research steps

Introduction
  • Fails to mention key prior reviews, and why this review is “added value”

  • Omits research question, or question is unclear

Methods
  • Omits type of review approach

  • Unclear steps to identify and select articles or evidence

  • Authors' stance, values, or perspective not clear

  • Unclear quality assessment process

Results
  • Important evidence (eg, articles) missing

  • Text describing findings is not organized and is difficult to follow

  • Tables are not concise and are difficult to follow

  • Highest-quality study findings lumped in with lowest-quality study findings

Discussion
  • Little attempt to synthesize or analyze findings, and point the way forward

  • No discussion of how method decisions and author values may have affected findings

Conclusions
  • Stating “more research is needed”

Other researchers and educators will be grateful for your efforts to craft a well-written review paper. Some reviews are read and cited for many years, even decades, after publication. Another benefit from writing (or presenting) a review is that, whether you are an expert on the topic at the start, you certainly will be by the end of a review project, which can raise one's visibility and stature locally and beyond.

1
Dijkers
MP,
Task Force on Systematic Reviewers and Guidelines. The value of traditional reviews in the era of systematic reviewing
.
Am J Phys Med Rehabil
.
2009
;
88
(
5
):
423
430
.
2
Cook
DA.
Tips for a great review article: crossing methodological boundaries
.
Med Educ
.
2016
;
50
(
4
):
384
387
.
3
Pautasso
M.
Ten simple rules for writing a literature review
.
PLoS Comput Biol
.
2013
;
9
(
7
):
e1003149
.
4
Dyrbye
L,
Shanafelt
T.
A narrative review on burnout experienced by medical students and residents
.
Med Educ
.
2016
;
50
(
1
):
132
149
.
5
Fromme
HB,
Ryan
MS,
Darden
A,
et al.
Top medical education studies of 2016: a narrative review
.
Acad Pediatr
.
2018
;
18
(
5
):
485
492
.
6
Harden
R,
Grant
J,
Buckley
G,
et al.
Best evidence medical education. BEME Guide No. 1
.
Med Teach
.
1999
;
21
(
6
):
553
562
.
7
Patricio
M,
Vaz Careneiro
A.
Systematic reviews of evidence in medical education and clinical medicine: is the nature of evidence similar?
Med Teach
.
2012
;
34
(
6
):
474
482
.
8
Cook
DA,
West
CP.
Conducting systematic reviews in medical education: a stepwise approach
.
Med Educ
.
2012
;
46
(
10
):
943
952
.
9
Wong
G,
Greenhalgh
T,
Westhorp
G,
et al.
Realist methods in medical education research: what are they and what can they contribute
.
Med Educ
.
2012
;
46
(
1
):
89
96
.
10
O'Brien
KK,
Colquhoun
H,
Levac
D,
et al.
Advancing scoping study methodology: a web-based survey and consultation of perceptions on terminology, definition and methodological steps
.
BMC Health Serv Res
.
2016
;
16
:
305
.
11
Thomas
A,
Lubarsky
S,
Durning
SJ,
et al.
Knowledge syntheses in medical education: demystifying scoping reviews
.
Acad Med
.
2017
;
92
(
2
):
161
166
.
12
Pham
MT,
Rajić
A,
Grieg,
JD,
et al.
A scoping review of scoping reviews: advancing the approach and enhancing the consistency
.
Res Synth Methods
.
2014
;
5
(
4
):
371
385
.
13
Reed
DA,
Cook
DA,
Beckman
TJ,
et al.
Association between funding and quality of published medical education research
.
JAMA
.
2007
;
298
(
9
):
1002
1009
.
14
Wells
GA,
Shea
B,
O'Connell
D,
et al.
The Newcastle–Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses
.
The Ottawa Hospital
. ,
2018
.
15
Côté
L,
Turgeon
J.
Appraising qualitative research articles in medicine and medical education
.
Med Teach
.
2005
;
27
(
1
):
71
75
.