Systematic reviews are rigorous and complex. It is paramount that readers—consumers of systematic reviews as well as researchers—understand the steps necessary for high-quality systematic reviews. Here are brief summaries of key quality considerations for readers of medical education systematic reviews. For detailed research steps aimed at those performing systematic reviews, see our companion paper.1
1. Evidence of Preparation
Did the researchers use resources such as the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) Statement2 or the STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES) Statement?3 Both the PRISMA and STORIES statements act as guidelines for what is necessary to report in the manuscript.
2. Key Team Members
3. A Focused Research Question
The research team should have a focused research question that is neither too broad nor too narrow. A question that is too broad can lead to retrieving far too many studies to review and failing to provide a focused overview for readers. A too narrow question may result in being unable to find any existing studies that meet the inclusion criteria.
4. A Registered Protocol
5. A Defined Search Process
A systematic review search strategy is based on the review's focused research question and includes a variety of search terms synonyms (eg, faculty, educator, teacher) and, depending on the database, controlled vocabulary terms (eg, medical subject headings [MeSH]). The intention is to comprehensively identify all studies that possibly meet the inclusion criteria. Once the main search elements are defined, the search is conducted in a minimum of 3 databases.4 If conducting a review for medical education, databases such as ERIC (Education Resources Information Center) or PsycINFO may be searched.10 The authors may have also searched grey literature sources such as theses, dissertations, and conference abstracts to reduce publication bias: some studies are conducted but never published. The search strategy for each resource and the date it was searched should be documented.11 (Note that these details are expected by most journals, and the PRISMA reporting guidelines require authors to include the full search for at least one database.2 )
6. Careful Screening of Search Results
There should be 2 levels of screening: title/abstract and full text, both guided by the review's inclusion and exclusion criteria. In both screening levels, 2 reviewers should independently screen each result; this minimizes bias in the application of inclusion and exclusion criteria. If the reviewers disagree, the conflict may be resolved by a third individual or by the 2 reviewers reaching consensus through discussion. While it is unnecessary for authors to report exclusion reasons when screening at the title/abstract level, it is necessary to report this information at the full-text level. This information is often presented as a PRISMA flow diagram2 using tracking software programs (eg, Covidence, Rayyan, DistillerSR).
7. Collaborative Data Extraction and Analysis
After completing the screening processes, the researchers will extract data from individual studies. To facilitate this process, authors should collaboratively draft and pilot a data extraction tool designed to capture all relevant data and extract it in a format that is conducive to analysis and synthesis. Data extraction should be conducted by 2 team members independently to assure reliability and reduce errors.
8. Study Appraisal and Data Synthesis
Authors should critically appraise each study through a quality and risk of bias assessment, such as with the Newcastle-Ottawa Scale-Education (NOS-E) or the Medical Education Research Study Quality Instrument (MERSQI).12 Depending on the review's research question, the extracted data may be analyzed qualitatively or through a meta-analysis. Meta-analysis of quantitative studies is possible only if the data from the individual studies is homogenous; if the data are too heterogeneous, as is often found in medical education, it is not possible to accurately perform a meta-analysis. In qualitative analyses, the data are analyzed, and themes are drawn out, synthesized, categorized, and presented.13
9. Manuscript Clarity
Authors may reference the PRISMA2 or STORIES statements3 as their writing and reporting structure. Whether or not these guidelines are cited, the article should present the rationale, methods, and findings in a clear, ordered, succinct fashion and discuss the relevance to existing literature and to readers.
Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University, the US Department of Defense, or the US Government.