Literature Reviews: Key Considerations and Tips From Knowledge Synthesis Librarians

The essential groundwork of a successful review involves developing a clear research question, along with inclusion and exclusion criteria. As will be described in more detail throughout this JGME Literature Review Series, the research question must align with the review type. While some questions are narrow in scope, defined before data collection, and precise (eg, systematic reviews), others are broad in scope, evolve over the course of the data collection and analysis, and become precise during the review process (eg, state-of-the-art reviews). In general, the research question should be both comprehensive and clear, with details about the key concepts: the population or problem; intervention, innovation, or exposure of interest; and particular learner, organization, or health outcomes. Researchers may use a question framework for either quantitative (eg, PICO [patient/population, intervention, comparison, outcome]) or qualitative questions²  to guide formulation of the review steps. The review on theory in interprofessional education by Hean et al is an excellent example of using a structured question framework to inform search strategy development and inclusion criteria, when using a theory-oriented question framework.^3,4 

Whether researchers write a formal review protocol or research proposal, as is strongly recommended for systematic and scoping reviews, authors should prepare for a substantial planning stage. Planning starts with developing and refining an appropriate research question(s), building a team, and selecting a suitable synthesis method.

Conducting a preliminary literature search, to confirm the need for a review and get an initial sense of the types and volume of data sources, is an important part of the planning stage. This search can include looking for other recent relevant syntheses and considering the methods choices of these existing reviews (eg, search strategy and inclusion criteria). Keep in mind that initially scanning the literature and refining the review question may be an iterative process, where the former can influence modifications to the latter.

From early stages it is important to determine the resources and technologies available to assist with the synthesis process. Citation management software, review management software, and data analysis and reporting tools should be considered in addition to bibliographic databases and journal subscriptions. The Table outlines the types of tools that can be used throughout the various stages of the review process, as well as the team members likely to have expertise in the skills and processes for each stage. For examples of existing tools, visit the Systematic Review Toolbox website⁵  or the knowledge synthesis research guide from an institutional library (Box).

Finally, the review team will need to discuss and decide on a list of inclusion and exclusion criteria. In other words, how will reviewers know whether a given article or other data source should be retained for the synthesis? Criteria could include population, type of publication, language, time period, type of study, and sample size, but will vary depending on the research question and type of review. These criteria can provide useful information about which keywords to include, which databases to search, and which other resources to use, if appropriate.

Pro Tip 1: Determine the ideal venue(s) for publication and consider the target audience during the planning phase to ensure alignment with the research question and purpose.

Identifying studies begins by searching the appropriate databases and is determined by the research question. Once database searches are complete, it may be important to supplement the search with studies that are not included in the databases. This often means conducting a supplemental search by reviewing conference abstracts, technical reports, association websites, and other grey literature resources (ie, research produced by organizations outside the traditional commercial or academic publishing and distribution channels).

It is important to work with an information specialist (IS), such as a librarian, to determine the most appropriate sources to search and the most efficient approaches to identify potentially relevant evidence. The IS can advise, or even help to develop, the search strategy using best practices for conducting and reporting searches. We recommend inviting an IS to join the review team or to serve as a consultant.^6,7  The IS expertise in conducting thorough searches, as well as reporting the process properly to enhance the rigor and reproducibility with new PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses literature search extension) guidelines,⁸  will strengthen the review. Information specialists can be found at academic and hospital libraries, by contacting professional medical associations, and through knowledge synthesis networks. If your project is not funded, many IS experts will provide at least some free consultation or may be invited to join the review team.

Pro Tip 2: For GME topics, don't forget to search education-specific databases such as the Education Resources Information Center (ERIC), Education Research Complete, or others at your institutional library.

Evaluating and appraising the evidence includes initially screening the retrieved studies or other data sources to determine eligibility, based on fit or relevance to the topic. For some review types, this may consist of a systematic screening of all studies to select those that meet the inclusion and exclusion criteria. For others, researchers may use a more strategic and purposeful means of sampling the data to address the review question. In some cases, eligibility may also include meeting certain methodological quality standards. Some review methods will include a formal appraisal process, such as using a checklist to determine the risk of bias within the data source (eg, Cochrane risk of bias tool⁹  or Medical Education Research Study Quality Instrument¹⁰ ), whereas other reviews may look for fit with the theoretical or conceptual framework of the issue being examined. Regardless of the method used, this evaluation and appraisal stage of the process involves winnowing all the data found in the search strategy to that which will best help the reviewers address the research question. This process can be depicted by the PRISMA flowchart that is included in some reviews, such as systematic and scoping reviews.

Pro Tip 3: Pilot your inclusion criteria with a sample (eg, 10% of the full set of retrieved records) to ensure the entire review team is clear on what should be included and how to operationalize the criteria. This will help improve the inter-rater reliability of the project.

Pro Tip 4: Consider using review management software to handle the large number of citations and facilitate collaborative screening.

Each review type will have different approaches for collecting and combining data, which involves pulling the relevant pieces of information out of the selected evidence. Refer back to the review question and the planned analysis when considering the variables that should be extracted from each study. Some review types (eg, systematic and scoping reviews) include extracting specific variables into forms or tables using a data extraction template. Other types of reviews (eg, state-of-the-art reviews) will use inductive or interpretive approaches to generate themes, codes, or other types of new data out of the texts of the included data sources. Regardless of the review type, the common aspect to this stage is distilling the primary data to the elements that will be used to answer the research question.

Pro Tip 5: Piloting the data extraction form with a handful of included studies will catch omissions in the data collection process and improve the inter-rater reliability of the project.

At this stage, researchers will bring the results from the individual studies together for analysis and discuss what they have identified through the synthesis. While this step will look different for each review type, this process is a distinguishing feature of all knowledge synthesis work. Whether the synthesis process is quantitative, qualitative, conceptually guided, or some combination thereof, this step is what makes a review an actual form of knowledge synthesis. At this step, researchers transparently and systematically begin drawing together data from disparate sources to address the research question. Whatever form this synthesis may take, the important commonality is that the synthesis process itself is clearly explained and key decisions made by the team are explicitly described for the reader.

Pro Tip 6: For some review types, there are software options that can assist with analysis, such as RevMan for meta-analyses or qualitative data analysis software for qualitative analyses.

Once the findings have been synthesized and the review team is ready for dissemination, several factors need to be considered. In addition to the text, visualizations, such as figures and tables, are key for disseminating reviews. Pulling these parts into a coherent narrative, with key findings highlighted in engaging ways, promotes communication. Researchers should use the reporting standards for their specific review type as a guide when formatting their report into a manuscript for publication.

From our experience supporting hundreds of reviews, we have seen review teams thrive, struggle, and sometimes do both. The most successful review teams have a good plan, use expert advice and technological tools, and establish excellent communication practices. Like any research project, GME knowledge syntheses require a thorough understanding of the selected methods and a clear research question and target audience. Whether the goal is to change policy, develop new programs, or design better assessments, a well-conducted review can provide valuable evidence to support decisions.