Understanding State-of-the-Art Literature Reviews

SotA literature reviews provide a time-based overview of the current state of knowledge about a phenomenon and suggest directions for future research.¹  They are organized in relation to how the understanding of the phenomena has evolved over time. Structured around turning points in the history of knowledge development, SotA reviews articulate: This is where we are now. This is how we got here. This is where we should go next. By synthesizing how the main characteristics of a topic have changed over time to give rise to current understandings, SotA reviews offer a modern knowledge synthesis that “tend[s] to address more current matters in contrast to other combined retrospective and current [literature review] approaches.”¹  SotA reviews are used prolifically in many fields, such as biomedical science, medicine, and engineering, to provide information on the current understanding of a topic, the historical roots that shaped the understanding, and potential next directions for future research.

Given their time-based and turning point-based orientations, SotA reviews are inherently different from other types of knowledge synthesis. For example, systematic reviews focus on specific research questions that are narrow in scope; in contrast, SotA reviews present a broader historical overview of knowledge development. Scoping reviews focus on mapping the present state of knowledge about a phenomenon, including, for example, the data currently available, the nature of that data, and the gaps in knowledge. Conversely, SotA reviews offer interpretations of the historical progression of knowledge relating to a phenomenon, centered on significant shifts that occurred during that history.² 

SotA reviews are especially useful within graduate medical education due to their purpose: these knowledge syntheses focus on the turning points that ended older ways of thinking and gave rise to current insights, while also evaluating where the field should go next. Thus, by conducting this type of review, educators and researchers in graduate medical education will be positioned to understand and apply modern best practices and to influence future directions. The Box illustrates the Case of Dr. Smith, which continues throughout this Journal of Graduate Medical Education (JGME) special review series, considering the same question using different review methodologies.

Although SotA reviews are frequently published in peer-reviewed journals, there are few descriptions of how to conduct these knowledge syntheses, their markers of methodical rigor, and their reporting standards. We set out to address this gap by: (1) analyzing all publicly available and indexed methods-related publications describing SotA reviews, and (2) studying all SotA reviews (n=398) published between 2016 and 2020 to identify the foundational principles and techniques underpinning them.³  Through this work, we developed a 6-stage process for conducting SotA reviews,³  which aligns with the existing brief descriptions.^1,2,4-7  These 6 stages are summarized in a short how-to guide accompanying this article.⁸  Here, we explain the orienting premises that shape SotA reviews.

SotA literature reviews are founded on the principle that there is no single objectively true or correct synthesis of a body of literature. Instead, SotA reviews rest on the premise that literature is open for interpretation and that the context in which the review is conducted will shape the synthesis developed. SotA literature reviews are steeped in a relativist ontology: the nature of reality is socially and experientially informed and constructed. Consequently, SotA reviews do not require the literature included in the review to use identical methodology to support meta-analyses to generate a right answer. That is, not all findings synthesized in the review need to be carried out in the same way to enable cross-study data amalgamations. Instead—because SotA reviews assume that multiple different understandings of a phenomenon are available—this synthesis does not exclude research using different methodologies.

In terms of epistemology (the origins, nature, and limits of knowledge about reality), SotA literature reviews embrace subjectivism, the premise that knowledge generated from the review is a construction, not an objective fact. The knowledge generated through the review is value-dependent; it grows out of the subjective interpretations of the researchers who performed the synthesis. SotA reviews generate an interpretation of the literature informed by the expertise, experiences, and social context of the review team. Furthermore, the knowledge developed through SotA reviews is informed by the point in time when the review was conducted. A SotA review from 2000 reflects the contemporary knowledge of the year 2000; a SotA review from 2022 would report different knowledge reflecting that year's perspectives.

SotA literature reviews seek (1) to create a critical summary of contemporary thinking about a topic; (2) to describe historical progressions and patterns in the literature; (3) to discuss how such modern perspectives have evolved over time; and (4) to propose a direction the field could take moving forward. Further, the SotA review presents an argument for how the literature could be interpreted; it is not a definitive statement about how the literature should or must be understood. The purpose of the SotA review is to engage in this critical summary at a specific point in time; it highlights the pivot points shaping the historical development of a topic, the factors that informed those changes in understanding, and the ways of thinking about and studying the topic that could newly inform the generation of further insights. Ultimately, the purpose of SotA literature reviews is to create a 3-part argument: This is where we are now in our understanding of this topic. This is how we got here. This is where we could go next.

To illustrate, Schuwirth and van der Vleuten's article, “A History of Assessment in Medical Education,”⁹  offers a temporally organized overview of the evolving thinking in medical education about learner assessment. The authors describe how learner assessment was originally perceived as a problem of measurement, where the goal was to differentiate competent learners from incompetent ones. Historically, assessment was concerned with tool validity and replicability; human judgement was largely ignored. Even when assessment moved to include workplace-based assessment methods, the field continued to foreground assessment as a measurement problem. When human judgment was considered, the field focused on training assessors to minimize bias. Modern perspectives conceive of assessment as a whole system. Today, assessment data are integrated together to meaningfully triangulate data into a fair and defensible whole. Human judgement is recognized, but not as a bias to be mitigated. Instead, learners and assessors work together “to create a meaningful holistic narrative rather than a set of individual measurements.”⁹  The authors suggest that the future of learner assessment will continue to focus on determining if a learner possesses and can apply appropriate knowledge and skills; in addition, information technologies and the availability of big data will shape future assessment considerations. These technologies will also require a reexamination of the knowledge and skills that will be required of future clinicians. Schuwirth and van der Vleuten's article thus offers a SotA review by providing an interpretation of the past, present, and future of learner assessment.⁹ 

A significant contribution of a SotA review is the historical overview of how thinking about a phenomenon has changed over time. Such descriptions are particularly valuable for those exploring a new phenomenon or field of inquiry, and for those seeking to identify contemporary best practices and conceptualizations. Further, a SotA review provides a comprehensive time-based overview of a body of knowledge. Educators and researchers have an opportunity not only to assess past, present, and future trends, but also to characterize the unique shifts and patterns occurring over a specific period of time. Finally, the scope of a SotA review can extend beyond peer-reviewed literature.

The purpose and foundations upon which SotA reviews are built constrains them from providing a direct answer to specific, narrow research questions. They do not offer definitive answers to readers; instead, they are subjective reviews offering one interpretation of how the literature could be interpreted. Alternative interpretations exist. Moreover, the moment in history when the review is conducted and the specific review team engaging in the synthesis will shape the SotA review. Thus, reflexivity considerations by the team should be provided so that readers fully understand how the research team reached their conclusions.

While many knowledge syntheses have reporting standards, no such guidance exists for SotA reviews. SotA reviews offer interpretations of a specific body of literature; therefore, appraising the quality of the literature and preserving objectivity of the analysis processes is not relevant. Instead, indicators of the quality of a review are connected to its transparency, including considerations such as: How was the collection of articles included in the synthesis created? What inclusion and exclusion criteria controlled the selection? What reflexivity considerations shaped the perspectives of the authors? What contextual factors contributed to shaping the analysis? The final search strategy must be included in the manuscript so that others can replicate the process. The purpose of a replication would not be to confirm the interpretations offered in the SotA review; instead, it would be for another team of researchers to offer their unique interpretations and insights. Another consideration is the breadth of literature included in the review. It is advisable to incorporate a wide range of papers (eg, commentaries, research articles, grey literature) since part of the purpose of a SotA review is to identify how and when a field of inquiry took on its current state. Such information is not necessarily found only in peer-reviewed journal articles.

A SotA review can be deemed a success if it offers a coherent description regarding the current state of knowledge of a phenomenon: This is where we are now in our current understanding of this topic. This is how we got here. This is where we could go next.

Until very recently, SotA reviews were highly used but underdescribed: no robust methodologies were offered, and no information existed about their epistemological and ontological backgrounds. We hope to redress this gap. It is important when reading older SotA reviews to look for the 6 stages presented in the accompanying short article⁸  to help make meaning of the findings (additional resources are provided in the Table). We recommend that the structure of future SotA reviews clearly articulate these 6 stages so that researchers can more easily assess the interpretations offered within.