Building Blocks for Critical Reviews in Health Professions Education

Most health professions education (HPE) researchers eventually come across a problem they find themselves returning to repeatedly because the problem is messy and resists resolution. Getting “unstuck” and enabling progress often requires stepping back to think differently about how the issue is conceptualized. These are the problems for which critical reviews are most useful. Researchers conducting critical reviews use their expertise and judgment to look for alternative ways of thinking, by drawing together disparate ideas, empirical evidence, or theory. Getting unstuck in a particular way of thinking often requires finding key pieces of information that can shift research and practice; these key ideas seldom originate from a single defined literature and, thus, rarely lend themselves to clearly defined searches. Hence, in critical reviews, being systematic is less important than using flexibility, creativity, and judgement to deeply explore literatures while selectively highlighting articles and perspectives that advance thinking on the topic.¹ 

As a result, there is no methodological “recipe” for critical reviews—this is both part of their charm and their challenge. Here we offer advice on the building blocks of a critical review (Figure). We aim to provide guidance without loss of the disciplinary breadth and methodological flexibility that make critical reviews valuable. These building blocks are often interwoven and iterative such that there are likely to be moments when reviewers consider 2 or more elements simultaneously and other moments when they benefit from returning to an element to layer on a new perspective.

Critical reviews are often prompted by a sense that there must be a better way of thinking about a problem, yet the road to a new conceptualization is unclear. To address these problems, critical reviewers use a constructivist approach, which draws on their unique insights and expertise and promotes developing a nuanced and meaningful analysis. Because the journey is often long and filled with false starts, it can be difficult to pinpoint exactly where the review began. Nonetheless, as with other forms of research, it is imperative that critical reviewers take the time to generate a clear, explicit focus.

Many critical reviews start with efforts to progressively refine the description of a problem of interest, clarify its importance, and probe existing explanations or models that seem to be missing something. Often this involves describing your thoughts to others and monitoring their reactions to develop a formal research question or purpose statement that can guide and constrain your review. Losing focus, even if there is much more to explore, risks meandering in a way that will cause a critical review to seem like a random collection of thoughts.

Discuss your problem “early and often”²  with a wide array of individuals. The “elevator pitch” offers a useful model, but should not be considered a sales job: it is a launching point from which to find out how clinicians, educators, policymakers, and researchers from different backgrounds think about the problem of interest. Use these conversations to refine arguments about relevance to practice, focus your question in a way that might inform policy, and identify sources of evidence to be gathered from different disciplines, theories, and methodologies.

While refining your problem through discussion, watch for individuals whose expertise and insight might provide a particularly useful addition to your team. Not every person you consult will be a good co-investigator for your review, but strive to fill in gaps in expertise, and force yourself to think outside the box by engaging others whose views and backgrounds are complementary.

Taking a meaningful dive into topics that benefit from a critical review typically involves a cyclical and iterative process. This process includes conversing with experts, testing various literatures for their relevance and potential impact on the field's approach to the problem, and honing the research question. Searches are thus intentionally selective rather than systematic. Instead of attempting to marshal every study on a topic, critical reviewers target specific perspectives or findings. Clarity of thought is more important than replicability and, while you should justify where you looked and why, it is rarely helpful to detail every search term and database considered.

As the review progresses, early efforts to use an elevator pitch to get feedback on your topic can morph into more intentional data gathering aimed at ensuring that your explorations of unfamiliar literature or theories resonate with experts in diverse fields. Diversity in disciplinary and clinical perspectives is key, given that new and useful approaches are unlikely to use the language (and search terms) already in circulation around your topic.

Given the broad nature of many critical reviews and their tendency to span multiple disciplines, it can be difficult to zero in on the most valuable sources without wading through reams of off-topic or unhelpful publications. An academic librarian can be beneficial if they are willing and able to talk through the problem you are trying to better understand, rather than simply requesting search terms.

Hand-searching reference lists in the most relevant sources can be helpful for identifying important works and related literature, particularly those from unfamiliar fields. High-quality literature reviews are particularly useful resources. Search engines like Google Scholar can be used to track citations, to gain a sense of which publications are most cited and where the field has gone since their publication, rather than risk relying on outdated information.

While searching, critical reviewers appraise the literature they uncover. Rather than screening articles through a priori inclusion and exclusion criteria, many decisions about what to include are nuanced and based on the research team's unique expertise and judgement regarding source quality and relevance to the research question. These decisions require deep engagement with the topic. As a result, appraisal can rarely be offloaded to a research assistant. Though such judgments can be difficult to represent briefly in the resultant publication, reviewers should keep a record of the rationale for their decisions while being reflexive and transparent in their reporting.

When navigating the literature, you will need to make judgements regarding the quality and value of uncovered resources, by drawing on expertise within and outside your research team, rather than implementing specific quality criteria or limiting yourself to preconceptions of gold standard study designs. Critical reviews can include any type of source, from systematic reviews to editorials, if the information is deemed to contribute to the reviewers' evolving conceptualization of the topic.

Include only the sources that speak best to your research question. As in the data generation phase, you need not include every source. Decisions may change over the course of your exploration. You may choose to include a source initially, but then deem it less relevant later (or vice versa), as you continue to refine your thinking through reading and discussion with others. Better to delete even extensively annotated bibliographies than to force things in that do not fit.

Even after appraisal for quality and relevance, critical reviewers are inevitably limited to sharing only a sample of the sources they encountered. The sample should prioritize works that offer the evidence or argument that was most influential and effective in shaping the reviewer's understanding, but care should be taken to ensure it is representative of the broader literature or theory from which it was drawn. As a result, the review team should be watchful for opportunities to expand or shift conversations in the field by demonstrating how their search has expanded and shifted their own thinking.

“Significance” is relative to the source material and research question. It can reflect the sources deemed to best represent a particular concept, foundational works from the discipline being explored, or articles deemed most likely to offer compelling shifts in the conversation about HPE.

There is unlikely to be a point at which you run out of theories to consider or are unable to add new terms to your searches. Rather, you must determine sample sufficiency based on a sense that new explorations begin to yield concepts you have seen before and conversations with others suggest you have a thorough understanding of the phenomenon. Your exploration ends not when you have found a certain number of articles, but when your results achieve your goals (ie, your synthesis offers a meaningful contribution). Concepts from qualitative research such as “theoretical sufficiency,”³  “saturation,”⁴  or “information power,”⁵  as congruent with your methodology, may help to clarify and report these decisions.

As a genre, critical reviews do not offer a predetermined analytic protocol. Thus, ensuring that the research question, analytic methods, and research products align is essential and cannot be assumed.

Consider drawing on techniques from qualitative research methodologies or other review types to support your process. Your approach may be inductive, generating new theoretical perspectives from identified articles, or more deductive, applying existing concepts or theories to interpret a topic differently.⁶  Some critical reviewers engage specific analysis methods relevant to their research question, such as content analysis⁷  or textual analysis⁸ ; others take a more generic approach.⁹  Either way, a thoughtful approach to and articulation of the process is essential to ensuring and communicating analytic rigor.

Consider the type of contribution you are hoping to make. Critical reviews tend to take on a practice-relevant problem, to propose a new theoretical perspective, conceptual framework, or best practice to shift the way others in the field think.¹⁰  As with all phases of the review, it's important to think through the end product of your analysis. What methods will lead from exploration to creation of a cohesive, sensible, and compelling research output?

Critical reviews offer a valuable tool for tackling difficult problems,¹¹  but their flexibility comes with challenges, and there is little existing methodological guidance. Critical reviewers must exercise their judgement throughout the process and often rely heavily on approaches and techniques from qualitative research. As a result, critical reviewers must rely on reflexivity to ensure rigor¹²  and transparency to communicate their work.¹³