Educational Outcomes of Diversity Curricula in Graduate Medical Education

Graduate medical education about health care equity is an important step toward eliminating disparities. In recognition of this, the Accreditation Council for Graduate Medical Education (ACGME) has enacted several changes to the Common Program Requirements addressing issues of diversity, equity, and inclusion, as well as the ACGME Equity Matters initiative.¹  In order to meet the growing demand for education in this space, residency programs across all specialties have introduced topics such as diversity, equity, and inclusion (DEI) in health care into their formal curricula, which vary widely by regional patient population needs, medical specialty, and scope of practice.

Despite the educational efforts to improve resident physician knowledge, skills, and attitudes, the sum of the published literature describing outcomes, such as learner changes in behavior or impact on patient care, has not been well characterized to date. We conducted a needs assessment of the existing literature to identify gaps and inform future efforts toward designing curricula in this space. The objective of our study was to perform a scoping review of the existing literature on curricular interventions in graduate medical education regarding DEI with a focus on educational outcomes using the Kirkpatrick Model²  as a framework.

We conducted a review of the medical education literature published prior to August 12, 2021, within the MEDLINE database, using methodology described by Cook and West and following PRIMSA guidelines for scoping reviews.^3,4  Using an iterative process among the authors, one with expertise in health equity (C.O.) and one in medical education research (A.S.C.), we developed inclusion criteria and search terms. Our population of interest was resident physicians in graduate medical education programs of all specialties. We looked specifically for curricular interventions with measurable and reported outcomes focused on the topics of DEI in medical education and health care. Using the Medical Subject Heading “graduate medical education” and the search term “curriculum,” we cross-referenced the literature for the following search terms: diversity, inclusion, health equity, inequity, antiracism, cultural competency, critical race theory, implicit bias, microaggressions, and racism. Titles and abstracts of identified articles were screened for relevance by the first author of this study (A.S.C.). Intra-rater agreement was performed following the initial selection, and no additional studies were included or excluded. Publications were included for final analysis if they had a clearly stated and primary curricular objective focused specifically on DEI with reported outcomes. All other publication types were excluded, including abstracts, opinion pieces, and literature reviews. References of relevant articles were subsequently screened by one member of the study team (A.C., S.D., E.P., R.G., J.A., or C.O.) for additional publications that may have been missed by the search protocol.

For studies that met inclusion criteria through the search strategy identified above, the full article was reviewed by one member of the study team (A.S.C., A.C., S.D., E.P., R.G., J.A., or C.O.) to collect additional data including publication year, specialty, study population, number of learners, type of study, type of curriculum and brief description, measured outcomes, and main findings. Statistical findings were also included if available. Using previously published methodology, outcomes were classified by the authors according to the Kirkpatrick Model, which categorizes educational outcomes using increasing levels of complexity and impact.⁵  In this model, Level 1=reaction/satisfaction, Level 2=learning, Level 3=behavioral change, and Level 4=patient outcomes.²  Outcomes for each study were determined by the first author and one other study team member (A.C., S.D., E.P., R.G., J.A., or C.O.). Any disagreements were resolved through discussion and consensus.

This study was determined to be exempt by the Institutional Review Board of Maimonides Medical Center.

Our initial search strategy yielded 336 results. After screening titles and abstracts for inclusion, 60 articles remained for review. These articles were read in their entirety and assessed for eligibility based on the inclusion criteria. Nineteen articles were included for final analysis (Figure 1). No additional studies were identified through a review of references. Publication dates ranged from 2000 to 2021, with 74% (14 of 19) of included studies having been published since 2017 (Table).

We found a total of 899 learners who participated in curricular interventions that met our inclusion criteria. Internal medicine represented the largest number of studies from a single specialty (37%, 7 of 19), followed by family medicine (32%, 6 of 19) and emergency medicine (16%, 3 of 19; Figure 2). One study did not specify the specialties of the residents involved.

While the majority of studies involved residents from a single program (63%, 12 of 19), several included residents from multiple programs within the same institution (32%, 6 of 19). One program studied residents in the same specialty (general surgery) from 3 different institutions. The number of total learners participating in each curriculum ranged from 10 to 181. A few studies also included learners who were not residents, such as medical students, fellows, faculty, and non-physician hospital staff.

Educational methods ranged from online modules to single workshops to multiyear longitudinal curricula. The most common educational method used was facilitated small group discussion (79%, 15 of 19). Other educational methods included formal lectures (53%, 10 of 19), simulation cases or standardized patients (21%, 4 of 19), and online self-directed modules (16%, 3 of 19). A few unique methods included a journal club, field trips or tours, and self-reflective writing. Many curricula incorporated multiple different educational methods. The majority of curricular interventions occurred as a single session (37%, 7 of 19), with the length of the session ranging from 90 minutes to 3 hours. The remainder occurred as multiple sessions over a span of several months to a year. One study described a 3-year longitudinal multi-modal curriculum.

The majority of studies reported Kirkpatrick Level 1 or 2 outcomes (84%, 16 of 19). Level 1 outcomes were most commonly assessed using post-intervention satisfaction surveys. Two studies conducted focus groups to assess learner reaction. Learning was most commonly demonstrated through a comparison of pre- and post-intervention knowledge testing. Two studies reported Level 3 outcomes using simulation and scholarly output of learners. One study included patient satisfaction surveys, a measure of impact on patient outcomes due to the curricular intervention.

Overall, the studies included for analysis demonstrated feasibility of implementation, and their curricula were positively received by learners. Eleven studies were able to show an improvement in knowledge or behavior with regards to DEI in medical education and health care.

Despite the recognized importance of education on DEI, we found a paucity of evidence describing meaningful educational outcomes for residents.

We found only 19 studies that met our inclusion criteria. Despite our search of the peer-reviewed literature accessible online any time prior to 2021, the earliest study we found was published in 2000, and the overwhelming majority were published in just the last 5 years. This is both disheartening and encouraging. Quality medical education research in this area has been sparse in the past but seems to be increasing in line with our greater sociocultural awareness of the importance of these issues and changes to the ACGME Common Program Requirements. It is possible that many more educational interventions are in their early phases of development and implementation, and thus not captured in this review, which would be consistent with the exponential growth we have observed to date.

Specialties at the front line of care access, such as internal medicine and family medicine, had the highest number of studies represented in this review of the literature. Interestingly, a greater number of physicians who identify as Black, Hispanic, or Native American currently practice primary care (41.4%, 36.7%, and 41.5%, respectively) compared to physicians who identify as White (30.6%).²⁵  The ACGME recognizes and provides accreditation for 28 specialties, and it is worth noting that only 8 of these specialties have published educational interventions with measured outcomes. Physicians of all specialties have a responsibility to provide equitable and compassionate care to patients of diverse backgrounds, and there is still much work to be done.

We found that the most frequently used educational method was small group facilitated discussion. We also observed a surprising number of methods outside of lecture-based didactics, such as field trips, tours, written self-reflection, electronic or online learning, and journal clubs. Issues such as DEI are highly nuanced and complex. The effectiveness in delivering this content to improve the knowledge, skills, and attitudes of resident physicians is profoundly affected by each learner's unique background and life experience. Interactive methods, such as small group facilitated discussion, have been demonstrated to be well-received by learners and more effective than traditional lectures for complex topics.²⁶ ,²⁷ 

We used the Kirkpatrick Model to characterize the studies included in our analysis, following the methodology of other scoping reviews of curricular interventions.⁷ ,²⁸  The majority of the studies reported Kirkpatrick Level 1 or 2 outcomes. We recognize that the Kirkpatrick levels are not a definitive hierarchy of quality, and that learner satisfaction may be the primary goal depending on the objectives of the educational intervention. Only 3 studies reported a Kirkpatrick Level 3 or 4. We acknowledge the difficulty in assessing behavioral change and patient outcomes. However, these types of outcomes are important in order to inform educators of best practices for training resident physicians. Further study is needed to understand how resident education in this area can impact patient care.

We felt that a scoping review, rather than a comprehensive systematic review, was a more appropriate methodology to map the literature. It is possible that we would have identified additional studies for final inclusion had we used more than one author to screen titles and abstracts. We instead employed a defined set of screening criteria developed by 2 authors and engaged the screening author to ascertain intra-rater agreement for inclusion. Interventions may also exist in the non-peer reviewed literature or in the gray literature, including abstracts, conference proceedings, reports, or dissertations.

Many types of educational interventions that broadly prepare residents to care for patients of different backgrounds were not included in our analysis. We focused our inclusion criteria on interventions with a clearly stated and primary curricular objective focused on DEI in health care or medical education. All other studies fell outside the scope of this review, such as studies on the use of interpreters, medical management of special populations, and development of global health rotations. It is also possible that we may have missed relevant studies if our key concepts of interest were not included explicitly in the title or abstract.

In this scoping review of the literature, we found a small number of studies of curricular interventions for resident physicians that directly address DEI in medical education and health care. Specialties at the front line of care access had the greatest number of studies represented in this sample. Overall, these studies employed a wide array of interactive educational methods, demonstrated feasibility of implementation, and were positively received by learners.

The authors would like to thank Zakir Gulam for research support.