Diversity, Equity, and Inclusion Milestones: Creation of a Tool to Evaluate Graduate Medical Education Programs

In 2020, the Accreditation Council for Graduate Medical Education (ACGME) released Common Program Requirements for residency programs to address diversity, equity, and inclusion (DEI) without providing specific benchmarks.¹  Available DEI templates for organizations^2-6  neither meet the needs of graduate medical education (GME), especially in terms of strategies to reduce bias in learner assessment,^7-11  resident recruitment,^12,13  and faculty promotion,^14-16  nor provide a framework to measure progress. In March 2019, the Association of Family Medicine Residency Directors (AFMRD) polled its membership and found that 50% of 220 respondents answered “no” or “unsure” to the statement, “I feel confident implementing the common program requirement change pertaining to diversity.”¹⁷ 

The AFMRD Board created the Diversity and Health Equity (DHE) Task Force in 2019 to advance antiracism, diversity, and inclusion in family medicine residency training. This task force identified milestones^18-20  as a familiar tool all programs use to assess residents' progress toward academic and professional competencies and to set goals for growth. The task force sought to create a similar guide for programs—not individuals—to evaluate and track progress toward longitudinal DEI goals across multiple domains essential to residency training to assist in (1) accurate evaluation of the current state of DEI within the program, (2) identification of needs, and (3) progressive goal setting for programs at different stages of this work. In this article, we describe the creation of this DEI tool, results from a pilot implementation in family medicine residency programs, and validity evidence²¹  to support its use.

The AFMRD DHE Task Force is composed of nationally representative family medicine program leaders with experience in health equity and DEI chosen after a competitive selection process. As widely accepted standardized tools do not exist for DEI evaluation in GME training, the task force reviewed available tools from the medical field^2,3  and business world^4-6  designed to enhance DEI and discussed opportunities for their use along with barriers limiting implementation. These discussions highlighted the utility of milestones, a tool familiar to program faculty.

DEI milestone development followed a blueprint similar to that used by the ACGME for residency milestones.¹⁸  Initial meetings focused on identification of key DEI domains for GME programs. Understanding the range of factors impacting DEI within GME, the task force ultimately focused on the people in the program, the content being taught, and the system in which programs are working. The task force devised 5 domains foundational for DEI in residency programs: (1) Institution, (2) Curriculum, (3) Evaluation, (4) Resident Personnel, and (5) Faculty Personnel. Smaller workgroups within the committee drafted content for each domain, adapting the ACGME template^18-20  for progression across each level to focus on program evaluation rather than resident assessment. Level 1 reflects programs early in their DEI work in a specific domain, while Level 5 reflects local/national leaders that could be used as exemplars. Progression for the resident and faculty personnel milestones begins with changes in recruitment (Level 1) and progresses to changes in program leadership (Level 5) to reflect steps needed to build sustained positive change. Content was developed based on examples from exemplary programs within family medicine and the collective experience and expertise of task force members. Nationally recognized senior family medicine faculty with DEI and medical education expertise also reviewed early drafts.

After finalizing a working draft, the task force presented the milestones to more than 400 family medicine residency leaders at the American Academy of Family Physicians (AAFP) virtual Residency Leadership Summit in March 2021. A 2-week open commentary period for all AFMRD members followed this presentation. This peer feedback was incorporated into the piloted version of the DEI milestones (Figure and see online supplementary data).

A pilot implementation of the DEI milestones was conducted from April to May 2021. The pilot cohort—a convenience sample of the residency programs of the AFMRD DHE Task Force members—consisted of 10 family medicine programs from across the country in urban, suburban, and rural areas working in academic and community settings. Strict guidelines for milestone completion were not given; instead, the committee desired programs to complete the milestones through different platforms allowing for input on tool utilization. The task force determined this strategy, as within GME programs Clinical Competency Committees utilize different methods to assess an individual resident's milestones. Pilot programs completed a form, similar to the Table, providing demographic information, completion time, persons completing evaluation, and milestone scores. An additional box was provided for comments on clarity, ease of use, and impact of the milestone tool in helping set DEI-focused goals for their program.

As this pilot focused on instrument development rather than human subjects research, it qualified for Institutional Review Board exemption.

Ten program directors and their faculty completed the milestones, with 3 programs choosing to include residents in the process (Table). Program Evaluation Committee, Clinical Competency Committee, or program faculty meetings were settings for milestone completion. Average time spent completing milestones was 20 minutes. Scores varied widely across surveyed programs for all milestones, with the highest average scores being for the Curriculum milestones (2.65) and lowest for the Faculty Personnel milestones (2.0). A range of milestone scores (0–4) was seen within and across programs. Some programs struggled with scoring, choosing a range instead of a number for a milestone, noting that they satisfied one component of various levels but failed to satisfy all components of any one level. Five of 10 programs provided comments, which included that the tool provided a benchmark assessment, did not take a significant amount of time, and was easy to incorporate into meetings.

This pilot study of the newly developed DEI milestones for programs demonstrated that the tool is easy to use, takes an average of 20 minutes to score, and can be completed by various preexisting committees. Despite having interested core faculty, pilot programs were still at the beginning phases of their DEI work and had much room for improvement. DEI milestone development sought content validity²¹  by using nationally recognized content experts in family medicine and medical education. However, since we were unable to find any comparable tools, we could not correlate responses with a previously developed tool.

The familiarity of using milestones for assessment may ease use by program directors, providing a unified way to assess the overall state of DEI in GME. Subsequent studies with more family medicine and other GME programs are needed to further evaluate milestones' utility in promoting meaningful change toward advancing DEI in physician training. National milestone data would be immensely valuable in understanding the state of DEI across GME, developing best practices, and facilitating collaboration across programs and specialties. This data can be used when advocating for the support necessary to advance DEI within health care systems.

This study is not without limitations. Response process validity was limited since programs were not given uniform instructions for how to complete the milestone tool and each of the pilot programs had a task force member with intimate knowledge of the tool and motivation to use it. The convenience sample used for the pilot was also not necessarily representative of all programs in family medicine. Finally, assessment and evaluation are occasionally used interchangeably in the DEI milestone tool itself, which may have caused confusion.

The AFMRD DEI milestones were implemented in less than 1 hour in the pilot study and generated initial validity evidence for baseline assessment across 5 key domains.

The authors would like to acknowledge the entire AFMRD Diversity and Health Equity Task Force who collaborated on the creation of the Diversity, Equity, and Inclusion Milestones.