Background

As the Accreditation Council for Graduate Medical Education (ACGME) began to ask programs to report their efforts surrounding diversity, equity, and inclusion (DEI), program directors felt ill prepared to evaluate their programs and measure change.

Objective

To develop a tool that would allow graduate medical education (GME) programs to evaluate the current state of DEI within their residencies, identify areas of need, and track progress; to evaluate feasibility of using this assessment method within family medicine training programs; and to analyze and report pilot data from implementation of these milestones within family medicine residency programs.

Methods

The Association of Family Medicine Residency Directors (AFMRD) Diversity and Health Equity (DHE) Task Force developed a tool for program DEI evaluation modeled after the ACGME Milestones. These milestones focus on DEI assessment in 5 key domains: Institution, Curriculum, Evaluation, Resident Personnel, and Faculty Personnel. After finalizing a draft, a pilot implementation of the milestones was conducted by a convenience sample of 10 AFMRD DHE Task Force members for their own programs.

Results

Scores varied widely across surveyed programs for all milestones. Highest average scores were seen for the Curriculum milestone (2.65) and the lowest for the Faculty Personnel milestone (2.0). Milestone assessments were completed within 10 to 40 minutes using various methods.

Conclusions

The AFMRD DEI Milestones were developed for program assessment, goal setting, and tracking of progress related to DEI within residency programs. The pilot implementation showed these milestones were easily used by family medicine faculty members in diverse settings.

In 2020, the Accreditation Council for Graduate Medical Education (ACGME) released Common Program Requirements for residency programs to address diversity, equity, and inclusion (DEI) without providing specific benchmarks.1  Available DEI templates for organizations2-6  neither meet the needs of graduate medical education (GME), especially in terms of strategies to reduce bias in learner assessment,7-11  resident recruitment,12,13  and faculty promotion,14-16  nor provide a framework to measure progress. In March 2019, the Association of Family Medicine Residency Directors (AFMRD) polled its membership and found that 50% of 220 respondents answered “no” or “unsure” to the statement, “I feel confident implementing the common program requirement change pertaining to diversity.”17 

The AFMRD Board created the Diversity and Health Equity (DHE) Task Force in 2019 to advance antiracism, diversity, and inclusion in family medicine residency training. This task force identified milestones18-20  as a familiar tool all programs use to assess residents' progress toward academic and professional competencies and to set goals for growth. The task force sought to create a similar guide for programs—not individuals—to evaluate and track progress toward longitudinal DEI goals across multiple domains essential to residency training to assist in (1) accurate evaluation of the current state of DEI within the program, (2) identification of needs, and (3) progressive goal setting for programs at different stages of this work. In this article, we describe the creation of this DEI tool, results from a pilot implementation in family medicine residency programs, and validity evidence21  to support its use.

The AFMRD DHE Task Force is composed of nationally representative family medicine program leaders with experience in health equity and DEI chosen after a competitive selection process. As widely accepted standardized tools do not exist for DEI evaluation in GME training, the task force reviewed available tools from the medical field2,3  and business world4-6  designed to enhance DEI and discussed opportunities for their use along with barriers limiting implementation. These discussions highlighted the utility of milestones, a tool familiar to program faculty.

DEI milestone development followed a blueprint similar to that used by the ACGME for residency milestones.18  Initial meetings focused on identification of key DEI domains for GME programs. Understanding the range of factors impacting DEI within GME, the task force ultimately focused on the people in the program, the content being taught, and the system in which programs are working. The task force devised 5 domains foundational for DEI in residency programs: (1) Institution, (2) Curriculum, (3) Evaluation, (4) Resident Personnel, and (5) Faculty Personnel. Smaller workgroups within the committee drafted content for each domain, adapting the ACGME template18-20  for progression across each level to focus on program evaluation rather than resident assessment. Level 1 reflects programs early in their DEI work in a specific domain, while Level 5 reflects local/national leaders that could be used as exemplars. Progression for the resident and faculty personnel milestones begins with changes in recruitment (Level 1) and progresses to changes in program leadership (Level 5) to reflect steps needed to build sustained positive change. Content was developed based on examples from exemplary programs within family medicine and the collective experience and expertise of task force members. Nationally recognized senior family medicine faculty with DEI and medical education expertise also reviewed early drafts.

After finalizing a working draft, the task force presented the milestones to more than 400 family medicine residency leaders at the American Academy of Family Physicians (AAFP) virtual Residency Leadership Summit in March 2021. A 2-week open commentary period for all AFMRD members followed this presentation. This peer feedback was incorporated into the piloted version of the DEI milestones (Figure and see online supplementary data).

A pilot implementation of the DEI milestones was conducted from April to May 2021. The pilot cohort—a convenience sample of the residency programs of the AFMRD DHE Task Force members—consisted of 10 family medicine programs from across the country in urban, suburban, and rural areas working in academic and community settings. Strict guidelines for milestone completion were not given; instead, the committee desired programs to complete the milestones through different platforms allowing for input on tool utilization. The task force determined this strategy, as within GME programs Clinical Competency Committees utilize different methods to assess an individual resident's milestones. Pilot programs completed a form, similar to the Table, providing demographic information, completion time, persons completing evaluation, and milestone scores. An additional box was provided for comments on clarity, ease of use, and impact of the milestone tool in helping set DEI-focused goals for their program.

As this pilot focused on instrument development rather than human subjects research, it qualified for Institutional Review Board exemption.

Ten program directors and their faculty completed the milestones, with 3 programs choosing to include residents in the process (Table). Program Evaluation Committee, Clinical Competency Committee, or program faculty meetings were settings for milestone completion. Average time spent completing milestones was 20 minutes. Scores varied widely across surveyed programs for all milestones, with the highest average scores being for the Curriculum milestones (2.65) and lowest for the Faculty Personnel milestones (2.0). A range of milestone scores (0–4) was seen within and across programs. Some programs struggled with scoring, choosing a range instead of a number for a milestone, noting that they satisfied one component of various levels but failed to satisfy all components of any one level. Five of 10 programs provided comments, which included that the tool provided a benchmark assessment, did not take a significant amount of time, and was easy to incorporate into meetings.

This pilot study of the newly developed DEI milestones for programs demonstrated that the tool is easy to use, takes an average of 20 minutes to score, and can be completed by various preexisting committees. Despite having interested core faculty, pilot programs were still at the beginning phases of their DEI work and had much room for improvement. DEI milestone development sought content validity21  by using nationally recognized content experts in family medicine and medical education. However, since we were unable to find any comparable tools, we could not correlate responses with a previously developed tool.

The familiarity of using milestones for assessment may ease use by program directors, providing a unified way to assess the overall state of DEI in GME. Subsequent studies with more family medicine and other GME programs are needed to further evaluate milestones' utility in promoting meaningful change toward advancing DEI in physician training. National milestone data would be immensely valuable in understanding the state of DEI across GME, developing best practices, and facilitating collaboration across programs and specialties. This data can be used when advocating for the support necessary to advance DEI within health care systems.

This study is not without limitations. Response process validity was limited since programs were not given uniform instructions for how to complete the milestone tool and each of the pilot programs had a task force member with intimate knowledge of the tool and motivation to use it. The convenience sample used for the pilot was also not necessarily representative of all programs in family medicine. Finally, assessment and evaluation are occasionally used interchangeably in the DEI milestone tool itself, which may have caused confusion.

The AFMRD DEI milestones were implemented in less than 1 hour in the pilot study and generated initial validity evidence for baseline assessment across 5 key domains.

The authors would like to acknowledge the entire AFMRD Diversity and Health Equity Task Force who collaborated on the creation of the Diversity, Equity, and Inclusion Milestones.

1. 
Accreditation Council for Graduate Medical Education.
Common Program Requirements (Residency)
.
2. 
American Public Health Association.
Oyetunde
KMO,
Boulin
A,
Holt
J.
Equity Diversity Inclusion Action Toolkit for Organizations.
3. 
American Academy of Family Physicians.
Starfield II Health Equity Summit
.
Primary Care's Role in Achieving Health Equity: A Guidebook to the Health Equity Curricular Toolkit.
4. 
Housing Development Consortium Organization.
Joe
M,
Waddy
R.
Racial Equity Toolkit: Applying a Racial Equity Lens to Your Organization.
5. 
Racial Equity Tools. Organizational Assessment Tools and Resources.
6. 
Crossroads Ministry.
Jackson
B,
Hardiman
R.
Continuum on Becoming an Anti-Racist Multicultural Organization.
7. 
Low
D,
Pollack
SW,
Liao
ZC,
et al
Racial/ethnic disparities in clinical grading in medical school
.
Teach Learn Med
.
2019
;
31
(5)
:
487
496
.
8. 
Boatright
D,
Ross
D,
O'Connor
P,
Moore
E,
Nunez-Smith
M.
Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society
.
JAMA Intern Med
.
2017
;
177
(5)
:
659
665
.
9. 
Rojek
AE,
Khanna
R,
Yim
JWL,
et al
Differences in narrative language in evaluations of medical students by gender and under-represented minority status
.
J Gen Intern Med
.
2019
;
34
(5)
:
684
691
.
10. 
Zhang
N,
Blissett
S,
Anderson
D,
O'Sullivan
P,
Qasim
A.
Race and gender bias in internal medicine program director letters of recommendation
.
J Grad Med Educ
.
2021
;
13
(3)
:
335
344
.
11. 
Osseo-Asare
A,
Balasuriya
L,
Huot
SJ,
et al
Minority resident physicians' views on the role of race/ethnicity in their training experiences in the workplace
.
JAMA Netw Open
.
2018
;
1
(5)
:
e182723
.
12. 
Agawu
A,
Fahl
C,
Alexis
D,
et al
The influence of gender and underrepresented minority status on medical student ranking of residency programs
.
J Natl Med Assoc
.
2019
;
111
(6)
:
665
673
.
13. 
Wusu
MH,
Tepperberg
S,
Weinberg
JM,
Saper
RB.
Matching our mission: a strategic plan to create a diverse family medicine residency
.
Fam Med
.
2019
;
51
(1)
:
31
36
.
14. 
Nunez-Smith
M,
Ciarleglio
M,
Sandoval-Schaefer
T,
et al
Institutional variation in the promotion of racial/ethnic minority faculty at US medical schools
.
AM J Pub Health
.
2012
;
102
(5)
:
852
858
.
15. 
Society of General Internal Medicine
.
Hasnain
M,
Goss
E,
Yepes-Rios
M,
et al
Ten Tips for Dismantling Racism: A Roadmap for Ensuring Diversity, Equity, and Inclusion Across the Academic Continuum.
16. 
American Surgical Association.
Ensuring Equity, Diversity, and Inclusion in Academic Surgery.
17. 
Association of Family Medicine Residency Directors.
2020 Annual Quick Hitter Survey.
Accessed June 21, 2021. https://www.afmrd.org/
18. 
Accreditation Council for Graduate Medical Education.
Edgar
L,
McLean
S,
Hogan
SO,
Hamstra
S,
Holmboe
ES.
The Milestones Guidebook.
19. 
Andolsek
KM,
Jones
MD
Jr,
Ibrahim
H,
Edgar
L.
Introduction to the Milestones 2.0: assessment, implementation, and clinical competency committees supplement
.
J Grad Med Educ
.
2021
;
13
(suppl 2)
:
1
4
.
20. 
Manious
AG,
Fang
B,
Peterson
L.
Competency assessment in family medicine residency: observations, knowledge-based examinations, and advancement
.
J Grad Med Educ
.
2017
;
9
(6)
:
730
734
.
21. 
Sullivan
GM.
A primer on the validity of assessment instruments
.
J Grad Med Educ
.
2011
;
3
(2)
:
119
120
.

Author notes

Editor's Note: The online version of this article contains the Diversity, Equity, and Inclusion Competency Milestones.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

The rubric in this article was presented at the virtual AAFP Residency Leadership Summit, March 4–6, 2021.

Supplementary data