ABSTRACT

Background

Many programs struggle to recruit, select, and match a diverse class of residents, and the most effective strategies for holistic review of applications to enhance diversity are not clear.

Objective

We determined if holistic pediatric residency application review guided by frameworks that assess for bias along structural, interpersonal, and individual levels would increase the number of matched residents from racial and ethnic groups that are underrepresented in medicine (UiM).

Methods

Between 2017 and 2020, University of California San Francisco Pediatrics Department identified structural, interpersonal, and individual biases in existing selection processes and developed mitigation strategies in each area. Interventions included creating a shared mental model of desirable qualities in residents, employing a new scoring rubric, intentional inclusion of UiM faculty and trainees in the selection process, and requiring anti-bias training for everyone involved with recruitment and selection.

Results

Since implementing these changes, the percentage of entering interns who self-identify as UIM increased from 11% in 2015 to 45% (OR 6.8, P = .008) in 2019 and to 35% (OR 4.6, P = .035) in 2020.

Conclusions

Using an equity framework to guide implementation of a pediatric residency program's holistic review of applications increased the numbers of matched UiM residents over a 3-year period.

Objectives

Can applying frameworks that assess for bias along structural, interpersonal, and individual levels in the residency selection process help increase the number of matched residents from racial and ethnic groups that are underrepresented in medicine (UiM)?

Findings

Recognizing and addressing biases in the residency selection process along structural, interpersonal, and individual levels of oppression resulted in an increase in percentage of entering interns who self-identify as UiM.

Limitations

This work was done at a large urban academic medical center and may not be generalizable to smaller programs.

Bottom Line

Residency programs must consider not only the interpersonal and individual biases that impact selection, but also must recognize and address the structural biases that can be built into residency selection processes.

Introduction

Despite the evidence affirming the benefits of a diverse workforce in pediatrics,1  residency programs have struggled to recruit residents from backgrounds that are underrepresented in medicine (UiM). Many factors contribute to a lack of diversity in medicine, including a relatively low number of UiM applicants, lack of an inclusive climate in medicine, and bias in the recruitment and selection process.2 

Multiple studies demonstrate that bias influences residency selection.39  To ensure an equitable recruitment and selection process, the Association of American Medical Colleges (AAMC) suggests the use of holistic review, defined as “a flexible, individualized way of assessing an applicant's capabilities by which balanced consideration is given to experiences, attributes, and academic metrics.”10  Because holistic review is mission driven, its implementation varies at each institution. Effective implementation of holistic review requires an understanding of the different ways in which bias can manifest.

The National Equity Project Lens of Systemic Oppression provides a helpful framework to understand the levels on which bias acts.11  The framework describes 3 levels of oppression—structural, interpersonal, and individual. Structural oppression is comprised of the institutional practices that create and perpetuate inequitable outcomes. Interpersonal oppression relates to individual interactions that perpetuate bias. In individual oppression, one's personal beliefs and actions—even when unconscious—perpetuate bias.

Prior to 2015, the University of California San Francisco (UCSF) Pediatric Residency Program had relatively few UiM residents. We sought to determine whether implementing holistic review using an equity framework that assesses for bias in our residency selection process along structural, interpersonal, and individual levels would increase the number of matched residents who self-identify as UiM.

Methods

UCSF Pediatrics is a large, academic, urban residency program with 84 residents. To inform implementation of holistic review, the intern selection faculty team (associate program director [J.M.] with 25% FTE support for recruitment and a former chief resident [R.B.] with 30% FTE support for recruitment) worked with the department of pediatrics diversity director (A.M.) to identify potential sources of bias at each level, described by the systems of oppression framework. We collaborated with the residency program director to develop approaches to mitigate these biases. The majority of this work occurred during the fall and winter of 2017, with ongoing changes and adjustments made annually based on feedback from residents, faculty, and applicants.

We highlight examples of strategies to mitigate bias below. Additional strategies are listed in the Table.

Table

Strategies to Mitigate Bias Along Individual, Interpersonal, and Structural Levels

Strategies to Mitigate Bias Along Individual, Interpersonal, and Structural Levels
Strategies to Mitigate Bias Along Individual, Interpersonal, and Structural Levels

Systems Bias

Upon review, we noted that our scoring system prioritized metrics such as United States Medical Licensing Examination (USMLE) Step scores, grades, and Alpha Omega Alpha (AOA) membership. These metrics may amplify existing biases12  and reinforce structural inequities, benefiting those who are White and/or from more affluent backgrounds with higher parental education.5,8,13  We recognized using such markers might systematically benefit already advantaged applicants, and that by relying on them, we might be missing other valuable personal attributes in applicants. To broaden our approach, we held a retreat for program leadership to create a shared mental model of attributes of successful residents. Through individual reflection, art, and group discussion, we developed a list of such attributes, and then shared this list with our residents for their input. In line with best practices, we developed standardized interview questions14  to capture consistent information on these qualities. We revised our scoring rubric to prioritize the newly defined qualities and deemphasize metrics such as grades, USMLE scores, and AOA.

Interpersonal Bias

We recognized that other programs reported success engaging a diverse selection committee as one strategy to mitigate bias.15  We collaborated with the department of pediatrics resident-led diversity committee to implement a process in which a subgroup of the committee reviews our UiM applicant pool to decide whom to invite for interviews. Engaging people from diverse backgrounds in screening applications allows for deeper conversations around applicants' achievements, life experiences, and backgrounds. This collaboration also ensures robust and diverse representation on our Ranking Advisory Committee (RAC), which advises on the program's final rank list.

To minimize bias based on visual appearance,5,6  we stopped using photographs of the applicants during file review and ranking.

Individual Bias

We followed the recommendation from the AAMC that those involved with interviews participate in unconscious bias training.16  We offered 3 training options: (1) in-person training specifically related to anti-bias in recruitment facilitated by a professional trainer; (2) in-person, campus-wide UCSF diversity training; or (3) a link to an online AAMC training on unconscious bias in recruitment and selection.17  Both in-person trainings transitioned to videoconference in 2020 due to COVID-19-imposed safety considerations.

We used logistic regression to predict the odds that matched applicants would identify as UiM, using residents matched in 2015 as a baseline.

The project was reviewed by the UCSF Institutional Review Board and determined not to require further approval.

Results

Faculty working on recruitment spent more than the allocated FTE to create new systems the first year; subsequent years allowed faculty to spend less time as systems were refined. The holistic review process does require an increased time commitment from residency program leadership and the diversity committee to read and screen all applications. Others involved with recruitment and selection—including other faculty, administrative staff, and residents—are not spending substantively more time on selection activities.

Systems Bias

The characteristics we identified as desirable in residents largely align with previous research,18  and include adaptability, initiative, commitment to diversity, and ability to work on a team. We implemented standardized interview questions to assess these qualities. The scoring rubric was revised iteratively over each season from 2017 to 2020.

Interpersonal Bias

We increased the size and diversity of the RAC over time. In 2016, the RAC had 20 members; racial self-identity was not recorded, but we estimate that approximately 10% of members identified as UiM. From 2017 to 2019, the RAC steadily increased membership from 60 to 80, with 25% to 30% of members identifying as UiM. The RAC has retained a stable group of core faculty with new residents, fellows, and faculty joining each year.

Individual Bias

All interviewers and RAC members completed at least one of the required anti-bias trainings. Completion was verified by self-attestation. The majority of interviewers completed in-person anti-bias trainings, and this percentage increased over time. In 2018, 78% (57 of 73) of interviewers completed an in-person UCSF-wide or recruitment specific training while 22% (16 of 73) completed the online AAMC training; in 2020, 95% (121 of 127) completed an in-person training while only 5% (6 of 127) opted for the online AAMC training.

The percentage of interns who self-identified as UiM increased from 3 of 28 (11%) in 2015 to 13 of 29 (45%) in 2019 and 10 of 28 (36%) in 2020 (Figure). Compared to residents matching in 2015, those matching in 2019 had 6.8 times increased odds of identifying as UiM (P = .008, CI 1.66–27.6), and those matching in 2020 had 4.6 times increased odds of identifying as UiM (P = .035, CI 1.11–19.3).

Figure

Percentage of Interns Self-Identifying as Underrepresented in Medicine by Academic Year

Figure

Percentage of Interns Self-Identifying as Underrepresented in Medicine by Academic Year

Discussion

Using an equity framework to systematically address the structural, interpersonal, and individual biases in our residency selection process resulted in more holistic application review, reduced reliance on inherently biased metrics, and increased numbers of matched UiM residents over 3 years in our pediatric residency program.

Consistent with previous research, we found that implementing strategies to mitigate individual and interpersonal biases, such as increasing the size and diversity of selection committees and implementing anti-bias training, were useful in increasing residency diversity. This is supported by Aibana et al who were able to increase the number of UiM residents matched in internal medicine from 12.5% in 2015–2016 to 31.7% in 2017–2018 using strategies including faculty unconscious bias training and standardized interviews.19  Our approach adds the use of an equity lens to specifically name and address structural bias in residency selection, which we have not found described in the literature. Of note, Lucey and Saguil described how structural racism affects economic success, educational opportunity, and bias in the educational environment; these differences in opportunity can result in differences in MCAT scores for UiM applicants, which can affect medical school admission.20  It is reasonable to assume these same factors also affect residency applicants. Using an equity framework that specifically called attention to structural oppression allowed us to identify and decrease reliance on metrics reflecting structural oppression in our selection process.

Given the complex nature of systems change, it is difficult to attribute our success to holistic review alone. Our institution has had a steadily improving culture of diversity, equity, and inclusion, exemplified by an institution-wide “Differences Matter” initiative21  and a tremendous grassroots impetus for change by our resident-led diversity committee. Recent literature has emphasized the importance of institutional support in enacting policies and practices to reduce bias and promote diversity, equity, inclusion, and anti-racism.22,23  We suggest institutions seeking to improve diversity in residency classes mobilize both institutional and trainee level support. We further note that culture change takes longer than policy change; some stakeholders continue to place outsized value on traditional academic metrics of “success” such as board scores or elite school pedigrees. It is helpful to be explicit and deliberate with all stakeholders about the merits of an equitable selection process as well as prepared with evidence of the demonstrable value of a diverse workforce in order to counter such views. Finally, we note that residency programs currently compete for a relatively small number of UiM applicants. We must all participate in programs and initiatives that address the structural, interpersonal, and individual barriers preventing UiM students from enrolling in medical schools in order to close this critical gap.

This study has limitations. It reflects our experience at a single, large, urban residency program based in the relatively diverse San Francisco Bay Area. While this may limit generalizability, we believe the application of an equity framework to selection processes is an approach that merits consideration for any program seeking to improve diversity. We note that the percentage of UiM matriculants to our 2019–2020 class was lower than the previous year. We suspect this may be due to regression to the mean, but we are curious to see how the upcoming recruitment season will unfold.

For the 2020–2021 academic year, our program is further reducing emphasis on traditional academic markers in residency selection. Future research should focus on understanding how individual, interpersonal, and structural oppression affect academic performance in medical school, and how academic markers might be equitably used in residency selection.

Conclusions

Using an equity framework to guide implementation of holistic review was associated with an increase in diversity in our intern class.

The authors would like to thank Dr. Matt Pantell for his assistance with the statistical analysis and the members of the Department of Pediatrics Diversity Committee for their invaluable contributions to creating a diverse and inclusive climate.

References

1. 
Association of American Medical Colleges.
Data-Driven Diversity and Inclusion Change
.
2021
.
2. 
Lupkin
S.
Deep Roots for Lack of Minotities in American Medical Schools—Problem starts in 3rd grade
.
MedPage Today
.
2021
.
3. 
Campion
MA,
Palmer
DK,
Campion
JE.
A review of structure in the selection interview
.
Pers Psychol
.
1997
;
50
(3)
:
655
702
.
4. 
Stephenson-Famy
A,
Houmard
BS,
Oberoi
S,
Manyak
A,
Chiang
S,
Kim
S.
Use of the interview in resident candidate selection: a review of the literature
.
J Grad Med Educ
.
2015
;
7
(4)
:
539
548
.
5. 
Edmond
MB,
Deschenes
JL,
Eckler
M,
Wenzel
RP.
Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews
.
Acad Med
.
2001
;
76
(12)
:
1253
1256
.
6. 
Corcimaru
A,
Morrell
MC,
Morrell
DS.
Do looks matter? The role of the Electronic Residency Application Service photograph in dermatology residency selection
.
Dermatol Online J.
2018
;
24(4):13030/qt5qc988jz.
7. 
Maxfield
CM,
Thorpe
MP,
Desser
TS,
et al.
Bias in radiology resident selection: do we discriminate against the obese and unattractive?
Acad Med
.
2019
;
94
(11)
:
1774
1780
.
8. 
Kassam
AF,
Cortez
AR,
Winer
LK,
et al.
Swipe right for surgical residency: exploring the unconscious bias in resident selection
.
Surgery
.
2020
;
168
(4)
:
724
729
.
9. 
Boatright
D,
Ross
D,
O'Connor
P,
Moore
E,
Nunez-Smith
M.
Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society
.
JAMA Intern Med
.
2017
;
177
(5)
:
659
665
.
10. 
Association of American Medical Colleges.
Holistic review
.
2021
.
11. 
National Equity Project.
The Lens of Systemic Oppression
.
2021
.
12. 
Teherani
A,
Hauer
KE,
Fernandez
A,
King
TE
Jr,
Lucey
C.
How small differences in assessed clinical performance amplify to large differences in grades and awards: a cascade with serious consequences for students underrepresented in medicine
.
Acad Med
.
2018
;
93
(9)
:
1286
1292
.
13. 
Low
D,
Pollack
SW,
Liao
ZC,
et al.
Racial/ethnic disparities in clinical grading in medical school
.
Teach Learn Med
.
2019
;
31
(5)
:
487
496
.
14. 
Association of American Medical Colleges.
Best Practices for Conducting Residency Program Interviews
.
2021
.
15. 
Tunson
J,
Boatright
D,
Oberfoell
S,
et al.
Increasing resident diversity in an emergency medicine residency program: a pilot intervention with three principal strategies
.
Acad Med
.
2016
;
91
(7)
:
958
961
.
16. 
17. 
Association of American Medical Colleges.
Online Seminar: The science of unconscious bias and what to do about it in the search and recruitment process
.
2021
.
18. 
Rosenbluth
G,
O'Brien
B,
Asher
EM,
Cho
CS.
The “zing factor"—how do faculty describe the best pediatrics residents?
J Grad Med Educ
.
2014
;
6
(1)
:
106
111
.
19. 
Aibana
O,
Swails
JL,
Flores
RJ,
Love
L.
Bridging the gap: holistic review to increase diversity in graduate medical education
.
Acad Med
.
2019
;
94
(8)
:
1137
1141
.
20. 
Lucey
CR,
Saguil
A.
The consequences of structural racism on MCAT scores and medical school admissions: the past is prologue
.
Acad Med
.
2020
;
95
(3)
:
351
356
.
21. 
UCSF School of Medicine.
Differences Matter
.
2021
.
22. 
Shim
RS.
Dismantling structural racism in academic medicine: a skeptical optimism
.
Acad Med
.
2020
;
95
(12)
:
1793
1795
.
23. 
Mateo
CM,
Williams
DR.
More than words: a vision to address bias and reduce discrimination in the health professions learning environment
.
Acad Med.
2020
;
95(12S Addressing Harmful Bias and Eliminating Discrimination in Health Professions Learning Environments):169–177.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.