Improving Diversity in Pediatric Residency Selection: Using an Equity Framework to Implement Holistic Review

Despite the evidence affirming the benefits of a diverse workforce in pediatrics,¹  residency programs have struggled to recruit residents from backgrounds that are underrepresented in medicine (UiM). Many factors contribute to a lack of diversity in medicine, including a relatively low number of UiM applicants, lack of an inclusive climate in medicine, and bias in the recruitment and selection process.² 

Multiple studies demonstrate that bias influences residency selection.^3–9  To ensure an equitable recruitment and selection process, the Association of American Medical Colleges (AAMC) suggests the use of holistic review, defined as “a flexible, individualized way of assessing an applicant's capabilities by which balanced consideration is given to experiences, attributes, and academic metrics.”¹⁰  Because holistic review is mission driven, its implementation varies at each institution. Effective implementation of holistic review requires an understanding of the different ways in which bias can manifest.

The National Equity Project Lens of Systemic Oppression provides a helpful framework to understand the levels on which bias acts.¹¹  The framework describes 3 levels of oppression—structural, interpersonal, and individual. Structural oppression is comprised of the institutional practices that create and perpetuate inequitable outcomes. Interpersonal oppression relates to individual interactions that perpetuate bias. In individual oppression, one's personal beliefs and actions—even when unconscious—perpetuate bias.

Prior to 2015, the University of California San Francisco (UCSF) Pediatric Residency Program had relatively few UiM residents. We sought to determine whether implementing holistic review using an equity framework that assesses for bias in our residency selection process along structural, interpersonal, and individual levels would increase the number of matched residents who self-identify as UiM.

UCSF Pediatrics is a large, academic, urban residency program with 84 residents. To inform implementation of holistic review, the intern selection faculty team (associate program director [J.M.] with 25% FTE support for recruitment and a former chief resident [R.B.] with 30% FTE support for recruitment) worked with the department of pediatrics diversity director (A.M.) to identify potential sources of bias at each level, described by the systems of oppression framework. We collaborated with the residency program director to develop approaches to mitigate these biases. The majority of this work occurred during the fall and winter of 2017, with ongoing changes and adjustments made annually based on feedback from residents, faculty, and applicants.

We highlight examples of strategies to mitigate bias below. Additional strategies are listed in the Table.

Upon review, we noted that our scoring system prioritized metrics such as United States Medical Licensing Examination (USMLE) Step scores, grades, and Alpha Omega Alpha (AOA) membership. These metrics may amplify existing biases¹²  and reinforce structural inequities, benefiting those who are White and/or from more affluent backgrounds with higher parental education.^5,8,13  We recognized using such markers might systematically benefit already advantaged applicants, and that by relying on them, we might be missing other valuable personal attributes in applicants. To broaden our approach, we held a retreat for program leadership to create a shared mental model of attributes of successful residents. Through individual reflection, art, and group discussion, we developed a list of such attributes, and then shared this list with our residents for their input. In line with best practices, we developed standardized interview questions¹⁴  to capture consistent information on these qualities. We revised our scoring rubric to prioritize the newly defined qualities and deemphasize metrics such as grades, USMLE scores, and AOA.

We recognized that other programs reported success engaging a diverse selection committee as one strategy to mitigate bias.¹⁵  We collaborated with the department of pediatrics resident-led diversity committee to implement a process in which a subgroup of the committee reviews our UiM applicant pool to decide whom to invite for interviews. Engaging people from diverse backgrounds in screening applications allows for deeper conversations around applicants' achievements, life experiences, and backgrounds. This collaboration also ensures robust and diverse representation on our Ranking Advisory Committee (RAC), which advises on the program's final rank list.

To minimize bias based on visual appearance,^5,6  we stopped using photographs of the applicants during file review and ranking.

We followed the recommendation from the AAMC that those involved with interviews participate in unconscious bias training.¹⁶  We offered 3 training options: (1) in-person training specifically related to anti-bias in recruitment facilitated by a professional trainer; (2) in-person, campus-wide UCSF diversity training; or (3) a link to an online AAMC training on unconscious bias in recruitment and selection.¹⁷  Both in-person trainings transitioned to videoconference in 2020 due to COVID-19-imposed safety considerations.

We used logistic regression to predict the odds that matched applicants would identify as UiM, using residents matched in 2015 as a baseline.

The project was reviewed by the UCSF Institutional Review Board and determined not to require further approval.

Faculty working on recruitment spent more than the allocated FTE to create new systems the first year; subsequent years allowed faculty to spend less time as systems were refined. The holistic review process does require an increased time commitment from residency program leadership and the diversity committee to read and screen all applications. Others involved with recruitment and selection—including other faculty, administrative staff, and residents—are not spending substantively more time on selection activities.

The characteristics we identified as desirable in residents largely align with previous research,¹⁸  and include adaptability, initiative, commitment to diversity, and ability to work on a team. We implemented standardized interview questions to assess these qualities. The scoring rubric was revised iteratively over each season from 2017 to 2020.

We increased the size and diversity of the RAC over time. In 2016, the RAC had 20 members; racial self-identity was not recorded, but we estimate that approximately 10% of members identified as UiM. From 2017 to 2019, the RAC steadily increased membership from 60 to 80, with 25% to 30% of members identifying as UiM. The RAC has retained a stable group of core faculty with new residents, fellows, and faculty joining each year.

All interviewers and RAC members completed at least one of the required anti-bias trainings. Completion was verified by self-attestation. The majority of interviewers completed in-person anti-bias trainings, and this percentage increased over time. In 2018, 78% (57 of 73) of interviewers completed an in-person UCSF-wide or recruitment specific training while 22% (16 of 73) completed the online AAMC training; in 2020, 95% (121 of 127) completed an in-person training while only 5% (6 of 127) opted for the online AAMC training.

The percentage of interns who self-identified as UiM increased from 3 of 28 (11%) in 2015 to 13 of 29 (45%) in 2019 and 10 of 28 (36%) in 2020 (Figure). Compared to residents matching in 2015, those matching in 2019 had 6.8 times increased odds of identifying as UiM (P = .008, CI 1.66–27.6), and those matching in 2020 had 4.6 times increased odds of identifying as UiM (P = .035, CI 1.11–19.3).

Using an equity framework to systematically address the structural, interpersonal, and individual biases in our residency selection process resulted in more holistic application review, reduced reliance on inherently biased metrics, and increased numbers of matched UiM residents over 3 years in our pediatric residency program.

Consistent with previous research, we found that implementing strategies to mitigate individual and interpersonal biases, such as increasing the size and diversity of selection committees and implementing anti-bias training, were useful in increasing residency diversity. This is supported by Aibana et al who were able to increase the number of UiM residents matched in internal medicine from 12.5% in 2015–2016 to 31.7% in 2017–2018 using strategies including faculty unconscious bias training and standardized interviews.¹⁹  Our approach adds the use of an equity lens to specifically name and address structural bias in residency selection, which we have not found described in the literature. Of note, Lucey and Saguil described how structural racism affects economic success, educational opportunity, and bias in the educational environment; these differences in opportunity can result in differences in MCAT scores for UiM applicants, which can affect medical school admission.²⁰  It is reasonable to assume these same factors also affect residency applicants. Using an equity framework that specifically called attention to structural oppression allowed us to identify and decrease reliance on metrics reflecting structural oppression in our selection process.

Given the complex nature of systems change, it is difficult to attribute our success to holistic review alone. Our institution has had a steadily improving culture of diversity, equity, and inclusion, exemplified by an institution-wide “Differences Matter” initiative²¹  and a tremendous grassroots impetus for change by our resident-led diversity committee. Recent literature has emphasized the importance of institutional support in enacting policies and practices to reduce bias and promote diversity, equity, inclusion, and anti-racism.^22,23  We suggest institutions seeking to improve diversity in residency classes mobilize both institutional and trainee level support. We further note that culture change takes longer than policy change; some stakeholders continue to place outsized value on traditional academic metrics of “success” such as board scores or elite school pedigrees. It is helpful to be explicit and deliberate with all stakeholders about the merits of an equitable selection process as well as prepared with evidence of the demonstrable value of a diverse workforce in order to counter such views. Finally, we note that residency programs currently compete for a relatively small number of UiM applicants. We must all participate in programs and initiatives that address the structural, interpersonal, and individual barriers preventing UiM students from enrolling in medical schools in order to close this critical gap.

This study has limitations. It reflects our experience at a single, large, urban residency program based in the relatively diverse San Francisco Bay Area. While this may limit generalizability, we believe the application of an equity framework to selection processes is an approach that merits consideration for any program seeking to improve diversity. We note that the percentage of UiM matriculants to our 2019–2020 class was lower than the previous year. We suspect this may be due to regression to the mean, but we are curious to see how the upcoming recruitment season will unfold.

For the 2020–2021 academic year, our program is further reducing emphasis on traditional academic markers in residency selection. Future research should focus on understanding how individual, interpersonal, and structural oppression affect academic performance in medical school, and how academic markers might be equitably used in residency selection.

Using an equity framework to guide implementation of holistic review was associated with an increase in diversity in our intern class.

The authors would like to thank Dr. Matt Pantell for his assistance with the statistical analysis and the members of the Department of Pediatrics Diversity Committee for their invaluable contributions to creating a diverse and inclusive climate.