Background

Holistic review promotes diversity, but widespread implementation remains limited.

Objective

We aimed to develop a practical approach to incorporate holistic review principles in screening applicants in the Electronic Residency Application Service (ERAS) and to assess the impact on diversity.

Methods

Three residency programs (internal medicine [IM], pediatrics, and surgery) at McGovern Medical School developed filters to identify applicants with experiences/attributes aligned with the institutional mission. These filters were retroactively applied to each program's 2019–2020 applicant pool using built-in ERAS capabilities to group applicants by user-defined features. We compared the demographics of applicants reviewed during the cycle with those identified retrospectively through experiences/attributes filters.

Results

The IM, pediatrics, and surgery programs received 3527, 1341, and 1313 applications, respectively, in 2019–2020. Retrospective use of experiences/attributes filters, without scores, narrowed the IM applicant pool for review to 1301 compared to 1323 applicants reviewed during actual recruitment, while the pediatrics filters identified 514 applicants compared to 384 at baseline. The surgery filters resulted in 582 applicants, but data were missing for baseline comparison. Compared to the baseline screening approach utilizing scores, mission-based filters increased the proportions of underrepresented in medicine applicants selected for review in IM (54.8% [95% CI 52.1–57.5] vs 22.7% [20.4–24.9], P < .0001) and pediatrics (63.2% [95% CI 59.1–67.4] vs 25.3% [20.9–29.6], P < .0001).

Conclusions

Program directors can leverage existing ERAS features to conduct application screening in alignment with holistic review principles. Widespread implementation could have important repercussions for enhancing physician workforce diversity.

Objectives

To develop a practical approach for applying holistic review principles to screening large applicant pools in the Electronic Residency Application Service.

Findings

Mission-based experiences and attributes filters increased racial and ethnic diversity of applicants selected for review compared to metric-based filters.

Limitations

Study findings are limited to the use of holistic review principles in the application review stage.

Bottom Line

Widescale application of holistic review principles through all stages of graduate medical education recruitment can be an important tool for enhancing physician workforce diversity.

In February 2020, the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) announced that the United States Medical Licensing Examination (USMLE) Step 1 three-digit score would be reported as pass/fail starting as early as January 2022. A debate has since ensued about the potential impact on the residency selection process. Many programs receive more applications than they can manually review and must use filters to identify desired candidates for a more thorough evaluation. Prior data suggest that program directors (PDs) utilize USMLE score filters for this purpose.1  However, heavy reliance on score filters can exclude applicants traditionally underrepresented in medicine (UiM); and approaches, such as mission-based holistic review, that give balanced consideration to experiences, attributes, and scores, have been shown to improve residency program diversity.26  Yet, there is limited evidence on practical approaches to implementing holistic principles in screening without reviewing all applications.

Although the long-term impact of holistic review on institutions, physician quality, and patient outcomes has not been described, it is well recognized that diversifying the US physician workforce is an important strategy to combat disparities.710  Thus, many hoped the transition to pass/fail score reporting would represent an opportunity to advance holistic review to increase diversity in graduate medical education (GME).11,12  However, others voiced concerns that eliminating Step 1 scores would simply shift emphasis to other biased indicators of applicants' worthiness, such as USMLE Step 2 scores, medical school reputation, or social connections.13  Furthermore, international medical graduates (IMGs), who fill nearly a quarter of positions in the US Match and a higher proportion of primary care specialties, represent an especially vulnerable population, since USMLE scores represent one of the few ways IMGs have traditionally distinguished themselves.14 

Unfortunately, the COVID-19 pandemic turned out to be a much more immediate disturbance to GME recruitment, as clinical rotations and USMLE examinations were abruptly cancelled. In response, the 2020–2021 application cycle was delayed, and the Association of American Medical Colleges (AAMC) recommended virtual interviews for all programs, a recommendation now extended into 2021–2022.15  These adjustments promote short-term safety and have potential long-term benefits, but they could substantially impede the movement toward holistic review in GME. Virtual interviews may encourage applicants to apply more broadly, especially without away electives to demonstrate interest to a specific program. Notably, a recent survey of PDs indicated that rising application numbers reduce the likelihood of holistic review.16  Although a few specialties are now piloting strategies such as preference signals and supplemental applications to address challenges in GME selection,1719  large application pools still make detailed holistic review untenable.

Concurrently, the COVID-19 pandemic has highlighted dramatic racial/ethnic health disparities, further demonstrating the need to bolster efforts, such as holistic review in recruitment, that enhance physician workforce diversity. Thus, there is an urgent need for practical approaches to implementing principles of holistic review in the context of the current Electronic Residency Application Service (ERAS). This will allow PDs to identify diverse, mission-aligned applicants well suited for their programs without requiring substantially more time than score-based filters alone.

Here, we describe the development and use of mission-based “experiences and attributes filters” (EA filters) as the initial tool to sort applicants in ERAS. We hypothesized that EA filters would increase the proportion of UiM applicants reviewed without a meaningful difference in average USMLE scores among applicants selected by EA filters and applicants selected by metric-based filters. Based on test characteristics and previous data correlating USMLE performance with board pass rates,20,21  we defined, a priori, a meaningful change as more than 10 points (half a standard deviation of mean USMLE scores in previous years).

To demonstrate the feasibility of employing holistic review principles using existing ERAS filtering capabilities across a spectrum of GME programs, we assessed the 2019–2020 categorical applicant pool for 3 specialties: internal medicine (IM), pediatrics, and surgery. The IM program, the largest at McGovern Medical School (MMS), receives approximately 3500 categorical applicants, reviews about 1300 applications, and matriculates 40 interns annually. The pediatrics and surgery programs each receive approximately 1300 to 1400 applications, review 300 to 400 applications, and fill 24 and 8 intern positions, respectively. The original approach to holistic review in the IM residency has been previously described.2  At baseline, all 3 programs utilize a metric filter based on USMLE Step 1 cutoff with some combination of attributes filters that include Spanish language proficiency, UiM status, and Texas residency as the initial steps to narrow applicants for detailed review. Selected applications are subsequently reviewed manually for other academic merits, attributes, and experiences that would contribute value to each program. Full application review takes up to 10 minutes.

All authors (PDs and associate PDs of the 3 programs) were familiar with the concept of holistic review and use of limited ERAS filters at baseline. After completing the 2019–2020 application cycle, we convened to discuss the best practical approach to implementing a mission-aligned holistic review process without relying on USMLE score filters. ERAS is equipped to filter applicants by various characteristics beyond test scores, such as language proficiency, and for free text searches within the Professional and Training Experience section to identify all individuals with a desired experience (eg, prior work as a “teacher”). Using the medical school's mission statement as a template, the team identified the core values best aligned with their respective program's goals. Then, we enumerated a potential list of relevant applicant experiences or attributes illustrative of each core value (Table 1). We subsequently attempted to translate each potential experience/attribute into one or more searchable EA filters in ERAS based on the available filter categories. For instance, for the MMS core value of “promoting interprofessional collaboration,” one program described prior experience as a Scribe as an applicant trait that might illustrate this core value. Therefore, we used a free text search in ERAS for “Scribe” under the Professional Experience Position section to filter applicants whose prior professional position contained “Scribe.” Table 1 provides details about each core value and corresponding EA filters selected.

Table 1

McGovern Medical School Core Values With Corresponding Applicant Experiences and Attributes and ERAS Filters Designed by Internal Medicine, Pediatrics, and Surgery Residency Programs

McGovern Medical School Core Values With Corresponding Applicant Experiences and Attributes and ERAS Filters Designed by Internal Medicine, Pediatrics, and Surgery Residency Programs
McGovern Medical School Core Values With Corresponding Applicant Experiences and Attributes and ERAS Filters Designed by Internal Medicine, Pediatrics, and Surgery Residency Programs

Each program then retroactively applied their chosen EA filters to their 2019–2020 categorical applicant pools in ERAS and exported de-identified datasets as an Excel document. We compared the total number of applicants captured by each program's desired EA filters with the number of applicants typically reviewed during a recruitment cycle. This initial pool exceeded the usual number of applications reviewed by over 300 for each program, surpassing available faculty resources for thorough review in an actual recruitment cycle. Therefore, each program's representatives (associate PD and/or PD) further prioritized and narrowed their core values based on respective program goals. This pilot involved only the authors in the design and finalization of EA filters.

The process of enumerating potential experiences/attributes to match MMS core values and defining ERAS search filters occurred over 4 meetings among authors lasting up to 1.5 hours each. Applying the EA filters in ERAS to applicant pools for all 3 programs took 1 hour, and we convened for another hour to finalize core values. For each final core value chosen by a program, we identified the number of unique individuals captured by the corresponding EA filters. Then, we calculated the proportion of selected applicants who were female, UiM, and IMGs as well as mean USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores. We categorized applicants as UiM if they self-identified as American Indian, Alaska Native, Black/African American, Hispanic/Latino, Native Hawaiian, or Pacific Islander. We categorized gender as self-identified in ERAS. Finally, we compared the demographic characteristics and USMLE scores of applicants identified by mission-based EA filters with those of applicants actually selected for detailed review by each program during the 2019–2020 application cycle. We compared continuous variables with a t test and categorical variables with a chi-square test. Data were analyzed using Microsoft Excel and SAS 9.4 (SAS Institute Inc, Cary, NC). This study was deemed exempt by the Institutional Review Board of the University of Texas at Houston.

Table 2 describes characteristics of categorical applicants for all 3 programs during the 2019–2020 application cycle as well as characteristics of applicants selected by the mission-aligned EA filters. The IM program prioritized 5 MMS core values, and the corresponding EA filters identified 1301 applicants for detailed review (Table 2). Among them, 713 (54.8%, 95% CI 52.1–57.5) were UiM, and this was significantly higher than the proportion of UiM applicants (22.7%, 95% CI 20.4–24.9) reviewed in the 2019–2020 recruitment cycle (P < .0001; Table 3). Among 514 applicants identified for detailed review by the pediatrics program's EA filters, 325 (63.2%, 95% CI 59.1–67.4) were UiM compared to 97 (25.3%, 95% CI 20.9–29.6) UiM applicants reviewed in 2019–2020 (P < .0001; Table 3). The mean USMLE Step 1 scores of applicants selected by EA filters for IM and pediatrics were lower than mean scores of applicants reviewed in 2019–2020 (Table 3). The surgery program did not utilize the Application Reviewed function in ERAS; therefore, we were unable to compare applicants reviewed in the 2019–2020 recruitment cycle with those identified by EA filters.

Table 2

Demographic Characteristics and USMLE Scores of Categorical Applicants Selected by Using Experiences and Attributes Filters in ERAS by Internal Medicine, Pediatrics, and Surgery Residency Programs at McGovern Medical School

Demographic Characteristics and USMLE Scores of Categorical Applicants Selected by Using Experiences and Attributes Filters in ERAS by Internal Medicine, Pediatrics, and Surgery Residency Programs at McGovern Medical School
Demographic Characteristics and USMLE Scores of Categorical Applicants Selected by Using Experiences and Attributes Filters in ERAS by Internal Medicine, Pediatrics, and Surgery Residency Programs at McGovern Medical School
Table 3

Comparison of Demographic Characteristics and USMLE Scores of Applicants Selected by Experiences and Attributes Filters and Applicants Reviewed in the 2019–2020 Recruitment Cycle

Comparison of Demographic Characteristics and USMLE Scores of Applicants Selected by Experiences and Attributes Filters and Applicants Reviewed in the 2019–2020 Recruitment Cycle
Comparison of Demographic Characteristics and USMLE Scores of Applicants Selected by Experiences and Attributes Filters and Applicants Reviewed in the 2019–2020 Recruitment Cycle

Three residency programs at MMS used their institutional core values to craft individualized ERAS search strategies using EA filters as the initial screening tool to select applicants for further review. Compared to our baseline screening process that included USMLE score filters, this mission-based approach incorporating holistic review principles improved the diversity of the selected applicant pool with respect to self-identified gender and race/ethnicity. We found that applicants identified through mission-based EA filters had slightly lower mean USMLE scores than those selected for review during the actual application cycle. Although some data support the role of USMLE in predicting licensing examination pass rates,20,21  evidence suggests other factors have better predictive value than Step 1 alone.20,21  It is unlikely that the 3- to 5-point mean differences in our study will translate into appreciable differences in board pass rates or predict future success as a physician. Nevertheless, we did not thoroughly review applications identified with EA filters to identify risk factors for board failure. We present EA filters as an efficient and holistic approach to initially screen large application pools in anticipation of the USMLE Step 1 pass/fail change.

It is important to note that the purpose of holistic review methods is not to eliminate academic metrics. Holistic review necessitates broad selection criteria that promote various types of diversity, and identifying applicants using multiple data points potentially reduces the impact of bias inherent in any one factor. Hence, similar to the limitations of exclusively using score-based filters to narrow applicant pools, a singular EA filter would not attain broad-based diversity. As a state institution serving a diverse population and training physicians for Texas, we prioritized the initial use of EA filters to identify applicants who can contribute to this mission. This would be followed by a manual review of academic performance to select applicants for interview. Other programs can combine metric-based and EA filters as their initial sorting tool in alignment with their respective goals. Additionally, if PDs use USMLE scores as one component of a multifaceted screening strategy, test scores can allow students to demonstrate academic excellence without becoming the focal point of residency recruitment. This could potentially promote the longevity of USMLE Step 2 CK reporting as a 3-digit score. Overall, there is a need for further assessment of the most balanced approaches to recruit diverse trainees who can meet the needs of an increasingly diverse US population and alleviate health disparities.

Nonetheless, the challenge remains in balancing the number of EA filters with a program's capacity for thorough application reviews. In this instance, each program had to narrow its initial set of desired EA filters to accommodate available resources for detailed application review. The surgery and pediatrics programs captured at least 150 more applicants for review compared to baseline, and without application caps, programs must make individualized decisions to add resources to review more applications or restrict EA filter options. The initial process of developing a new mission-based search strategy might be time-consuming, especially if it involves many stakeholders. Programs should also consider the need to train personnel on the principles of holistic review and the use of ERAS filters. However, once defined, EA filters can be applied with small modifications to future applicant pools, since program missions do not change drastically year to year. In contrast to expectations that IMGs will have a harder time distinguishing themselves without Step 1 scores,22  we found that the use of EA filters might provide an opportunity to increase the number of IMGs reviewed by programs. However, program leaders with a mission to enhance IMG representation will need an intentional approach, and further research is needed to identify best practices.

While each program utilized a few filters based on experiences and attributes during usual recruitment, we found that this intentional and comprehensive mission-based approach led to greater diversity among applicants selected for review. However, our study is limited to the application review stage and may not enhance diversity among candidates interviewed, ranked, or matched. Future efforts will assess the ongoing impact of mission-based filters on the diversity of matriculants. It is also unclear if EA filters developed by other programs will produce similar results with respect to diversity. We tracked gender and race/ethnicity as markers of diversity because they were available as self-reported data in ERAS. We cannot assess the impact of our EA filters on other potential groups of interest that would signify broad-based diversity (eg, applicants with disabilities, LGBTQIA+ applicants, etc). The absence of comparison data for surgery further limits our findings to primary care fields. Notably, expanding access to primary care is also important for addressing health disparities. Hence, the approach of EA filters might provide an opportunity for future assessments of the most effective filters to identify applicants who eventually choose primary care careers. Additional studies are needed to understand how EA filters might impact surgical and specialty applicants. This study focused on a small part of the complex system of undergraduate medical education (UME) to GME transition. However, selecting applicants using a multifaceted, mission-driven approach based on holistic review principles could be an important piece in addressing inequity in GME recruitment and diversifying the health care workforce.

We also recognize ERAS remains limited in its search features, and some programs may not easily identify relevant EA filters. Free text searches could miss appropriate applicants who use a different key word; thus, utilizing a database of inclusive key words (eg, “instructor” vs “teacher”) might be necessary. Furthermore, searches and filters in ERAS are currently limited to certain applicant-generated entries. Many documents cannot be searched with our EA filter approach, including personal statements, letters of recommendation, and medical school performance evaluations. Expanding the availability of searchable sections might also prove useful. For example, programs could search for applicants who at least 3 letter writers described as “compassionate.” Moreover, there is a broader need within medical education for implementation of objective appraisals of skills critical to success as a physician, such as effective collaboration, communication, leadership, and resilience. Increasing the number and objectivity of metrics, experiences, and attributes filterable in ERAS would facilitate programs' ability to find the applicants best poised to contribute to the institution's mission. It would also give more opportunities for applicants to demonstrate excellence in line with a program's mission. However, just as USMLE scores might exhibit bias,23  it is possible that other elements entered in ERAS could perpetuate inequity. Therefore, caution must be taken to adequately scrutinize any new appraisal methods for true objectivity.

We demonstrate that the use of mission-based experiences and attributes filters is currently feasible in ERAS and could enhance diversity of applicants selected in the context of application inflation and time constraints.

1. 
Garber
AM,
Kwan
B,
Williams
CM,
et al
Use of filters for residency application review: results from the Internal Medicine In-Training Examination program director survey
.
J Grad Med Educ
.
2019
;
11
(6)
:
704
707
.
2. 
Aibana
O,
Swails
JL,
Flores
RJ,
Love
L.
Bridging the gap: holistic review to increase diversity in graduate medical education
.
Acad Med
.
2019
;
94
(8)
:
1137
1141
.
3. 
Garrick
JF,
Perez
B,
Anaebere
TC,
Craine
P,
Lyons
C,
Lee
T.
The diversity snowball effect: the quest to increase diversity in emergency medicine: a case study of highland's emergency medicine residency program
.
Ann Emerg Med
.
2019
;
73
(6)
:
639
647
.
4. 
Butler
PD,
Aarons
CB,
Ahn
J,
et al
Leading from the front: an approach to increasing racial and ethnic diversity in surgical training programs
.
Ann Surg
.
2019
;
269
(6)
:
1012
1015
.
5. 
Marbin
J,
Rosenbluth
G,
Brim
R,
Cruz
E,
Martinez
A,
McNamara
M.
Improving diversity in pediatric residency selection: using an equity framework to implement holistic review
.
J Grad Med Educ
.
2021
;
13
(2)
:
195
200
.
6. 
Nehemiah
A,
Roberts
SE,
Song
Y,
et al
Looking beyond the numbers: increasing diversity and inclusion through holistic review in general surgery recruitment
.
J Surg Educ
.
2021
;
78
(3)
:
763
769
.
7. 
Marrast
LM,
Zallman
L,
Woolhander
S,
Bor
DH,
McCormick
D.
Minority physicians' role in the care of underserved patients: diversifying the physician workforce may be key in addressing health disparities
.
JAMA Intern Med
.
2014
;
174
(2)
:
289
291
.
8. 
Alsan
M,
Garrick
O,
Graziani
G.
Does diversity matter for health? Experimental evidence from Oakland
.
Am Econ Rev
.
2019
;
109
(12)
:
4071
4111
.
9. 
Greenwood
BN,
Hardeman
RR,
Huang
L,
Sojourner
A.
Physician-patient racial concordance and disparities in birthing mortality for newborns
.
Proc Natl Acad Sci U S A
.
2020
;
117
(35)
:
21194
21200
.
10. 
Bach
PB,
Pham
HH,
Schrag
D,
Tate
RC,
Hargraves
JL.
Primary care physicians who treat Blacks and Whites
.
N Engl J Med
.
2004
;
351
(6)
:
575
584
.
11. 
Swails
JL,
Aibana
O,
Stoll
BJ.
The conundrum of the United States Medical Licensing Examination score reporting structure
.
JAMA
.
2019
;
322
(7)
:
605
606
.
12. 
Lin
GL,
Nwora
C,
Warton
L.
Pass/fail score reporting for USMLE Step 1: an opportunity to redefine the transition to residency together
.
Acad Med
.
2020
;
95
(9)
:
1308
1311
.
13. 
Makhoul
AT,
Pontell
ME,
Kumar
NG,
Drolet
BC.
Objective measures needed—program directors' perspectives on a pass/fail USMLE Step 1
.
N Engl J Med
.
2020
;
382
(25)
:
2389
2392
.
14. 
Boulet
JR,
Pinsky
WW.
Reporting a pass/fail outcome for USMLE Step 1: consequences and challenges for international medical graduates
.
Acad Med
.
2020
;
95
(9)
:
1322
1324
.
15. 
The Coalition for Physician Accountability
.
Recommendations on 2021–22 Residency Season Interviewing for Medical Education Institutions Considering Applicants from LCME-Accredited, U.S. Osteopathic, and Non-U.S. Medical Schools.
2021
.
16. 
Angus
SV,
Williams
CM,
Stewart
EA,
Sweet
M,
Kisielewski
M,
Willett
LL.
Internal medicine residency program directors' screening practices and perceptions about recruitment challenges
.
Acad Med
.
2020
;
95
(4)
:
582
589
.
17. 
American Urologic Association.
Introducing Preference Signaling Pilot into the Urology Match
.
2021
.
18. 
Chang
CWD,
Pletcher
SD,
Thorne
MC,
Malekzadeh
S.
Preference signaling for the otolaryngology interview market
.
Laryngoscope
.
2021
;
131
(3)
:
e744
e745
.
19. 
Association of American Medical Colleges.
Supplemental ERAS Application Guide. 2021.
2021
.
20. 
McDonald
FS,
Jurich
D,
Duhigg
LM,
et al
Correlations between the USMLE Step examinations, American College of Physicians In-Training Examination, and ABIM Internal Medicine Certification Examination
.
Acad Med
.
2020
;
95
(9)
:
1388
1395
.
21. 
Rayamajhi
S,
Dhakal
P,
Wang
L,
Rai
MP,
Shrotriya
S.
Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?
BMC Med Educ
.
2020
;
20
(1)
:
79
.
22. 
Desai
A,
Hegde
A,
Das
D.
Change in reporting of USMLE Step 1 scores and potential implications for international medical graduates
.
JAMA
.
2020
;
323
(20)
:
2015
2016
.
23. 
Rubright
JD,
Jodoin
M,
Barone
MA.
Examining demographics, prior academic performance, and United States Medical Licensing Examination Scores
.
Acad Med
.
2019
;
94
(3)
:
364
370
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.