Mission-Based Filters in the Electronic Residency Application Service: Saving Time and Promoting Diversity

In February 2020, the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) announced that the United States Medical Licensing Examination (USMLE) Step 1 three-digit score would be reported as pass/fail starting as early as January 2022. A debate has since ensued about the potential impact on the residency selection process. Many programs receive more applications than they can manually review and must use filters to identify desired candidates for a more thorough evaluation. Prior data suggest that program directors (PDs) utilize USMLE score filters for this purpose.¹  However, heavy reliance on score filters can exclude applicants traditionally underrepresented in medicine (UiM); and approaches, such as mission-based holistic review, that give balanced consideration to experiences, attributes, and scores, have been shown to improve residency program diversity.^2–6  Yet, there is limited evidence on practical approaches to implementing holistic principles in screening without reviewing all applications.

Although the long-term impact of holistic review on institutions, physician quality, and patient outcomes has not been described, it is well recognized that diversifying the US physician workforce is an important strategy to combat disparities.^7–10  Thus, many hoped the transition to pass/fail score reporting would represent an opportunity to advance holistic review to increase diversity in graduate medical education (GME).^11,12  However, others voiced concerns that eliminating Step 1 scores would simply shift emphasis to other biased indicators of applicants' worthiness, such as USMLE Step 2 scores, medical school reputation, or social connections.¹³  Furthermore, international medical graduates (IMGs), who fill nearly a quarter of positions in the US Match and a higher proportion of primary care specialties, represent an especially vulnerable population, since USMLE scores represent one of the few ways IMGs have traditionally distinguished themselves.¹⁴ 

Unfortunately, the COVID-19 pandemic turned out to be a much more immediate disturbance to GME recruitment, as clinical rotations and USMLE examinations were abruptly cancelled. In response, the 2020–2021 application cycle was delayed, and the Association of American Medical Colleges (AAMC) recommended virtual interviews for all programs, a recommendation now extended into 2021–2022.¹⁵  These adjustments promote short-term safety and have potential long-term benefits, but they could substantially impede the movement toward holistic review in GME. Virtual interviews may encourage applicants to apply more broadly, especially without away electives to demonstrate interest to a specific program. Notably, a recent survey of PDs indicated that rising application numbers reduce the likelihood of holistic review.¹⁶  Although a few specialties are now piloting strategies such as preference signals and supplemental applications to address challenges in GME selection,^17–19  large application pools still make detailed holistic review untenable.

Concurrently, the COVID-19 pandemic has highlighted dramatic racial/ethnic health disparities, further demonstrating the need to bolster efforts, such as holistic review in recruitment, that enhance physician workforce diversity. Thus, there is an urgent need for practical approaches to implementing principles of holistic review in the context of the current Electronic Residency Application Service (ERAS). This will allow PDs to identify diverse, mission-aligned applicants well suited for their programs without requiring substantially more time than score-based filters alone.

Here, we describe the development and use of mission-based “experiences and attributes filters” (EA filters) as the initial tool to sort applicants in ERAS. We hypothesized that EA filters would increase the proportion of UiM applicants reviewed without a meaningful difference in average USMLE scores among applicants selected by EA filters and applicants selected by metric-based filters. Based on test characteristics and previous data correlating USMLE performance with board pass rates,^20,21  we defined, a priori, a meaningful change as more than 10 points (half a standard deviation of mean USMLE scores in previous years).

To demonstrate the feasibility of employing holistic review principles using existing ERAS filtering capabilities across a spectrum of GME programs, we assessed the 2019–2020 categorical applicant pool for 3 specialties: internal medicine (IM), pediatrics, and surgery. The IM program, the largest at McGovern Medical School (MMS), receives approximately 3500 categorical applicants, reviews about 1300 applications, and matriculates 40 interns annually. The pediatrics and surgery programs each receive approximately 1300 to 1400 applications, review 300 to 400 applications, and fill 24 and 8 intern positions, respectively. The original approach to holistic review in the IM residency has been previously described.²  At baseline, all 3 programs utilize a metric filter based on USMLE Step 1 cutoff with some combination of attributes filters that include Spanish language proficiency, UiM status, and Texas residency as the initial steps to narrow applicants for detailed review. Selected applications are subsequently reviewed manually for other academic merits, attributes, and experiences that would contribute value to each program. Full application review takes up to 10 minutes.

All authors (PDs and associate PDs of the 3 programs) were familiar with the concept of holistic review and use of limited ERAS filters at baseline. After completing the 2019–2020 application cycle, we convened to discuss the best practical approach to implementing a mission-aligned holistic review process without relying on USMLE score filters. ERAS is equipped to filter applicants by various characteristics beyond test scores, such as language proficiency, and for free text searches within the Professional and Training Experience section to identify all individuals with a desired experience (eg, prior work as a “teacher”). Using the medical school's mission statement as a template, the team identified the core values best aligned with their respective program's goals. Then, we enumerated a potential list of relevant applicant experiences or attributes illustrative of each core value (Table 1). We subsequently attempted to translate each potential experience/attribute into one or more searchable EA filters in ERAS based on the available filter categories. For instance, for the MMS core value of “promoting interprofessional collaboration,” one program described prior experience as a Scribe as an applicant trait that might illustrate this core value. Therefore, we used a free text search in ERAS for “Scribe” under the Professional Experience Position section to filter applicants whose prior professional position contained “Scribe.” Table 1 provides details about each core value and corresponding EA filters selected.

Each program then retroactively applied their chosen EA filters to their 2019–2020 categorical applicant pools in ERAS and exported de-identified datasets as an Excel document. We compared the total number of applicants captured by each program's desired EA filters with the number of applicants typically reviewed during a recruitment cycle. This initial pool exceeded the usual number of applications reviewed by over 300 for each program, surpassing available faculty resources for thorough review in an actual recruitment cycle. Therefore, each program's representatives (associate PD and/or PD) further prioritized and narrowed their core values based on respective program goals. This pilot involved only the authors in the design and finalization of EA filters.

The process of enumerating potential experiences/attributes to match MMS core values and defining ERAS search filters occurred over 4 meetings among authors lasting up to 1.5 hours each. Applying the EA filters in ERAS to applicant pools for all 3 programs took 1 hour, and we convened for another hour to finalize core values. For each final core value chosen by a program, we identified the number of unique individuals captured by the corresponding EA filters. Then, we calculated the proportion of selected applicants who were female, UiM, and IMGs as well as mean USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores. We categorized applicants as UiM if they self-identified as American Indian, Alaska Native, Black/African American, Hispanic/Latino, Native Hawaiian, or Pacific Islander. We categorized gender as self-identified in ERAS. Finally, we compared the demographic characteristics and USMLE scores of applicants identified by mission-based EA filters with those of applicants actually selected for detailed review by each program during the 2019–2020 application cycle. We compared continuous variables with a t test and categorical variables with a chi-square test. Data were analyzed using Microsoft Excel and SAS 9.4 (SAS Institute Inc, Cary, NC). This study was deemed exempt by the Institutional Review Board of the University of Texas at Houston.

Table 2 describes characteristics of categorical applicants for all 3 programs during the 2019–2020 application cycle as well as characteristics of applicants selected by the mission-aligned EA filters. The IM program prioritized 5 MMS core values, and the corresponding EA filters identified 1301 applicants for detailed review (Table 2). Among them, 713 (54.8%, 95% CI 52.1–57.5) were UiM, and this was significantly higher than the proportion of UiM applicants (22.7%, 95% CI 20.4–24.9) reviewed in the 2019–2020 recruitment cycle (P < .0001; Table 3). Among 514 applicants identified for detailed review by the pediatrics program's EA filters, 325 (63.2%, 95% CI 59.1–67.4) were UiM compared to 97 (25.3%, 95% CI 20.9–29.6) UiM applicants reviewed in 2019–2020 (P < .0001; Table 3). The mean USMLE Step 1 scores of applicants selected by EA filters for IM and pediatrics were lower than mean scores of applicants reviewed in 2019–2020 (Table 3). The surgery program did not utilize the Application Reviewed function in ERAS; therefore, we were unable to compare applicants reviewed in the 2019–2020 recruitment cycle with those identified by EA filters.

Three residency programs at MMS used their institutional core values to craft individualized ERAS search strategies using EA filters as the initial screening tool to select applicants for further review. Compared to our baseline screening process that included USMLE score filters, this mission-based approach incorporating holistic review principles improved the diversity of the selected applicant pool with respect to self-identified gender and race/ethnicity. We found that applicants identified through mission-based EA filters had slightly lower mean USMLE scores than those selected for review during the actual application cycle. Although some data support the role of USMLE in predicting licensing examination pass rates,^20,21  evidence suggests other factors have better predictive value than Step 1 alone.^20,21  It is unlikely that the 3- to 5-point mean differences in our study will translate into appreciable differences in board pass rates or predict future success as a physician. Nevertheless, we did not thoroughly review applications identified with EA filters to identify risk factors for board failure. We present EA filters as an efficient and holistic approach to initially screen large application pools in anticipation of the USMLE Step 1 pass/fail change.

It is important to note that the purpose of holistic review methods is not to eliminate academic metrics. Holistic review necessitates broad selection criteria that promote various types of diversity, and identifying applicants using multiple data points potentially reduces the impact of bias inherent in any one factor. Hence, similar to the limitations of exclusively using score-based filters to narrow applicant pools, a singular EA filter would not attain broad-based diversity. As a state institution serving a diverse population and training physicians for Texas, we prioritized the initial use of EA filters to identify applicants who can contribute to this mission. This would be followed by a manual review of academic performance to select applicants for interview. Other programs can combine metric-based and EA filters as their initial sorting tool in alignment with their respective goals. Additionally, if PDs use USMLE scores as one component of a multifaceted screening strategy, test scores can allow students to demonstrate academic excellence without becoming the focal point of residency recruitment. This could potentially promote the longevity of USMLE Step 2 CK reporting as a 3-digit score. Overall, there is a need for further assessment of the most balanced approaches to recruit diverse trainees who can meet the needs of an increasingly diverse US population and alleviate health disparities.

Nonetheless, the challenge remains in balancing the number of EA filters with a program's capacity for thorough application reviews. In this instance, each program had to narrow its initial set of desired EA filters to accommodate available resources for detailed application review. The surgery and pediatrics programs captured at least 150 more applicants for review compared to baseline, and without application caps, programs must make individualized decisions to add resources to review more applications or restrict EA filter options. The initial process of developing a new mission-based search strategy might be time-consuming, especially if it involves many stakeholders. Programs should also consider the need to train personnel on the principles of holistic review and the use of ERAS filters. However, once defined, EA filters can be applied with small modifications to future applicant pools, since program missions do not change drastically year to year. In contrast to expectations that IMGs will have a harder time distinguishing themselves without Step 1 scores,²²  we found that the use of EA filters might provide an opportunity to increase the number of IMGs reviewed by programs. However, program leaders with a mission to enhance IMG representation will need an intentional approach, and further research is needed to identify best practices.

While each program utilized a few filters based on experiences and attributes during usual recruitment, we found that this intentional and comprehensive mission-based approach led to greater diversity among applicants selected for review. However, our study is limited to the application review stage and may not enhance diversity among candidates interviewed, ranked, or matched. Future efforts will assess the ongoing impact of mission-based filters on the diversity of matriculants. It is also unclear if EA filters developed by other programs will produce similar results with respect to diversity. We tracked gender and race/ethnicity as markers of diversity because they were available as self-reported data in ERAS. We cannot assess the impact of our EA filters on other potential groups of interest that would signify broad-based diversity (eg, applicants with disabilities, LGBTQIA+ applicants, etc). The absence of comparison data for surgery further limits our findings to primary care fields. Notably, expanding access to primary care is also important for addressing health disparities. Hence, the approach of EA filters might provide an opportunity for future assessments of the most effective filters to identify applicants who eventually choose primary care careers. Additional studies are needed to understand how EA filters might impact surgical and specialty applicants. This study focused on a small part of the complex system of undergraduate medical education (UME) to GME transition. However, selecting applicants using a multifaceted, mission-driven approach based on holistic review principles could be an important piece in addressing inequity in GME recruitment and diversifying the health care workforce.

We also recognize ERAS remains limited in its search features, and some programs may not easily identify relevant EA filters. Free text searches could miss appropriate applicants who use a different key word; thus, utilizing a database of inclusive key words (eg, “instructor” vs “teacher”) might be necessary. Furthermore, searches and filters in ERAS are currently limited to certain applicant-generated entries. Many documents cannot be searched with our EA filter approach, including personal statements, letters of recommendation, and medical school performance evaluations. Expanding the availability of searchable sections might also prove useful. For example, programs could search for applicants who at least 3 letter writers described as “compassionate.” Moreover, there is a broader need within medical education for implementation of objective appraisals of skills critical to success as a physician, such as effective collaboration, communication, leadership, and resilience. Increasing the number and objectivity of metrics, experiences, and attributes filterable in ERAS would facilitate programs' ability to find the applicants best poised to contribute to the institution's mission. It would also give more opportunities for applicants to demonstrate excellence in line with a program's mission. However, just as USMLE scores might exhibit bias,²³  it is possible that other elements entered in ERAS could perpetuate inequity. Therefore, caution must be taken to adequately scrutinize any new appraisal methods for true objectivity.

We demonstrate that the use of mission-based experiences and attributes filters is currently feasible in ERAS and could enhance diversity of applicants selected in the context of application inflation and time constraints.