Using a Consensus-Driven Approach to Incorporate Holistic Review Into an Internal Medicine Residency Program

There is a growing interest in incorporating mission-based holistic review within residency admissions. Holistic review may better align residency selection criteria with institutional values and increase program diversity.^1-5  Furthermore, many traditional residency selection metrics, such as United States Medical Licensing Examination (USMLE) scores and Alpha Omega Alpha (AOA) membership, are not strongly predictive of residency performance.^6-8 

However, limited guidance exists on how to practically implement holistic review in residency admissions, including how to identify or assess applicant attributes that align with institutional values.^2,3,9-13  Only about half (54%) of program directors report using the Association of American Medical Colleges’ holistic review framework in application review.⁵  It is unclear if it is feasible to use holistic review for all applications, and many programs rely on nonholistic approaches to screen and rank applicants.^5,11,14  Moreover, Electronic Residency Application Service (ERAS) applications provide limited information on family background, disadvantaged status, or other attributes that may support holistic review.

To provide practical guidance on implementing holistic review, we aim to describe our experience incorporating a holistic review process into an internal medicine (IM) residency program using a consensus-driven approach to identify desired applicant attributes.

The University of Wisconsin IM residency recruits 25 categorical residents annually. The program receives approximately 1200 applications from US medical graduates and 1000 from international medical graduates (IMGs) each year.

Prior to implementing our new system, we selected applicants for interviews and generated an initial rank list using USMLE scores, AOA membership, Gold Humanism Honor Society (GHHS) membership, medical school class rank, clerkship grades, strength of narrative comments from the Medical Student Performance Evaluation (MSPE) and letters of recommendation (LOR), and perceived medical school quality (as determined by a discussion among residency leadership). We refined our rank list through consideration of additional factors, including faculty interviews.

During the 2020-2021 academic year, we implemented a novel holistic review process. Our goal was to achieve consensus within our educational team regarding the attributes we sought in residents and to recruit residents who possessed these qualities.

We first solicited potential attributes from a stakeholder group consisting of the program director (PD), 5 associate program directors (APDs), program coordinators, chief residents, the vice chair for education, and the chair of the Department of Internal Medicine. We further refined and consolidated these characteristics through group discussion and consensus. Stakeholders then completed an anonymous survey (Qualtrics, Provo, UT) wherein they were given a set number of points to distribute among the different characteristics, resulting in a weighted set of attributes.

Because we felt we could not feasibly perform holistic review on all applications, we performed an initial screen using perceived medical school quality, class rank, and clerkship grades, which are easily accessible through ERAS (Figure). We extended a small number of interviews to candidates whom we felt would likely be granted interviews (eg, local students, individuals with outstanding grades and class rank) and declined some candidates based on screening metrics without an initial holistic review. We reviewed IMG applications through a separate process. Prior to interview day, the remaining applications were reviewed by an APD or the PD who sought evidence of the designated attributes through review of the entire MSPE. We awarded the applicant points if they demonstrated exceptional achievement in one or more of the attributes. We allocated 2 points for the most heavily weighted attributes and 1 point for the less highly ranked attributes (Table 1). Prior to this review, we conducted an hourlong training session for the PD and APDs, wherein we worked through several sample applications to reach consensus about point allocation.

On interview day, a second APD or the PD reviewed each interviewee’s entire application packet, including personal statement, CV, and LORs, and could allocate points if not previously awarded. To capture certain characteristics that were difficult to discern from the application, we developed standardized interview questions relating to these attributes (Table 2). Each applicant interviewed with the PD or an APD, who, for the purposes of time, randomly chose 2 of the standardized questions to ask. Recognizing that most applicants likely possess these attributes to some degree, and wishing to not dilute the value of these points in our process, we sought to award points only when attributes were present to an exceptional degree, with a goal of 10% to 15% of applicants receiving additional points for any given attribute.

We reduced the weight of certain elements in our ranking algorithm that we felt may contribute to bias, including USMLE scores and AOA membership.^15,16  Finally, we increased the weight of GHHS membership as we felt this aligned with many of our prioritized attributes.

In late 2021, following application review for the 2021-2022 cycle, we surveyed stakeholders who participated in the consensus process regarding their perspectives of the experience (survey provided as online supplementary data).

As per University of Wisconsin-Madison policy, we self-certified that this was a program evaluation and did not meet criteria for Institutional Review Board review.

Our initial group work yielded 22 distinct attributes valued by stakeholders. Many of these were character traits (eg, curiosity and passion for learning, tolerance with ambiguity). The attributes also reflected prior experiences (eg, diversity of skills/experiences) and accomplishments (eg, clinical aptitude, demonstrated integrity and professionalism). The initial list of attributes also included characteristics such as medical knowledge, long-term plans to work in Wisconsin, and commitment to the field of IM.

Our process resulted in a final list of 10 attributes (Table 1). We removed an attribute (“the IT factor”) due to concern that this may be prone to implicit bias. We removed clinical aptitude because we felt that we already attempted to assess it elsewhere within the review process (eg, clerkship grades, MSPE narrative comments). We combined “understanding others’ perspectives” with “compassion and empathy.” We did not explicitly build cultural/ethnic diversity into our holistic review process, acknowledging that our program and institution had multiple other concurrent interventions to promote diversity, equity, and inclusion. Finally, we added an additional attribute (“community service”) that aligned with the chair’s vision for the department. Our process to identify and finalize attributes took approximately 6 hours to complete.

Out of approximately 1200 initial applicants, 549 (45.8%) were offered an interview and 443 (36.9%) applicants attended an interview. Among these 443, 135 (30.5%) were allocated points for possessing one of the attributes; the mean number of points per applicant receiving additional points was 1.65 (range 1-5). The gender and race-ethnicity demographics were similar between those who received additional points and those who did not. Only 17 applicants were recognized to have exceptional achievement within more than one domain (15 in 2 domains, 2 in 3 domains). The mean scores for all ranked applicants was 82.8 (range 69-88). Among the 25 matched categorical IM residents, 8 (32%) were allocated points for having exceptional achievement within one of the attributes. While the exact impact of holistic review on match outcomes is challenging to assess, 4 members of the incoming intern class (16%) would have dropped to a lower rank position than our historical cutoff for matched residents had they not received additional points for attributes.

The most recognized attributes were exceptional community service (42 of 443, 9.5%), grit/resilience/growth mindset (33 of 443, 7.4%), and extraordinary leadership (27 of 443, 6.1%). Attributes for which points were rarely awarded included integrity and professionalism (0 of 443, 0%), strong work ethic (2 of 443, 0.5%), and curiosity and passion for learning (4 of 443, 0.9%).

Among 13 individuals who participated in the consensus process, 12 (92.3%) completed a survey. All participants indicated that the final list of attributes was either a “very accurate” (6 of 12, 50%) or “somewhat accurate” (6 of 12, 50%) representation of what they felt should be included. Overall, respondents indicated their voice was heard during the process of identifying attributes (mean response 4.6 on a scale of 1 to 5, with 1 being “not at all” and 5 being “to a great extent”).

The 6 survey respondents who were an APD or PD reported spending on average between 10 and 30 minutes reviewing a residency application (mean=17.8, SD=6.6). They estimated that the process of awarding additional points for attributes required between 2 to 5 minutes per application (mean=3.8, SD=1.2). Of the 6 respondents, 4 (66.7%) felt that they were able to complete this process more quickly during the 2021-2022 cycle than during the initial 2020-2021 cycle.

Survey respondents were asked which elements of the application material were most helpful in identifying desired attributes (Table 3). Standardized interview questions were thought to be the most useful tool (mean=6.7 on 0-10 scale, with 10 being most helpful), and LORs least useful (mean=3.9).

We queried respondents about proposed strategies to optimize assessment for attributes, and suggestions included standardized letters of evaluation from medical schools, additional personal statements and questions, and more time for interviews. Faculty reported that community service and diversity of skills and experiences were the easiest attributes to identify, whereas curiosity for learning and professionalism were the most challenging. For the attributes that were rarely awarded points, most respondents (5 of 6, 83.3%) felt that this reflected difficulty in ascertaining this attribute from the available materials. One respondent wrote that “in general, the quality of the information provided in the application is low.” Another individual indicated that it was “difficult to determine standouts” since most applicants displayed these attributes to some extent.

Using an iterative process with stakeholder engagement, we revised our residency selection criteria to identify new applicant attributes that better reflect our program and department mission and culture. We also de-emphasized metrics that may be less reflective of our values, prone to implicit bias, or poorly predictive of residency performance.

We observed several challenges to implementing holistic review. We relied on a small cohort to review applications and conduct standardized interviews, which required a significant time investment. Faculty spent an additional 3.8 minutes on average per applicant and did not receive additional protected time to complete holistic review, though most faculty expected to require less time in subsequent application cycles. We were not able to apply a holistic approach equally across all stages of our selection process. The sheer number of applications and the pressure to provide timely interview invitations made it challenging to use a fully holistic approach during initial screening. Other programs have explored using ERAS filters or machine learning approaches as potential strategies to prioritize applicants.^11,17  Many applicants embody strong work ethic and professionalism, and so identifying “exceptional” achievement may be difficult. It may be more important to identify negative (rather than positive) outliers, and our process was not initially conceived to do so.

Our program faculty also indicated that it was challenging to assess applicant strengths through the current application materials, underscoring the need for reform within residency application materials. The use of standardized letters of recommendation and interview questions may not only be more conducive to holistic review but may also help mitigate bias.^18,19  The ERAS supplemental application, components of which will be incorporated into ERAS in 2023-2024, may help programs identify applicants who are well-aligned with their institution.²⁰ 

There are important limitations to our work. First, our single-institution experience and approach may not be generalizable to other settings. However, given the intrinsic mission-driven nature of holistic review, selection criteria must be adapted to each program. Second, the long-term outcomes of our revised selection criteria are unknown. Third, our stakeholder group was limited to those traditionally participating in recruitment. Fourth, due to time constraints, we selected 2 standardized interview questions per applicant, which may introduce bias. Finally, while we de-emphasized certain selection metrics that may be prone to bias, other new biases could have been introduced into our process. We tried to mitigate bias by incorporating multiple reviewers and metrics into our assessment process.

The constraints of the current application materials and process were felt to be a potential barrier to implementing holistic review at our institution. However, the optimal materials and processes to support holistic review are uncertain, and future work should address how to refine these tools. Future work will examine the impact of this holistic review process on educational outcomes, including selection as chief resident, need for remediation, and post-residency career paths.

The addition of holistic review to our residency admissions process was achieved using a consensus-driven approach to identify desired applicant attributes with initially positive feasibility and acceptability data.

The online version of this article contains the survey used in the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

A preliminary version of this work was presented at the APDIM Virtual Conference, September 23-25, 2021.