Initial Experiences With a 2-Stage Residency Interview Process

During the COVID-19 pandemic, residency interviews primarily transitioned to a virtual format, which has continued in the post-COVID-19 Public Health Emergency era.¹  The Association of American Medical Colleges (AAMC) and the National Resident Matching Program (NRMP) recommend the virtual interview format and discourage hybrid interviewing.^2,3  Applicants have appreciated virtual interviews for their convenience and cost saving as well as the ability to attend more interviews.^2,4,5  The AAMC also reports benefits on the environmental impact of reduced travel.² 

However, applicants and programs have reported concerns with the virtual interview process.

Program directors reported more difficulty engaging applicants and higher reliance on their program website.^5,6  They also noted more time to train staff on software usage and having technical issues during interviews.^5,7  Studies have shown that interviewers rate applicants lower virtually.^8,9  Programs expressed difficulties assessing an applicant’s fit with the program as well as their interpersonal skills.^5-7  Program directors expressed concerns with understanding applicant interest in the program as well as if their program is accurately reflected.^5-7  Residents have reported that the quality of interaction in virtual socials is inferior to in-person.⁶ 

The 2021 NRMP Applicant Survey showed that geographic location and “goodness of fit” were the highest considerations in ranking.⁵  However, applicants reported that the virtual process is not sufficient to elucidate a program’s culture and fit with the program’s faculty or residents.^4,5,10,11  They also expressed a desire to get a sense of the city where they would be living.^10,12  Another major area of concern was over software issues leading to less fluid conversations.^5,13-15  Surveys including the 2021 NRMP survey reported that most applicant responders preferred in-person interviews.^5,16 

One option to improve assessment of fit on the applicant side is a post-evaluation second look after a program has finalized its rank list.¹⁷  This has been adopted by many surgery programs as recommended by the Association of Program Directors in Surgery Task Force.¹⁷ 

We describe an alternative strategy to the interview process combining both in-person and virtual interviews. Our hypothesis is that this dual approach would be satisfactory to applicants and faculty and feasible to administer.

A novel multistage approach to the interview process at our general surgery program was introduced for the 2022-2023 application cycle. This process began with a holistic review of all applications from September through October 2022. Each application submitted to our program for a categorical position was reviewed by 2 separate reviewers who were drawn randomly from a 30-person volunteer committee consisting of faculty and residents. These individuals were provided instructional materials on how to conduct a holistic review, including 20 minutes of instructional videos on holistic review from the AAMC,¹⁸  1 hour of a departmental lecture on implicit bias, and 1 hour to complete Harvard’s Project Implicit Social Attitude tests.¹⁹  All reviewers were instructed to return an application to our coordinators for redistribution if they felt they could not objectively review an application assigned to them. No screening criteria were utilized for applicants, and every applicant was included in this holistic review. Each reviewer was expected to review roughly 100 applications each, taking around 10 minutes per application. This time was estimated based on multiple prior years of holistic review.

From October through November 2022, selected candidates (planned for 10% of applications reviewed) were invited for a virtual interview. There were 13 interview dates planned consisting of a prerecorded introduction from the program director, and two 25-minute interviews with a resident and faculty. Throughout November, there were also 4 virtual socials that the candidates had their choice to attend. A score based on candidates’ degree of leadership potential, teamwork compatibility, history of integrity, and desire to serve in an underserved community was assigned to the applicant from these interviews. The applicants who received the highest scores (planned for top 80) during their virtual interview were invited for an in-person interview.

From December 2022 through February 2023, there were 4 in-person interview dates planned. The in-person interview consisted of two 25-minute interviews with faculty, one 25-minute interview with a resident, four 10-minute interviews with residency leadership (surgery department chair, program director, and associate program directors), campus tours, and an in-person social. The NRMP policy office was made aware of this protocol prior to its start to ensure no violations occurred. At their suggestion, we also included a fifth interview day as a virtual option for this second-stage interview, specifically for applicants who were unable to accommodate a campus visit due to time and/or financial constraints. The intent behind offering an in-person interview rather than a nonevaluative campus visit is that we felt any in-person interactions with applicants during the interview season would change our perceptions of those individuals either positively or negatively. Therefore, we could not guarantee that the meeting would be truly nonevaluative. The virtual screening interview was used only as a determination of who should be offered an in-person interview opportunity, and the score from the first interview did not contribute to final ranking.

The exception to this application protocol was for our own institution’s medical students. They were all offered a virtual interview. They also all met individually with the program director and chair of surgery. Ultimately, as in previous years, their ranking was based on their summative performance during their surgery rotations rather than their interviews.

In March, after the rank lists were submitted but before Match day, all interviewed applicants and faculty were asked to complete a survey (surveys provided as online supplementary data). These surveys are routinely performed after the end of each interview cycle at University of Texas Southwestern (UTSW) with specific survey questions written by the program coordinator to gauge issues and benefits of the interview process for that year.

A Pearson’s correlation coefficient was used to evaluate if making it to the second round impacted the respondent’s opinions. The program rank list was also compared to the Match list to evaluate the demographics of the candidates. The schedules of the interview days were evaluated to obtain estimates of faculty time spent during the interview process.

The project was submitted to the institution’s Surgical Partners in Research board for review and found to be institutional review board exempt.

Out of the 1175 applications received and reviewed holistically, 190 (16.2%) candidates were invited, and 188 of 1175 (16.0%) interviewed virtually. Of the 188, 82 candidates (43.6%) were selected for second round interviews, 69 of 82 (84.1%) completing them in-person and 13 (15.9%) virtually. This breakdown of candidate selection is illustrated in the Figure. Out of all interviewed candidates, 68 of 188 (36.2%) answered the post-interview survey. The demographics of these respondents are described in Table 1.

Nearly all candidate respondents (63 of 68; 92.6%) agreed that the 2-stage interview process was fair. Fifty-one of 68 candidates (75%) felt that nonevaluative second looks were not truly nonevaluative, and only 22 (32.4%) did not feel pressured to attend nonevaluative second looks. From the candidates who attended the in-person second round, all 39 agreed that they met enough people to get a sense of the program and that the in-person social event was useful. This compares to 42 of 68 candidates (61.2%) who felt that the virtual social event was useful. This data is summarized in Table 2.

Of the total responders, only 5 of 68 (7.5%) were UTSW medical students. There were 10 UTSW students who interviewed. Within this cohort, 4 of 5 (80%) felt that the meetings they had with program leadership were not evaluated the same as being part of the second round interviews and that that this interview structure did not serve them well.

Out of all applicants, 2 matched from UTSW Medical School. All other applicants who matched attended the in-person interviews. However, 4 out of 13 applicants who did the second interview virtually were within matching ranking but did not match.

There were 15 faculty members who participated in holistic review and interviews, and 9 more who participated only in interviews (24 total). With the holistic review taking 10 minutes per application and 100 applications per reviewer over 6 weeks, this resulted in roughly 3 hours of holistic review per week per faculty. For the first stage of interviews, the faculty were given 3 hours of interviews (6 interviews each) and completed 4 days of interviews. For the second stage of interviews, the faculty were given 2.5 hours (5 interviews) and completed 2 days of interviews.

Twelve of 24 (50%) of faculty responded to the survey. All 12 faculty reported they would participate in the process again. Table 3 summarizes the faculty survey responses.

Our 2-stage interview process was positively received by applicants and faculty. It was considered to be fair and allowed applicants to get a good sense of the program. Most of the respondents preferred in-person interviews and felt that nonevaluative second looks were not truly nonevaluative.

In the months leading up to the end of the COVID-19 Public Health Emergency, the interview process saw a transformation into virtual interviews focusing on reduction of cost, equity and diversity, and transparency of their processes and application criteria.^20-22  We conceptualized this 2-stage process to include these benefits of the virtual interview while also allowing for the benefits of the in-person format. While we could not subsidize costs for every applicant, we felt the virtual screening interviews provided a low-cost entry to exploring our program (and for our program to better learn about our applicants). By reducing the number of applicants visiting our campus by nearly 50% from pre-COVID-19 interview seasons, we achieved cost reduction from a global perspective. Applicants also agreed that the in-person visit was a good use of their money. We were therefore able to maintain one of the main benefits of virtual interviews perceived by applicants.^14,23 

Our aim was to overcome the main weakness of virtual interviews: that applicants could not experience the program address applicant and program concerns on getting a sense of a program, the main weakness of virtual interviews. Our survey results show that with the 2-stage format, the applicants no longer had difficulty gaining a sense of the program as previously reported in other studies.^11,15,16  Our approach incorporates both virtual and in-person interviews but is different from previously reported hybrid structures that offer optional in-person interviews or events.^24,25  These led to applicants feeling pressured to come in-person and introduced a sense of bias.^24,25  Our method was largely perceived as being fair. Though, we found that those who did not advance to the second round did note an inverse correlation with the sense of respect the process offered them. While we attempted to be forthcoming about the process, the timeline required to do 2 rounds of interviews is condensed. It is likely this condensed timeline, while equally applied and thus fair to all candidates, contributed to the perception that the process was disrespectful to those who did not advance. Those who advanced to the second round were likely to still feel that the program respected them, as we clearly acknowledged their strengths as applicants by advancing them. The area in which we discovered that we may have introduced an inequity was with our own home institution’s students, who preferred to have a complete interview experience offered to them.

There are several limitations to note with this study. This interview process required a burden on faculty—in the holistic review, the virtual interview, and the in-person interview—but this was overall well received in the evaluation of their experience. Therefore, at our institution we deem this process as feasible to continue going forward. However, this process was for an intern class size of 13 residents. For other specialties with much larger class sizes and application pools, the burden on the faculty would be even larger, which may limit the generalizability of this structure. Additionally, the overall survey response rate (36.2%) was low, especially among the applicants who only attended the first virtual interview. It is unclear what drove this response rate, but it may be due to lack of interest in the program after not being invited to the second in-person round, survey fatigue, or timing of the survey at the end of the application cycle. Furthermore, the survey data was also limited to individuals who applied to our program, which, as a large residency program in an urban setting, may attract a specific type of applicant. Finally, we only surveyed those who were chosen for at least a first-round interview with us, which introduced further bias into our assessment.

Our novel interview approach offers a new fair blend of virtual and in-person methods to the residency application process, gleaning benefits from both structures of assessment. In the future, we are likely to continue with this interview structure and offer it as an example for other programs to consider.

This study shows that returning to in-person general surgery residency interviews is possible without causing an undue burden on applicants or faculty. The in-person approach is beneficial for applicants in obtaining a sense of the program and perceived as acceptable by participating faculty.

The authors would like to thank Lisa Bailey, Herbert Zeh III, MD, Angela Guzzetta, MD, Ryan Dumas, MD, John Mansour, MD, Valerie-Sue Emaukhagbon, MD, the holistic review committee, and all other faculty and residents who assisted with the interview process.

The online supplementary data contains the surveys used in the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

This work was previously presented at the Association of Program Directors in Surgery Annual Meeting, April 10-15, 2023, San Diego, California, USA.