Multicenter Study of Optional In-Person Visits to Residency Programs After Virtual Interviews

At the onset of pandemic travel restrictions, virtual residency recruitment replaced in-person interviewing. Virtual interviewing reduces costs for applicants and programs,^1-10  lessens the environmental impact of travel,^11,12  and minimizes medical school disruption, making the recruitment process more equitable for applicants.^6,7,13  Stakeholder organizations that have recommended exclusively virtual recruitment have also recommended research to better understand it.^14-16 

Despite the benefits of virtual recruitment and applicants’ general satisfaction with the process,^{1,3,5,6,9,10,17-21}  applicants’ ability to make an informed, high-stakes decision about which residency program is the best “fit” for them is diminished in a virtual-only format,^{1,3,5,10,18-24}  and applicants prefer in-person interviews over virtual.^1,6,24,25  In an experiment forced by the abrupt cessation of in-person interviewing for surgical fellowship in early 2020, Grova and colleagues directly compared opinions of applicants who had completed virtual or in-person interviewing and found that applicants who completed virtual interviews were significantly less likely to report they had an “adequate understanding of the culture of the program” (64% vs 100%) and that the “interview experience was sufficient to allow them to make a ranking decision” (54% vs 92%).¹⁸ 

A national survey that included 7593 internal medicine (IM) interns following the first pandemic recruitment season found that only 4% of interns who had completed a virtual-only recruitment preferred exclusively virtual recruitment over options that included an in-person component.²⁶  In this cohort and in 2 others, the preferred recruitment process consisted of virtual interviewing followed by an optional in-person visit.^19,26,27  Some applicants fear that programs may rank them lower if they choose not to visit a program when the opportunity exists, although we are unable to find evidence that this is true. Nevertheless, because of this applicant concern, the National Resident Matching Program (NRMP) is exploring a proposal for programs to voluntarily use a program rank list deadline weeks before applicants’ rank list deadline²⁸  to create a window of time in which applicants could choose whether to visit programs without fear of their rank position being jeopardized.

We studied a recruitment process for the 2022-2023 main residency match in which participating IM residency programs voluntarily finalized and certified their rank lists weeks before the formal rank list deadline, creating an opportunity for applicants to visit participating programs if they would find it helpful. We hypothesized that this process would be feasible for programs, that the financial costs, time expenditure, and administrative burden would be acceptable, and that applicants would find this process equitable, informative, and helpful.

We solicited volunteers from among the members of the Association of Program Directors in Medicine (APDIM) Survey Committee; a total of 6 IM residency programs agreed to participate. Participating programs included 3 from the South, 2 from the Midwest, and 1 from the Northeast regions of the United States, with 5 university-based programs and one community-based program. Participants included the program directors (PDs) and program administrators of each program, and applicants who interviewed with each program during the 2022-2023 residency recruitment cycle; preliminary program applicants were not included.

In order to participate in this study, each program agreed to finalize and not change its rank list at the beginning of February, then to host applicants who wanted to visit the program’s facilities, meet program leadership in a group format (no one-on-one meetings with program faculty allowed), meet with residents (group and individually permitted), and tour the program’s town. Specifically, participating programs each agreed to (1) secure the support and oversight of their designated institutional official (DIO); (2) finalize and certify their program’s rank list at least 3 weeks before the NRMP deadline; (3) not change their rank list based on applicants’ in-person visit decisions or behaviors after their program rank list was certified (with DIO oversight); (4) allow optional in-person visits after the program rank list finalization; and (5) not allow visiting applicants to meet in-person one-on-one with the PD, program leadership, or faculty.

Participating programs agreed to invite all interviewed applicants to an optional in-person visit except (1) applicants from their home institution, and (2) applicants who had completed a visiting rotation at their institution; these restrictions prioritized visits for applicants who had not previously had the opportunity to visit the program, given uncertain applicant interest in visiting and programs’ limited capacity and timeframe to accomodate in-person visitors. Each program hosted visiting applicants only on program-designated in-person visit dates. Participating programs were asked to share the numbers of applications their program received, the number of interviewed applicants, how many applicants had signed up for and completed in-person visits, how the in-person visits were structured, and how much the program spent on the in-person visits.

Surveys were designed collaboratively and iteratively by the participating authors who have extensive survey design experience working together on the APDIM Survey Committee. The in-person visit questions were adapted from a previously published survey of in-person visits following virtual interviews in the 2021-2022 application cycle.²⁹  Survey questions aimed to assess the stakeholders’ opinions about the value, equity, and potential downsides of this recruitment process. The surveys of applicants, PDs, and program administrators are available in their entirety as online supplementary data.

All applicants who were interviewed by any of the participating programs were emailed a survey using Research Electronic Data Capture (REDCap) on Match Day (March 17, 2023). Each applicant had a unique survey linked to their email address; nonrespondents were sent 3 reminder requests over the second half of March 2023, and the survey closed on April 17, 2023. Survey responses were anonymous. The surveys utilized “skip logic,” guiding respondents who completed an in-person visit to answer questions about their visit experience, while applicants who had not visited a program were asked to indicate the reasons they chose not to visit. Applicants who interviewed with more than one participating residency (defined by email address appearing on more than one program’s interviewed applicants list) were sent a variant of the survey that allowed the respondent to indicate which participating programs they had visited (if any), and, if they had visited more than one program, to choose one program about which they would answer in-person visit questions.

Descriptive data are provided. Chi-square test was used to compare survey responses between the different participating sites with alpha set at <.05 (using R version 4.3.0). Applicants who had applied to more than one of the participating sites were grouped together into a “Multiple” category, which was treated as an independent site. For analysis of questions with more than 2 response options, the most prevalent answer was compared with the combined group of less common responses (eg, “Yes” compared to the combined group of “No” and “Unsure” responses for “Was this recruitment process equitable?”).

The University of Rochester’s Research Subjects Review Board approved this study as the coordinating center, with each participating institution’s IRB approving their site’s participation. The applicant survey invitations included information about the study, and applicants’ completion of the survey was considered to represent informed consent.

Participating programs interviewed an average of 379 applicants (range 205-534) and completed an average of 39 in-person visits (10.3% of interviewed applicants, range 7.9%-12.8%). Programs spent, on average, $6,278 on in-person visit expenses ($162 per visiting applicant, range $50-435). While the characteristics of the in-person visits varied, programs all provided a tour of the hospital and facilities and lunch with residents. Two programs provided a guided tour of the program’s city, and one program paid for a night in a hotel the evening before the visit day (Table 1).

Participating programs interviewed 1808 unique individuals who comprised the applicant survey group, of whom 401 (22.2%) interviewed with 2, 3, or 4 participating programs (Table 2, “Multiple” column). Of the 1808 interviewed applicants, 464 responded to the voluntary survey (25.7% overall response rate); the response rate varied across programs (15.2% [45 of 296] to 36.1% [96 of 266], P<.001), but we did not detect differences in survey responses between low response programs and high response programs (Table 2). A large majority (87.7%, 407 of 464) responded that applicants should be offered an in-person visit after program rank list finalization if interviews are virtual again next year; of the 20 “No” respondents, 10 answered the “Why not?” follow up prompt, the majority of whom expressed concern that an applicant’s decision to visit or not visit might affect their rank position. Three quarters of respondents (347 of 464) felt that having the optional in-person visit after program rank list finalization was equitable and reflected favorably on the program. Only 258 of 464 respondents (55.6%) indicated that they trusted programs not to change their rank list based on applicant decisions to visit or not visit, despite repeated assurances from participating PDs and administrators, and reported adherence to their promise to applicants. General narrative comments about the option for in-person visits were positive from applicants, with applicants’ main concern being that programs could have incorporated information about whether an applicant chose to visit in their rank list using the studied process (ie, without a formal NRMP program rank list deadline before the visit; online supplementary data).

Three hundred and fifty-one of 464 respondents (75.6%) did not attend an in-person visit; the most common reasons for not visiting were: unable to fit a visit into their schedule (55.8%, 196 of 351), they had gotten the information they needed from the virtual process (46.2%, 162 of 351), and unable to afford the travel costs (33.0%, 116 of 351). One hundred and thirteen of 464 (24.4%) respondents attended at least one in-person visit. Of these, nearly all (96.5%, 109 of 113) felt that the in-person visit was a valuable tool to help them make a more informed decision about their rank list after the virtual interview. Most respondents (60.7%, 68 of 113) drove for the in-person visit, most traveled more than 200 miles, and the average cost to applicants for an in-person visit was $302 (Table 2). Most applicants who visited felt that they learned more about the program and the city from visiting, and 76.8% (86 of 113) reported that the in-person visit influenced their rank list.

All 12 participating PDs and program administrators responded that they liked having the in-person visits, and 9 (75%) found the process equitable for applicants (the other 3 answered “unsure”; Table 3). Most PDs characterized the time and stress burdens of the in-person visits as a “small amount,” and that it was only “slightly difficult” to finalize their rank lists early enough to accommodate the in-person visits. All PDs think there should be an optional in-person visit in future recruitment cycles, though some PDs would modify their in-person visit approach (Table 4). PD narrative feedback included that a few candidates rated by virtual interviews to have poor communication skills appeared to have much better communication skills during in-person visits. Program administrators estimated that the in-person visits added 7.8 hours to their workload and 21 hours of total staff time. Half of program administrators reported the stress added as a “small amount,” while 2 characterized it as a “moderate amount” added, and 1 “a lot.” Five of 6 program administrators think there should be an in-person visit option next year, while 1 was unsure. There were no incidents of applicant unprofessional behavior observed by PDs or program administrators during in-person visits.

This survey study of IM residency programs post-virtual interview optional in-person visits after programs locked their rank lists, found high satisfaction reported by applicants, PDs, and program administrators. Most PDs and program administrators reported relatively low costs and increased workload. Applicants perceived that the process was equitable but also expressed concerns regarding whether programs would honor the commitment to locking their rank lists. Nearly all applicants (96.5%) who visited a program found the visit valuable to them. A third of applicants reported that they did not choose to visit due to costs. Most survey respondents expressed interest in continuing the option of post-virtual interview, in-person visits.

Our findings are similar to those of a previous study reporting that a cohort of radiology residency applicants in the 2021-2022 season found visiting a program after virtually interviewing highly valuable.²⁹  Notably, our study design differed by including multiple IM residency programs, also surveying interviewed applicants who chose not to visit, and requiring participating programs to finalize their rank lists prior to conducting in-person visits.

Although this study did not assess the quality of residency virtual interviews, the finding that PDs observed strong communication skills in a few applicants whose communication skills were considered poor during virtual interviews requires further study. For example, are virtual interviews less effective in assessing communication skills among applicants than in-person interviews, or might similar observations have been seen with in-person interviews compared to informal in-person activities?

If future recruitment seasons allow applicants the opportunity for in-person visits after virtual interviews, there could be additional disruption to students’ rotations and school commitments. Scheduling in-person visit days on a Friday or Monday may reduce the travel time disruption to students’ regular schedules. If programs allowed in-person visits on the same days (eg, the last 2 or 3 Fridays in February), there would be a natural limit to how many in-person visits an applicant could attend, which might create a reasonable compromise balancing the competing priorities. Some concern has been raised among PDs that in-person visits might uncover unprofessional behavior in an applicant that would prompt a PD to want to change their rank list, though this did not occur in our study. If the NRMP adopts a staggered rank list deadline, a PD would not be able to change their rank list despite the identification of concerns; if such an applicant did match with the program, the PD would have the benefit of foresight into the potential risk for professionalism concerns that they otherwise would not have had.

This study is limited by the low applicant response rate to the survey and the small number of IM programs in the study; the respondents and included programs may not represent the total populations of interest. Also, the study included no residency programs in the Western United States, where travel may have greater cost and time implications for students. The survey was developed by individuals with survey expertise, but was not tested in the population of interest; thus, respondents may not have interpreted questions as intended.

With the students’ concerns in this study regarding programs honoring the agreement to lock their rank lists before in-person visits, a next step would be to evaluate this option with NRMP support, to set an earlier deadline to lock participating programs’ ranks lists. In addition, more work examining the effects of this approach on final year medical student learning, participation by less well-resourced students, and greenhouse gas emissions is needed.

This multicenter study of post-virtual interview, optional in-person visits to IM residency programs that agreed to lock their final rank lists prior to offering visits, found high acceptability from applicants, PDs, and program administrators. Costs and workload were considered reasonable, and applicants perceived that the process was equitable. However, many applicants were concerned that programs would not honor their commitment to lock the final rank list.

The authors would like to thank the applicants who completed the survey and their program staff and residents who make successful residency recruitment possible.

The online version of this article contains the surveys used in the study and representative narrative responses of applicants.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US Government.