Virtual Residency Interviews: Applicant Perceptions Regarding Virtual Interview Effectiveness, Advantages, and Barriers Open Access

Due to the COVID-19 pandemic, US graduate medical education (GME) programs conducted virtual interviews in 2020–2021. Applicant perceptions regarding the virtual interview format are needed to inform program leaders as they plan for subsequent interview cycles.

Most studies of GME program virtual interviews were published prior to the technological advances and widespread use of virtual conference platforms that occurred during the pandemic.^1-11  These studies were limited to single program assessments, and none to date included an assessment of potential impacts on diversity, equity, or inclusion.¹² 

In this study, we assessed applicant perceptions of virtual interviews in a multispecialty approach at one institution during the 2020–2021 virtual residency interview cycle. To begin to explore the impacts of virtual interviewing on equity and inclusion, we assessed key advantages, barriers, and perceived differences in experience based on gender, race, and location.

This observational study surveyed interviewees in family medicine, obstetrics and gynecology, psychiatry, emergency medicine, pediatrics, general surgery, and anesthesiology residency programs, as well as in clinical psychology internship and PhD programs, following their virtual interviews at one urban academic medical center in the Pacific Northwest in 2020–2021.

The survey was created by a team of clinician educators, including faculty, program directors, a statistician, residents, and medical students (Qualtrics.com; provided as online supplementary data). The instrument assesses effectiveness, advantages, barriers, and future preferences regarding virtual interviews. We included items that may amplify or alleviate structural inequalities in the setting of virtual interviews, such as pregnancy, breastfeeding, dependent care, disability, and mental health conditions. We also included self-reported urban/rural/suburban interview location, because rurally based applicants may not always have reliable access to internet but may particularly benefit from reduced travel in the setting of virtual interviews. The author group relied on the literature and expert discussion to select items that would optimize content validity. To increase response process validity, we piloted the survey with 10 people, including faculty, program directors, residents, and medical students, and revised for clarity, brevity, and readability.

Descriptive statistics of responses to items related to effectiveness, advantages and barriers, and future interview format preferences were analyzed. Subgroup analyses were performed on each item above by gender, race, underrepresented in medicine (UiM; excludes White or Asian) and urban/suburban/rural status. A narrow age distribution precluded analysis by age. Participants reporting mixed race with at least one UiM selection were included in our aggregated UiM category. Participant responses were excluded from subgroup analyses when requisite demographic data were not provided or a demographic category contained fewer than 5 responses.

All statistical tests were performed using SAS 9.4 software (SAS Institute Inc, Cary, NC). We conducted Wilcoxon–Mann–Whitney and Kruskal–Wallis tests for ordinal variables. Chi-square and Fisher's exact tests were utilized for analysis of nominal variables. To account for multiple comparisons in our subgroup analyses, we calculated Benjamini–Hochberg critical values using a false discovery rate of 0.05 and compared each test's P value with its corresponding Benjamini–Hochberg critical value to determine significance.

This study was exempted by the Oregon Health & Science University Institutional Review Board.

A total of 565 out of 1429 interviewees completed the survey, with a response rate of 40%. Demographics are reported in the Table. Our sample of residency applicants was representative of the Electronic Residency Application Service (ERAS) 2019–2020 applicant population in terms of Asian race and Latinx race.¹³  More respondents were female (63.9% vs 54.4%) and White (55.6 vs 42.4%) and fewer were Black (6.3% vs 9.4%) or categorized as UiM (17.9% vs 19.2%) than the ERAS population.

A large majority of respondents agreed or strongly agreed that virtual interviews were effective in establishing comfort (469 of 565, 83%) and connection (500 of 565, 89%), demonstrating applicant strengths (488 of 564, 87%), and answering their questions (539 of 564, 96%). A smaller majority of interviewees felt virtual interviews were effective for clinical tours (337 of 564, 60%), learning about the program culture (352 of 565, 62%), institutional social environment (384 of 565, 68%), and city (345 of 564, 61%). Forty-seven percent (265 of 563) of interviewees agreed or strongly agreed they would need to still visit the city before selecting a program. Additionally, when comparing median Likert scores, fewer Black (2.0 [95% CI 2-3] vs 3.0 [95% CI 1-5] to 4.0 [95% CI 2-5] for other races; P=.0009), female (3.0 [95% CI 3-4] vs 4.0 male [95% CI 3-4]; P=.0055), urban, and rural (3.0 [95% CI 3-4] and 3.0 [95% CI 2-3] vs 4.0 [95% CI 3-4] suburban; P=.0025) interviewees agreed that they would need to visit compared with their subgroup counterparts (online supplementary data).

Survey participants reported more advantages than barriers of virtual interviews (Figure). The most common perceived advantages were cost savings (539 of 555, 97%), time efficiency (505 of 555, 91%), less burdensome scheduling (469 of 555, 85%), and reduced carbon footprint (397 of 555, 72%). The most frequently reported barriers were time zone differences (233 of 555, 42%), access to an appropriate interview setting (148 of 555, 27%), and reliable access to internet (129 of 555, 23%).

Women reported more often than men that virtual interviews were less burdensome to their schedule (89.2% [95% CI 86.0-92.4] vs 75.7% [95% CI 69.4-81.9], P≤.0001). No other significant subgroup differences were found.

Eighty-four percent of interviewees (456 of 542) desired some incorporation of the virtual format into future interview cycles, and there were no significant differences in subgroup analysis (online supplementary data).

Most interviewees perceived virtual interviews to be effective and preferable as a component of future interview seasons. Applicants perceived virtual interviews to be less effective for communicating program culture and providing adequate socio-environmental information. We did not identify negative impacts on equity in our subgroup analyses of gender, race, UiM status, or rural/suburban/urban location.

The results of our multispecialty study support prior findings demonstrating the effectiveness and acceptability of virtual interviews.^1-12  Others have reported similar advantages of cost savings, time efficiency, and reduced scheduling burden. Carbon footprint reduction was among the most frequently selected advantages in our study, reflecting a similar portion of applicants concerned about the environmental impact of travel as reported by Fung et al (74.4%).¹⁴  Technological issues and suboptimal tours/social information are often cited as main barriers in other studies. Our study echoes these issues and adds time zone differences as a major barrier. To our knowledge, our study is the first to assess gaps in equity of virtual interviews by gender, UiM status, race, or rural/urban/suburban location.^12,15 

There were several limitations to our study. Validity may be affected by lack of a traditional interview comparator group and limited validity evidence for the survey. Generalizability is limited by the single collection site, but we did collect a relatively large multispecialty national sample of applicants. Despite a relatively low response rate (40%), our population was representative of the greater ERAS population in many ways. However, our overrepresentation of those identifying as female and White and underrepresentation of those identifying as Black may have limited our study's ability to identify equity gaps, particularly for Black applicants. We did not include objective criteria for urban/rural/suburban which may have introduced heterogeneity in these groups.

Future studies that explore factors affecting diversity, equity, and inclusion, such as the experience of applicants with different abilities or who are from international medical schools would be helpful with purposive or representative samples of the general and underrepresented applicant populations. As US medical students cite geographic location and “goodness of fit” as leading factors in program selection, ongoing attention should be paid to improving methods for conveying the institutional and regional environment.¹⁶ 

Virtual interviews were perceived as effective, more advantageous than burdensome, and widely acceptable to our respondents, with no significant differences by respondent race, gender, or location.

The authors would like to thank Elana Zuber, MBA, David Buckley, MD, Lalena Yarris, MD, Karen Brasel, MD, Joshua Rosen, MD, Deborah Cohen, PhD, and Joe Skariah, DO, MPH.