Evaluation of Residency Applicant Preferences for Continuing Virtual Interviews: A National Database Analysis

The COVID-19 pandemic necessitated a shift from in-person to virtual residency interviews during the 2020-2021 application cycle. As with telehealth and many pandemic-inspired innovations in medical education,¹  virtual interviews served as a stopgap measure, and the initial broad adoption has had only partial persistence.^2,3  Decisions regarding interview format are critical, as virtual interviews have the potential to decrease costs, reduce carbon emissions, and alter bias in the selection of medical residents.^2,4  Virtual interviews alleviate financial barriers but differ from in-person with less opportunity to evaluate program culture, connect with residents and faculty, and understand the geographic region.⁵ 

Although many programs and institutions have selected various interview formats since the end of the Federal COVID-19 Public Health Emergency, there is limited evidence available to guide these decisions, and even an approach that is perceived as working well for programs may not work well for applicants. Despite this, most of the studies on applicant preferences have been specialty-specific, single year, and limited by relatively small sample sizes.^6-9  Furthermore, the Association of American Medical Colleges recommended continuation of virtual interviews for the 2024-2025 application cycle,¹⁰  citing “financial costs associated with interviewing” as their first key consideration.²  Costs have potential equity implications, but few data exist on sociodemographic factors related to interview preference.

We sought to address the gap in the literature by analyzing applicant preferences on continuation of the virtual interview format across specialties from 2021 to 2023, using the Texas Seeking Transparency in Application to Residency (STAR) database. Our study will add evidence to help guide the decision-making process by providing a large specialty-wide evaluation of applicant preferences regarding virtual interviews that spans 3 consecutive application cycles. Findings related to sociodemographic differences in interview preferences could help guide decision-making from an equity lens and prompt reconsideration of virtual or hybrid options.

Based on the limited evidence available in current literature and a consideration of potential barriers to attending in-person interviews, we hypothesized that applicant characteristics such as underrepresented in medicine (UIM) status, history of food or housing insecurity, first generation to attend college status, and female sex would correlate with a preference for continuing virtual interviews.

Detailed information regarding the development of the Texas STAR survey and variable selection for this study is provided as online supplementary data. The Texas STAR Survey is a voluntary survey distributed in April of each year to graduating allopathic and osteopathic medical students (MD and DO seniors) at participating medical schools in the United States. The survey was distributed via email and was completed online. Applicants had one month to complete the survey, and all responses were anonymous. Medical schools were encouraged to send out weekly reminders via email about survey participation. There were no incentives for survey participation. Applicants were instructed to report data as it appeared on their residency applications. Participation in the survey served as consent for research purposes. The specific medical schools participating for each study year are listed on the Texas STAR website.¹¹  Overall, 23.4% of medical schools were located in the US Central region, 22.7% in the Northeast, 39.6% in the South, and 14.3% in the West (online supplementary data). Out of the 154 participating medical schools across study years, 15 (9.7%) were osteopathic.

Applicants from Match years 2021, 2022, and 2023 were included. We first tabulated responses to the survey question “I would be in favor of virtual interviews going forward,” and then we dichotomized the variable into applicants favoring (response of strongly agree or agree) versus not favoring or undecided (response of undecided, disagree, or strongly disagree). We used chi-square testing to assess differences in applicant characteristics between the 2 groups. Comparisons were limited to the covariates selected a priori for this study. Bonferroni correction was applied to account for the possibility of type I error with multiple comparisons.

We used univariate and multivariable logistic regression models to assess associations of applicant characteristics with the odds of favoring continuation of virtual interviews. The multivariable model adjusted for all significant variables in Table 1. We assessed for multicollinearity using variance inflation factor (VIF) testing, and all variables met inclusion criteria (VIF<5). Reference categories for the regression models were chosen based on the approach outlined in Johfre and Freese.¹² 

We used generalized linear models to estimate the marginal means and standard errors for the continuous variables (applications submitted, interview offers, and total spending) in applicants who did versus did not favor continuation of virtual interviews. The distributions for these variables were right-skewed, so we ran Poisson regression models for count outcomes (applications submitted and interview offers) and log-normal models for the cost outcome (total spending).

Sociodemographic variables were only available for Match years 2022 and 2023 (n=13 668). Further investigation into this subset revealed that missing observations exceeded 5% for these variables, so we performed a sensitivity analysis to assess if respondents with missing sociodemographic data were more likely to favor continuation of virtual interviews (online supplementary data). All other variables were examined and had <5% missing observations.

Statistical significance was initially set at P<.05 for all analyses, and it was adjusted to a threshold of P<.005 based on Bonferroni correction for multiple comparisons. Stata version 18 (StataCorp LP) was used for all analyses and GraphPad Prism version 9.1.0 (GraphPad Software) was used to create the figure.

This study was deemed not human subjects research by the University of Michigan Institutional Review Board (HUM00217169).

There were 20 547 respondents who were included in this study (6879 in 2021, 6918 in 2022, and 6750 in 2023). The response rate to the Texas STAR survey was 40% in 2021, 38% in 2022, and 34% in 2023. When compared to data from the National Resident Matching Program (NRMP),¹³  our sample represents 25.3% of US seniors who applied and submitted a rank order list from 2021 to 2023, including 31.9% of US MD seniors, 7.0% of US DO seniors, and 21.1% of couples in the couples Match.¹³ 

Among the sample of all respondents from 2021 to 2023, 6054 applicants (29.5%) strongly agreed, 8073 (39.3%) agreed, 2187 (10.6%) disagreed, 866 (4.2%) strongly disagreed, and 3367 (16.4%) were undecided about whether virtual interviews should continue in future application cycles. Overall, 68.8% of applicants were in favor of (agreed or strongly agreed) virtual interviews continuing. The percentage of applicants in favor of virtual interviews continuing significantly increased each year (57.5% in 2021, 70.2% in 2022, and 78.8% in 2023; OR 1.74, 95% CI 1.62-1.87 for 2021-2022 and OR 2.75, 95% CI 2.55-2.97 for 2022-2023 vs 2020-2021; P<.001). There were also significant differences by region, with 72.2% (1786 of 2472) of applicants from the US West favoring continuation of virtual interviews compared to 69.8% (3356 of 4811) from Central, 69.6% (3690 of 5299) from Northeast, and 66.5% (5295 of 7962) from the South (OR 1.16, 95% CI 1.08-1.25 for Central; OR 1.16, 95% CI 1.07-1.24 for Northeast; OR 1.31, 95% 1.18-1.44 for West vs South; P<.001; Table 1). Applicants favoring continuation of virtual interviews submitted fewer applications (41.6 vs 43.9, P<.001) but had more interview offers (14.8 vs 13.5, P<.001; Table 2). There was no association between total spending and preference for continuation of virtual interviews (P=.32).

There was a broad range of preferences for continuation of virtual interviews across specialties, ranging from only 44.3% (357 of 806) of orthopedic surgery applicants favoring continuation of virtual interviews to 79.2% (114 of 144) of child neurology and radiation oncology applicants favoring continuation of virtual interviews (Figure). Overall, applicants to nonsurgical specialties were significantly more likely to favor continuation of virtual interviews compared to surgical specialties (71.0% vs 62.1%; P<.001).

Self-reported sociodemographic data were available for the 2022 and 2023 Match years (n=13 668). Applicants identifying as female (OR 1.45; 95% CI 1.33-1.58; P<.001), UIM (OR 1.82; 95% CI 1.59-2.08; P<.001), first generation to attend college (OR 1.28; 95% CI 1.13-1.46; P<.001), and reporting a history of food or housing insecurity (OR 1.32; 95% CI 1.13-1.56; P<.001) were more likely to favor continuation of virtual interviews (Table 1). Sensitivity analysis revealed that applicants who did not report optional sociodemographic data were less likely to favor continuation of virtual interviews (online supplementary data).

In the fully adjusted model, year 2023 (OR 1.61; 95% CI 1.47-1.76; P<.001 vs year 2022), female sex (OR 1.43; 95% CI 1.30-1.56; P<.001), UIM status (OR 1.75; 95% CI 1.51-2.01; P<.001), geographic region (Central: OR 1.19; 95% CI 1.06-1.34; P=.003 and Northeast: OR 1.20; 95% CI 1.07-1.34; P=.002; vs South) were significantly associated with favoring continuation of virtual interviews (Table 3). Applicants to a surgical specialty (OR 0.58; 95% CI 0.53-0.64; P<.001) were significantly less likely to agree with the continuation of virtual interviews (Table 3).

In this national cohort of 20 547 applicants, individuals who identified as female or UIM were more likely to favor continuation of virtual interviews, which supports the notion that virtual interviews play a role in promoting equitable access. Respondents from a medical school in the Central and Northeast regions (vs South) and applying to nonsurgical specialties were also more likely to favor continuation of virtual interviews, as were applicants with more interview offers. We also found that overall, most applicants (68.8%, 14127 of 20547) favor continuation of virtual interviews, a preference that has sequentially increased during the past 3 application cycles.

Prior studies investigating applicant perspectives on virtual interviews have had narrower scope than the present study. Most have been limited to individual specialties or institutions, have not had sufficient number for detailed sociodemographic analyses, and have had inconsistent findings.^6-9,14-19  Nevertheless, most studies have found high satisfaction among applicants regarding virtual interviews,⁵  which corroborates our finding that 68.8% of applicants favor continuation of virtual interviews. A survey-based study of first-year residents who had participated in virtual interviews found that the majority of applicants felt that the virtual setting did not impact how they created rank lists, and it accurately portrayed their residency experience.²⁰ 

Applicants in favor of virtual interviews often cite advantages of cost-savings and reduced travel burden.^14,15  Prior to the COVID-19 pandemic, applicants reported spending up to $5,000 on interview-related travel.^4,21,22  The cost burden of in-person interviews can hinder applicants from financially disadvantaged backgrounds,²  who disproportionately identify as Black or Hispanic.²³  This may underly our finding that applicants who identified as UIM were more likely to favor continuation of virtual interviews. Studies have found that virtual interviews can increase applicants’ exposure to geographically diverse programs that they otherwise may not have had access to due to financial or time constraints.^24,25  More studies are needed to evaluate how virtual interviews may improve access to geographically diverse programs among specific groups of applicants.

The association of female sex with a preference for continuation of virtual interviews is challenging to interpret without qualitative data, but it is supported by other studies in literature. Shebrain et al performed a study in which general surgery applicants to a single program were allowed to choose between a virtual or in-person interview format during the 2022-2023 application cycle.²⁶  They found that interviewees who chose virtual interviews were more likely to be female than those who chose in-person interviews (63% vs 32%, P=.004).²⁶  A survey-based study by Domingo et al of 565 residency interviewees at a single institution in 2021 found that female applicants were more likely than male applicants to report that virtual interviews were “less burdensome to their schedule” (89% vs 76%, P<.001), and female applicants were less likely to agree that they would “need an in-person visit to the city before selecting a program” (P=.006).¹⁴  Further research is needed to better understand these sex-based differences.

One limitation of this study relates to the lack of qualitative data on applicant preferences. This makes it difficult to interpret certain findings that met statistical significance, such as the association between female sex and preference for continuation of virtual interviews. Nevertheless, the validity of this finding is supported by other studies in literature,^14,26  emphasizing the need for more research to better understand its drivers. There is risk of selection bias based on who received the Texas STAR survey, since participating medical schools were disproportionately located in the South (39.6%). In the context of our finding that applicants in the South were less likely to favor continuation of virtual interviews, this selection bias may have resulted in an underestimation of the overall percentage of applicants favoring continuation of virtual interviews. We are also unable to determine nonresponse bias since we did not have data on nonrespondents. The validity of our study is challenged by the potential that applicants who answered the survey differed systematically from those who did not.

Approximately 20% of applicants did not report sociodemographic data, and these applicants were less likely to favor continuation of virtual interviews based on sensitivity analysis. Although the magnitude of effect size is small (no more than 3.1% difference across any of the sociodemographic variables in the sensitivity analysis), our findings related to sex, race, first generation college status, and food or housing insecurity should be interpreted with caution. When accounting for the missing observations and availability of sociodemographic data only for Match years 2022 and 2023, the sample size for the multivariable regression models and sociodemographic variables was limited to 10 959 applicants, or 53.3% of the total sample.

In addition, the Texas STAR survey was not distributed to international medical graduates (IMGs), MD graduates, or DO graduates who are all important stakeholders in conversations surrounding virtual interviews. Based on NRMP data during our study years, this includes 39 582 IMGs, 5029 MD graduates, and 1991 DO graduates.¹³  Given that applicants from a medical school outside of the United States have been found to favor virtual interviews compared to applicants within the United States,²²  our study may underestimate the preference for virtual interviews among all residency applicants. Despite this limitation, applicant-reported data in the Texas STAR database have been found to closely resemble the data reported by NRMP in terms of research output, United States Medical Licensing Examination scores, leadership positions, and volunteer positions,^27,28  which strengthens our confidence that overall our findings reflect the population of interest.

Similarly, our study affords no insight into the perspectives of program directors, faculty, or other involved parties. There is also risk of recall bias. Applicants completed the survey after the Match, and therefore several months had elapsed since the start of interviews. If applicants were satisfied with their match overall, as is often the case even in the setting of virtual interviews,²⁰  they may have had a more favorable perspective of virtual interviews in hindsight; this potential effect would be expected to lead to a slight overestimation in preferences for continuation of virtual interviews. Finally, reliability of the survey items was not tested during the development process, which is an inherent weakness of our instrument.

Most applicants favor continuation of virtual residency interviews, and this trend has been increasing over the past 3 application cycles. Certain sociodemographic groups were more likely to favor continuation of virtual interviews.

The online version of this article contains further data from the study and a visual abstract.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.