Videoconference Interviews: A Timely Primary Care Residency Selection Approach

The process of residency application and candidate selection is a time-intensive and costly endeavor for medical student applicants and the training programs to which they apply. These factors of time and cost have, in fact, been cited by students as being decisive limiting factors in up to 70% of their interview acceptances to residency programs.¹  Additionally, restrictions related to the current COVID-19 pandemic have led to sizeable changes in the way programs will likely approach the 2020–2021 residency match cycle.^2,3 

Videoconference interviews (VCIs) are increasingly being used in the selection process of program candidates across a range of medical specialties, both in applicant screening and, in some cases, as a replacement for in-person interviews. Various studies have assessed the feasibility and cost-efficacy of using VCIs at the residency and fellowship levels of graduate medical education, exploring benefits from the perspectives of residency programs and medical student applicants. Although several fields have begun using VCIs in their selection process, most are in the preliminary stages,⁴  with the majority in surgical and related fields, namely anesthesia,⁵  orthopedics,⁶  ophthalmology,⁷  and plastic surgery.^8–10  This may be owing to the highly competitive nature of these fields, and the consequent need to more efficiently narrow a large candidate pool.

We are using “VCI” in the synchronous sense, with applicant and interviewer interacting simultaneously via a video-based communication platform. Asynchronous VCI approaches, whereby applicants respond via video-recording to a series of predetermined interview questions, have also been explored by obstetrics and gynecology programs,¹¹  and both approaches have been trialed by various emergency medicine programs,^12–16  including through the Standardized Video Interview project from the Association of American Medical Colleges (AAMC), which has yielded mixed outcomes.^17–19 

Current evidence supports web-based synchronous VCI as a reliable and effective adjunct to in-person interviews to contain costs and enhance convenience for programs and applicants alike.^20–22  The AAMC has published several informational guides to support best practice implementation of VCIs at the undergraduate medical institution level.²³  This approach nevertheless remains underutilized in the Match process for residency and fellowship programs nationwide,^24,25  particularly those in primary care, for which we were unable to find relevant published data.

This retrospective data review with cost analysis explores financial and acceptability outcomes of VCI implementation and use over a 9-year period, in order to share experience at a time when many programs are seeking ways to rapidly adopt VCI strategies due to the current COVID-19 pandemic.

The Phelps Family Medicine Residency Program in Sleepy Hollow, New York, a 3-year program matching 8 residents per year, was formed and accredited in 2011. An unanticipated influx of applications during the program's first interview season inspired the faculty to develop a system of web-based virtual interviews to identify ideal candidates for the Match. The program is currently using VCIs to interview all applicants, via the Skype platform, that meet certain qualifying thresholds, prior to granting in-person interviews. Skype is one of several software applications currently available that facilitate virtual meetings. The materials needed by each party in order to effectively communicate include a microphone, speakers, camera, and reliable wi-fi or data connection. Typically, these equipment needs can be met with desktop or laptop computers, tablets, or cell phones.

When initiated during the program's first recruitment season, faculty anticipated that the incorporation of VCIs into the recruitment process would afford increased access to candidates and, as opposed to traditional in-person interviewing alone, would prove cost effective for both program and candidate, while creating a more personal experience early in the recruitment process. Exceptional candidates were identified via VCIs by a standardized survey tool to semi-objectively quantify candidate responses to structured interview questions; they were subsequently invited for an in-person interview to determine final ranking decisions.

After going live, applications received through the Electronic Residency Application Service (ERAS) are filtered and divided among faculty members alphabetically by candidate surname for preliminary review. Based on these reviews, qualified candidates are offered a VCI with a core faculty member. Each faculty member can select an average of up to 20 people for a VCI. Additionally, to ensure the process allows for sufficient vetting of candidates, faculty members are generally not assigned virtual interviews with those whose application they have already reviewed. For example, faculty “A” will not complete the VCI for any applicant with last name A to F (table 1).

Each applicant who accepts a VCI receives an email with their appointment time and information relevant to the virtual interview, including instructions on using the software and general advice for digital interviews. The VCI lasts approximately 15 to 20 minutes. Structured forms developed internally are used during virtual interviews to assess candidates' interest in and knowledge of our program, as well as to assess specific Accreditation Council for Graduate Medical Education Milestones in patient care, medical knowledge, and practice-based learning and improvement (forms provided as online supplemental material). Validity testing was not performed for this tool. Promising VCI candidates meeting a threshold score on the assessment tool are then offered in-person interviews.

As an innovative approach, the process of VCI implementation underwent some degree of evolution during its first few years. In the initial interview season (2011–2012), VCIs were scheduled with 2 faculty members concurrently and were only offered on a trial basis to a small sample of the total applicant pool. During the subsequent interview season (2012–2013), VCIs were scheduled with just one faculty member, and the VCI process was further refined and incorporated into the screening process after application review, prior to granting an in-person interview. After the initial review of applications in ERAS, the applicant was chosen for either an automatic in-person interview or a preliminary VCI. If allocated to the latter, each applicant was assigned to 1 of 3 possible categories after the VCI: (1) invite for an in-person interview; (2) hold for discussion by the recruitment committee; or (3) do not rank. In every subsequent interview season since, prior to granting in-person interviews, VCIs have been utilized to screen all applicants selected through ERAS filters who meet the minimum eligibility criteria set by the program's recruitment committee; this was implemented after repeatedly interviewing apparently well-qualified applicants who would nevertheless have been screened out using VCIs. Furthermore, the structured VCI form has been revised to include standard behavioral and clinical knowledge questions for all applicants, with Likert-scale scoring of answers to determine their assignment in 1 of the 3 decision categories previously listed.

Institutional Review Board (IRB) review was sought through the Human Subjects Research Determination process, which stated that the proposed study activities did not constitute human subjects research, and therefore IRB review was not required.

From the standpoint of recruitment committee efficiency, the VCI process has proven a useful means of screening applicants prior to in-person interviewing. The program's first 4 recruitment seasons saw a decrease in the number of in-person interview invitations extended without a preliminary VCI; since 2015, no applicants have been interviewed in-person without first undergoing a VCI (figure). Every year a notable proportion of applicants (median 48%) are eliminated from consideration based on the VCI alone without compromising the quality of program-matched candidates, as assessed by average scores on licensing and board examinations (table 2).

The candidate review process outlined here has implications from a cost-effectiveness perspective. In the 2019–2020 interview season, to use the most recent available data, a typical interview day for 8 applicants cost an estimated $1,669 (table 3a). Comparatively, the cost of offering VCIs to 8 candidates was $384 (0.5 hr × $96/hr × 8 applicants; table 3b). Administrative costs related to scheduling and coordinating VCIs were nominal, owing to the “Scheduler” function made available by ERAS in recent years.

To accommodate an additional 48% candidate volume for in-person interviews in the 2019–2020 season (n = 195), if not eliminated through VCIs conducted in advance, the program would have required an additional 15 interview days for 94 additional applicants, at an estimated total cost of $25,035. The total cost of offering VCIs during that same season was $9,360 (0.5 hr × $96/hr × 195), for a cost savings of $15,675, or a 63% reduction in allocated program costs. Even when considered more modestly over the 9-year study period, a median 48% more applicants added to the mean annual volume (n = 157) of total interviews conducted equates to an additional 10 interview days for 76 additional applicants, at an estimated total annual cost of $16,690. The cost of offering VCIs to that number of applicants would be $7,536 (0.5 hour × $96/hr × 157), for a total cost savings of $9,154, or a 55% reduction in allocated program costs. This allows the program to interview a wider range of candidates, while simultaneously enabling multiple faculty members to more comprehensively review and establish contact with a growing applicant pool, and to do so across various interview settings (ie, in the applicant's home, onsite at the residency, etc). Of note, through the use of internally developed checklists and application review tools, the incremental increases in time required for subsequent faculty application reviewing is minimal.

With regard to acceptability by faculty and candidates, informal data suggest a mixed response. Faculty and candidates were asked for feedback via an annual post-interview survey during the first 8 years of VCI implementation. One faculty member commented that it was “a novel idea, which enabled us to get to know the applicants prior to actually meeting them on interview day,” and faculty members have consistently elected to keep the VCI format year after year as a successful and convenient recruitment method.

Applicant comments on the post-match survey varied in enthusiasm for the format. Positive statements included those such as “[The VCI] was innovative, exciting, an overall great experience”; “[the] Skype interview gave me a chance to meet the faculty and sort of be nervous on my own before actually getting [to meet them] in person”; and “It was a nice opportunity to learn more about the program and have some questions answered. Being interviewed at home was comfortable and convenient.” However, while only 2 comments in the survey's open-ended response portion addressed the VCI interview in the 8-year period, both reflected a negative perception by the respondents, who felt that the element should be eliminated.

The survey specifically queried applicants on the VCI interview during 3 successive interview seasons (2014–2017) with the question, “What did you think about the Skype interview?” Of a total 39 of 220 survey respondents (18% response rate), 27 (69%) rated it as “great,” “good,” or “fine” and 14 (36%) found it “interesting.” Only 5 respondents (13%) felt it had saved them time or expenses, and 4 (10%) felt it was “a waste of time.”

To our knowledge, this VCI-based interviewing approach is being done elsewhere in few primary care residency programs, and certainly not at this scale within their programs or for this extended period of time. Based on experience gleaned from both VCI and in-person interviewing, through a process now successfully replicated over 9 annual recruiting cycles, we are convinced that these methods complement one another. Utilizing both types of interview creates an efficient process that supports the assessment of large pools of highly qualified candidates. It is also an opportunity for individual programs to stand out as unique amidst the many training options nationwide, serving as a first touchpoint with candidates. This can aid in determining match probability early on, especially as applicants apply to greater numbers of programs. This aspect will prove increasingly salient as primary care fields such as family medicine are projected to become more competitive based on a review of recent trends.²⁶ 

It is important to state that the degree to which a program will see cost benefit with this model is likely to vary widely depending on program characteristics. Primary care residency programs typically interview 15 to 17 candidates and rank 8 to 10 candidates per available position.²⁷  Therefore, while an 8-8-8 program would generally rank approximately 80 candidates, various features of the program (new vs established reputation, rural vs urban setting) can impact how wide the net must be cast to optimize match outcomes. In the context of seismic changes being wrought by the COVID-19 pandemic on the 2020–2021 residency recruitment cycle, even more established programs are likely to process higher-than-normal applicant volumes this year, thereby benefitting from a more rigorous prescreening approach. Regardless of a program's chosen vetting process, virtual interviewing may well become the norm this year and beyond.

The study described had several limitations. Use of examination scores was our main measure of method efficacy, through demonstrating consistent quality among program-matched candidates year to year, although this has admittedly not been shown to be an accurate predictor of physician success in patient care. The cost estimate provided does not account for benefits of potential income generated through increased faculty and resident clinical productivity (owing to fewer in-person interview days). It furthermore does not quantify the benefit of flexible scheduling afforded by VCIs, which obviates the need for complex cross-coverage and schedule coordination for faculty, residents, and administrative personnel to allow in-person interviewing days. While consistency was ensured in large part year to year in the use of this web-based virtual interviewing and candidate selection process, there was some variation as the process was refined with subsequent iterations, including the vetting and implementation of interview tools to better standardize the VCI. Various changes in the program's staffing, curricular elements, and institutional partnerships also contributed numerous confounders to match outcomes from year to year. Additionally, while it has implications for generalizability to other residency programs within primary care and graduate medical education more broadly, the collated data nevertheless reflect the experience of only one program. Finally, our data on the acceptability to candidates are limited, as the post-match surveys were not designed to be used in a rigorous post-hoc analysis, and were therefore of limited utility in this retrospective review (ie, the survey questions were not standardized across all 8 years and had poor overall response rates).

In future recruiting seasons, particularly during the anticipated transition to an exclusive VCI approach in the 2020–2021 recruiting cycle (to accommodate COVID-19-related social distancing restrictions), we are considering a trial of reduced in-person interviews. We are also exploring the possibility of creating virtual “hangout groups” to enhance candidate-resident interactions and virtual tours of our facilities and practice locations to supplement the VCI experience.

Interviewing a pool of qualified candidates using VCIs is an effective way for primary care residency programs to interact with applicants. It optimizes the use of program and applicant resources for in-person interviews. It allows many candidates to interact with faculty and learn about the program without having to travel, and it is an innovative tool commensurate with current evolving norms for digital communication in a technologically oriented society. The implementation of VCIs enhances face-to-face contact at reduced cost, providing efficient applicant assessment early in the process, and enables utilization of technology for timely contact between programs and applicants, enhancing the decision-making process for both parties.