ABSTRACT

Background

Videoconference interviews (VCIs) are increasingly being used in the selection process of residency program candidates across a number of medical specialties, but nevertheless remain an underutilized approach, particularly in the field of primary care.

Objective

This retrospective data review with cost analysis explores financial and acceptability outcomes of VCI implementation over a 9-year period.

Methods

VCIs were incorporated into the recruitment process at a community-based academic family medicine residency program in 2011, whereby suitable candidates were selected for VCIs after Electronic Residency Application Service (ERAS) application review. Based on the outcome of VCI, candidates were invited via a structured interview tool for a subsequent in-person interviews to determine final rank decisions. Costs of the interview process were tracked, as well as perceptions of VCIs.

Results

VCI implementation over 9 years demonstrated a median 48% reduction of in-person interviews—or 95 applicants eliminated out of a total 195 VCIs performed. This represents a mean annual direct cost savings estimated at $9,154, equating to a 55% reduction in allocated program costs, in addition to indirect cost savings to both applicants and the program.

Conclusions

Compared to exclusively in-person interviewing, the utilization of VCIs is potentially more cost-effective for residency programs and candidates, while creating a more personal experience for applicants early in the recruitment process. Limited data of acceptability among faculty and candidates is generally favorable but remains mixed.

What was known and gap

Videoconference interviews (VCIs) are increasingly being used in the residency selection process, yet remain an underutilized approach particularly among primary care programs, with little published data pertaining to its cost or acceptability.

What is new

Cost analysis indicates that VCI implementation can support significant annual direct cost savings, in addition to indirect cost savings to applicants and the program.

Limitations

Single site data, incomplete assessment of method efficacy with respect to Match outcomes, internal study variation with potential confounders, and limited available data on perceived acceptability by candidates.

Bottom line

VCIs are a potentially cost-effective approach for residency programs and candidates; more information is needed on the perceived acceptability.

Introduction

The process of residency application and candidate selection is a time-intensive and costly endeavor for medical student applicants and the training programs to which they apply. These factors of time and cost have, in fact, been cited by students as being decisive limiting factors in up to 70% of their interview acceptances to residency programs.1  Additionally, restrictions related to the current COVID-19 pandemic have led to sizeable changes in the way programs will likely approach the 2020–2021 residency match cycle.2,3 

Videoconference interviews (VCIs) are increasingly being used in the selection process of program candidates across a range of medical specialties, both in applicant screening and, in some cases, as a replacement for in-person interviews. Various studies have assessed the feasibility and cost-efficacy of using VCIs at the residency and fellowship levels of graduate medical education, exploring benefits from the perspectives of residency programs and medical student applicants. Although several fields have begun using VCIs in their selection process, most are in the preliminary stages,4  with the majority in surgical and related fields, namely anesthesia,5  orthopedics,6  ophthalmology,7  and plastic surgery.810  This may be owing to the highly competitive nature of these fields, and the consequent need to more efficiently narrow a large candidate pool.

We are using “VCI” in the synchronous sense, with applicant and interviewer interacting simultaneously via a video-based communication platform. Asynchronous VCI approaches, whereby applicants respond via video-recording to a series of predetermined interview questions, have also been explored by obstetrics and gynecology programs,11  and both approaches have been trialed by various emergency medicine programs,1216  including through the Standardized Video Interview project from the Association of American Medical Colleges (AAMC), which has yielded mixed outcomes.1719 

Current evidence supports web-based synchronous VCI as a reliable and effective adjunct to in-person interviews to contain costs and enhance convenience for programs and applicants alike.2022  The AAMC has published several informational guides to support best practice implementation of VCIs at the undergraduate medical institution level.23  This approach nevertheless remains underutilized in the Match process for residency and fellowship programs nationwide,24,25  particularly those in primary care, for which we were unable to find relevant published data.

This retrospective data review with cost analysis explores financial and acceptability outcomes of VCI implementation and use over a 9-year period, in order to share experience at a time when many programs are seeking ways to rapidly adopt VCI strategies due to the current COVID-19 pandemic.

Methods

The Phelps Family Medicine Residency Program in Sleepy Hollow, New York, a 3-year program matching 8 residents per year, was formed and accredited in 2011. An unanticipated influx of applications during the program's first interview season inspired the faculty to develop a system of web-based virtual interviews to identify ideal candidates for the Match. The program is currently using VCIs to interview all applicants, via the Skype platform, that meet certain qualifying thresholds, prior to granting in-person interviews. Skype is one of several software applications currently available that facilitate virtual meetings. The materials needed by each party in order to effectively communicate include a microphone, speakers, camera, and reliable wi-fi or data connection. Typically, these equipment needs can be met with desktop or laptop computers, tablets, or cell phones.

When initiated during the program's first recruitment season, faculty anticipated that the incorporation of VCIs into the recruitment process would afford increased access to candidates and, as opposed to traditional in-person interviewing alone, would prove cost effective for both program and candidate, while creating a more personal experience early in the recruitment process. Exceptional candidates were identified via VCIs by a standardized survey tool to semi-objectively quantify candidate responses to structured interview questions; they were subsequently invited for an in-person interview to determine final ranking decisions.

After going live, applications received through the Electronic Residency Application Service (ERAS) are filtered and divided among faculty members alphabetically by candidate surname for preliminary review. Based on these reviews, qualified candidates are offered a VCI with a core faculty member. Each faculty member can select an average of up to 20 people for a VCI. Additionally, to ensure the process allows for sufficient vetting of candidates, faculty members are generally not assigned virtual interviews with those whose application they have already reviewed. For example, faculty “A” will not complete the VCI for any applicant with last name A to F (table 1).

table 1

Faculty Assignments for Applicant Review and Videoconference Interviews (VCIs) Allocation

Faculty Assignments for Applicant Review and Videoconference Interviews (VCIs) Allocation
Faculty Assignments for Applicant Review and Videoconference Interviews (VCIs) Allocation

Each applicant who accepts a VCI receives an email with their appointment time and information relevant to the virtual interview, including instructions on using the software and general advice for digital interviews. The VCI lasts approximately 15 to 20 minutes. Structured forms developed internally are used during virtual interviews to assess candidates' interest in and knowledge of our program, as well as to assess specific Accreditation Council for Graduate Medical Education Milestones in patient care, medical knowledge, and practice-based learning and improvement (forms provided as online supplemental material). Validity testing was not performed for this tool. Promising VCI candidates meeting a threshold score on the assessment tool are then offered in-person interviews.

As an innovative approach, the process of VCI implementation underwent some degree of evolution during its first few years. In the initial interview season (2011–2012), VCIs were scheduled with 2 faculty members concurrently and were only offered on a trial basis to a small sample of the total applicant pool. During the subsequent interview season (2012–2013), VCIs were scheduled with just one faculty member, and the VCI process was further refined and incorporated into the screening process after application review, prior to granting an in-person interview. After the initial review of applications in ERAS, the applicant was chosen for either an automatic in-person interview or a preliminary VCI. If allocated to the latter, each applicant was assigned to 1 of 3 possible categories after the VCI: (1) invite for an in-person interview; (2) hold for discussion by the recruitment committee; or (3) do not rank. In every subsequent interview season since, prior to granting in-person interviews, VCIs have been utilized to screen all applicants selected through ERAS filters who meet the minimum eligibility criteria set by the program's recruitment committee; this was implemented after repeatedly interviewing apparently well-qualified applicants who would nevertheless have been screened out using VCIs. Furthermore, the structured VCI form has been revised to include standard behavioral and clinical knowledge questions for all applicants, with Likert-scale scoring of answers to determine their assignment in 1 of the 3 decision categories previously listed.

Institutional Review Board (IRB) review was sought through the Human Subjects Research Determination process, which stated that the proposed study activities did not constitute human subjects research, and therefore IRB review was not required.

Results

From the standpoint of recruitment committee efficiency, the VCI process has proven a useful means of screening applicants prior to in-person interviewing. The program's first 4 recruitment seasons saw a decrease in the number of in-person interview invitations extended without a preliminary VCI; since 2015, no applicants have been interviewed in-person without first undergoing a VCI (figure). Every year a notable proportion of applicants (median 48%) are eliminated from consideration based on the VCI alone without compromising the quality of program-matched candidates, as assessed by average scores on licensing and board examinations (table 2).

figure

Proportion of Interviews Conducted via Virtual and/or In-Person Formats, by Year

figure

Proportion of Interviews Conducted via Virtual and/or In-Person Formats, by Year

table 2

Videoconference interview (VCI) and In-Person Interview Statistics

Videoconference interview (VCI) and In-Person Interview Statistics
Videoconference interview (VCI) and In-Person Interview Statistics

The candidate review process outlined here has implications from a cost-effectiveness perspective. In the 2019–2020 interview season, to use the most recent available data, a typical interview day for 8 applicants cost an estimated $1,669 (table 3 a). Comparatively, the cost of offering VCIs to 8 candidates was $384 (0.5 hr × $96/hr × 8 applicants; table 3 b). Administrative costs related to scheduling and coordinating VCIs were nominal, owing to the “Scheduler” function made available by ERAS in recent years.

table 3

Comparative Interview Strategy Costs

Comparative Interview Strategy Costs
Comparative Interview Strategy Costs

To accommodate an additional 48% candidate volume for in-person interviews in the 2019–2020 season (n = 195), if not eliminated through VCIs conducted in advance, the program would have required an additional 15 interview days for 94 additional applicants, at an estimated total cost of $25,035. The total cost of offering VCIs during that same season was $9,360 (0.5 hr × $96/hr × 195), for a cost savings of $15,675, or a 63% reduction in allocated program costs. Even when considered more modestly over the 9-year study period, a median 48% more applicants added to the mean annual volume (n = 157) of total interviews conducted equates to an additional 10 interview days for 76 additional applicants, at an estimated total annual cost of $16,690. The cost of offering VCIs to that number of applicants would be $7,536 (0.5 hour × $96/hr × 157), for a total cost savings of $9,154, or a 55% reduction in allocated program costs. This allows the program to interview a wider range of candidates, while simultaneously enabling multiple faculty members to more comprehensively review and establish contact with a growing applicant pool, and to do so across various interview settings (ie, in the applicant's home, onsite at the residency, etc). Of note, through the use of internally developed checklists and application review tools, the incremental increases in time required for subsequent faculty application reviewing is minimal.

With regard to acceptability by faculty and candidates, informal data suggest a mixed response. Faculty and candidates were asked for feedback via an annual post-interview survey during the first 8 years of VCI implementation. One faculty member commented that it was “a novel idea, which enabled us to get to know the applicants prior to actually meeting them on interview day,” and faculty members have consistently elected to keep the VCI format year after year as a successful and convenient recruitment method.

Applicant comments on the post-match survey varied in enthusiasm for the format. Positive statements included those such as “[The VCI] was innovative, exciting, an overall great experience”; “[the] Skype interview gave me a chance to meet the faculty and sort of be nervous on my own before actually getting [to meet them] in person”; and “It was a nice opportunity to learn more about the program and have some questions answered. Being interviewed at home was comfortable and convenient.” However, while only 2 comments in the survey's open-ended response portion addressed the VCI interview in the 8-year period, both reflected a negative perception by the respondents, who felt that the element should be eliminated.

The survey specifically queried applicants on the VCI interview during 3 successive interview seasons (2014–2017) with the question, “What did you think about the Skype interview?” Of a total 39 of 220 survey respondents (18% response rate), 27 (69%) rated it as “great,” “good,” or “fine” and 14 (36%) found it “interesting.” Only 5 respondents (13%) felt it had saved them time or expenses, and 4 (10%) felt it was “a waste of time.”

Discussion

To our knowledge, this VCI-based interviewing approach is being done elsewhere in few primary care residency programs, and certainly not at this scale within their programs or for this extended period of time. Based on experience gleaned from both VCI and in-person interviewing, through a process now successfully replicated over 9 annual recruiting cycles, we are convinced that these methods complement one another. Utilizing both types of interview creates an efficient process that supports the assessment of large pools of highly qualified candidates. It is also an opportunity for individual programs to stand out as unique amidst the many training options nationwide, serving as a first touchpoint with candidates. This can aid in determining match probability early on, especially as applicants apply to greater numbers of programs. This aspect will prove increasingly salient as primary care fields such as family medicine are projected to become more competitive based on a review of recent trends.26 

It is important to state that the degree to which a program will see cost benefit with this model is likely to vary widely depending on program characteristics. Primary care residency programs typically interview 15 to 17 candidates and rank 8 to 10 candidates per available position.27  Therefore, while an 8-8-8 program would generally rank approximately 80 candidates, various features of the program (new vs established reputation, rural vs urban setting) can impact how wide the net must be cast to optimize match outcomes. In the context of seismic changes being wrought by the COVID-19 pandemic on the 2020–2021 residency recruitment cycle, even more established programs are likely to process higher-than-normal applicant volumes this year, thereby benefitting from a more rigorous prescreening approach. Regardless of a program's chosen vetting process, virtual interviewing may well become the norm this year and beyond.

The study described had several limitations. Use of examination scores was our main measure of method efficacy, through demonstrating consistent quality among program-matched candidates year to year, although this has admittedly not been shown to be an accurate predictor of physician success in patient care. The cost estimate provided does not account for benefits of potential income generated through increased faculty and resident clinical productivity (owing to fewer in-person interview days). It furthermore does not quantify the benefit of flexible scheduling afforded by VCIs, which obviates the need for complex cross-coverage and schedule coordination for faculty, residents, and administrative personnel to allow in-person interviewing days. While consistency was ensured in large part year to year in the use of this web-based virtual interviewing and candidate selection process, there was some variation as the process was refined with subsequent iterations, including the vetting and implementation of interview tools to better standardize the VCI. Various changes in the program's staffing, curricular elements, and institutional partnerships also contributed numerous confounders to match outcomes from year to year. Additionally, while it has implications for generalizability to other residency programs within primary care and graduate medical education more broadly, the collated data nevertheless reflect the experience of only one program. Finally, our data on the acceptability to candidates are limited, as the post-match surveys were not designed to be used in a rigorous post-hoc analysis, and were therefore of limited utility in this retrospective review (ie, the survey questions were not standardized across all 8 years and had poor overall response rates).

In future recruiting seasons, particularly during the anticipated transition to an exclusive VCI approach in the 2020–2021 recruiting cycle (to accommodate COVID-19-related social distancing restrictions), we are considering a trial of reduced in-person interviews. We are also exploring the possibility of creating virtual “hangout groups” to enhance candidate-resident interactions and virtual tours of our facilities and practice locations to supplement the VCI experience.

Conclusions

Interviewing a pool of qualified candidates using VCIs is an effective way for primary care residency programs to interact with applicants. It optimizes the use of program and applicant resources for in-person interviews. It allows many candidates to interact with faculty and learn about the program without having to travel, and it is an innovative tool commensurate with current evolving norms for digital communication in a technologically oriented society. The implementation of VCIs enhances face-to-face contact at reduced cost, providing efficient applicant assessment early in the process, and enables utilization of technology for timely contact between programs and applicants, enhancing the decision-making process for both parties.

References

References
1.
Callaway
P,
Melhado
T,
Walling
A,
Groskurth
J.
Financial and time burdens for medical students interviewing for residency
.
Fam Med
.
2017
;
49
(2)
:
137
140
.
2.
Joshi
A,
Bloom
DA,
Spencer
A,
Gaetke-Udager
K,
Cohan
RH.
Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond
.
Acad Radiol
.
2020
;
27
(9)
:
1316
1322
.
3.
Gabrielson
AT,
Kohn
JR,
Sparks
HT,
Clifton
MM,
Kohn
TP.
Proposed changes to the 2021 residency application process in the wake of COVID-19
.
Acad Med
.
2020
;
95
(9)
:
1346
1349
.
4.
Edje
L,
Miller
C,
Kiefer
J,
Oram
D.
Using Skype as an alternative for residency selection interviews
.
J Grad Med Educ
.
2013
;
5
(3)
:
503
505
.
5.
Vadi
MG,
Malkin
MR,
Lenart J, Stier GR, Gatling JW, Applegate RL 2nd. Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study
.
Int J Med Ed
.
2016
;
7
:
102
108
.
6.
Healy
WL,
Bedair
H.
Videoconference interviews for an adult reconstruction fellowship: lessons learned
.
J Bone Joint Surg Am
.
2017
;
99
(21)
:
e114
.
7.
Pasadhika
S,
Altenbernd
T,
Ober
RR,
Harvey
EM,
Miller
JM.
Residency interview video conferencing
.
Ophthalmology
.
2012
;
119
(2)
:
426
426.e5
.
8.
Susarla
SM,
Swanson
EW,
Slezak
S,
Lifchez
SD,
Redett
RJ.
The perception and costs of the interview process for plastic surgery residency programs: can the process be streamlined?
Plast Reconstr Surg
.
2017
;
139
(1)
:
e302
e309
.
9.
Miotto
GC.
Tele-interview in the aesthetic fellowship selection process
.
Aesthet Surg J.
2018
;
38(11):NP175–NP177.
10.
Melendez
MM,
Dobryansky
M,
Alizadeh
K.
Live online video interviews dramatically improve the plastic surgery residency application process
.
Plast Reconstr Surg
.
2012
;
130
(1)
:
e240
e241
.
11.
Breitkopf
DM,
Green
IC,
Hopkins
MR,
Torbenson
VE,
Camp
CL,
Turner
NS
3rd.
Use of asynchronous video interviews for selecting obstetrics and gynecology residents
.
Obstet Gynecol
.
2019
;
134
(Suppl 1)
:
9
15
.
12.
Crawford
SB,
Monks
SM,
Wells
RN.
Virtual reality as an interview technique in evaluation of emergency medicine applicants
.
AEM Educ Train
.
2018
;
2
(4)
:
328
333
.
13.
Van Dermark
JT,
Wald
DA,
Corker
JR,
Reid
DG.
Financial implications of the emergency medicine interview process
.
AEM Educ Train
.
2017
;
1
(1)
:
60
69
.
14.
Blackshaw
AM,
Watson
SC,
Bush
JS.
The cost and burden of the residency Match in emergency medicine
.
West J Emerg Med
.
2017
;
18
(1)
:
169
173
.
15.
Pourmand
A,
Lee
H,
Fair
M,
Maloney
K,
Caggiula
A.
Feasibility and usability of tele-interview for medical residency interview
.
West J Emerg Med
.
2018
;
19
(1)
:
80
86
.
16.
Schnapp
BH,
Ritter
D,
Kraut
AS,
Fallon
S,
Westergaard
MC.
Assessing residency applicants' communication and professionalism: standardized video interview scores compared to faculty gestalt
.
West J Emerg Med
.
2019
;
20
(1)
:
132
137
.
17.
Deiorio
NM,
Jarou
ZJ,
Alker
A,
Bird
SB,
Druck
J,
Gallahue
FE,
et al.
Applicant reactions to the AAMC standardized video interview during the 2018 application cycle
.
Acad Med.
2019
;
94(10)1498–1505.
18.
Gallahue
FE,
Hiller
KM,
Bird
SB,
Calderone Haas
MR,
Deiorio
NM,
Hern
HG,
et al.
The AAMC standardized video interview: reactions and use by residency programs during the 2018 application cycle
.
Acad Med
.
2019
;
94
(10)
:
1506
1512
.
19.
Association of American Medical Colleges
.
AAMC Standardized Video Interview Evaluation Summary
.
2020
.
20.
Shah
SK,
Arora
S,
Skipper
B,
Kalishman
S,
Timm
TC,
Smith
AY.
Randomized evaluation of a web based interview process for urology resident selection
.
J Urol
.
2012
;
187
(4)
:
1380
1384
.
21.
Bird
SB,
Hern
HG,
Blomkalns
A,
Deiorio
NM,
Haywood
Y,
Hiller
KM,
et al.
Innovation in residency selection: the AAMC standardized video interview
.
Acad Med
.
2019
;
94
(10)
:
1489
1497
.
22.
Chung
AS,
Shah
KH,
Bond
M,
Ardolic
B,
Husain
A,
Li
I,
et al.
How well does the standardized video interview score correlate with traditional interview performance?
West J Emerg Med
.
2019
;
20
(5)
:
726
730
.
23.
Association of American Medical Colleges
.
Conducting Interviews During the Coronavirus Pandemic
.
2020
.
24.
Hariton
E,
Bortoletto
P,
Ayogu
N.
Using video-conference interviews in the residency application process
.
Acad Med
.
2017
;
92
(6)
:
728
729
.
25.
Hariton
E,
Bortoletto
P,
Ayogu
N.
Residency interviews in the 21st century
.
J Grad Med Educ
.
2016
;
8
(3)
:
322
324
.
26.
Phillips
JP,
Wendling
A,
Bentley
A,
Marsee
R,
Morley
CP.
Trends in US medical school contributions to the family physician workforce: 2018 update from the American academy of family physicians
.
Fam Med
.
2019
;
51
(3)
:
241
250
.
27.
Nilsen
K,
Callaway
P,
Phillips
JP,
Walling
A.
How much do family medicine residency programs spend on resident recruitment? A CERA study
.
Fam Med
.
2019
;
51
(5)
:
405
412
.

Author notes

Editor's Note: The online version of this article contains structured forms to assess candidate interest in and knowledge of the residency program.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

Supplementary data