Background Although the selection interview is a standard admission practice for graduate medical education (GME) programs in the United States, there is a dearth of recent reviews on optimizing the trainee interview process, which has low reliability, high cost, and major risk of bias.

Objective To investigate the evidence base for different selection interview practices in GME.

Methods We searched 4 literature databases from inception through September 2022. Two investigators independently conducted title/abstract screening, full-text review, data extraction, and quality assessment. Disagreements were mediated by discussion. We used backward reference searching of included articles to identify additional studies. We included studies of different interview methods and excluded literature reviews, non-GME related publications, and studies comparing different applicant populations. We examined study characteristics, applicant and interviewer preferences, and interview format. We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI).

Results Of 2192 studies, 39 (2%) met our inclusion criteria. The evidence base was rated as moderately low quality using MERSQI criteria. Applicants reported preferences for several one-on-one interviews lasting 15 to 20 minutes, interviews by current trainees, and interviews including social events with only trainees. Applicants had mixed perceptions of virtual versus in-person interviews and reported that virtual interviews saved costs. The multiple mini interview (MMI) required more applicant and interviewer time than individual interviews but demonstrated construct and predictive validity and was preferred by applicants and interviewers.

Conclusions Based on moderately low-quality evidence, using the MMI, training interviewers, and providing applicants with basic program information in advance should be considered for GME selection interviews.

The selection interview is a standard practice for admission to graduate medical education (GME) programs in the United States.1  Program directors (PDs) consider the interview important, allowing for the assessment of noncognitive traits and “fit” in applicants.2  However, there remain significant challenges with the selection interview, including low reliability,3  high cost,4  and major risk of bias.5,6 

Improving the interview process is important to optimize the trainee selection process and match outcomes. Recent reviews studied evidence-based practices for admissions interviews in medical schools7  and evaluated applicant and program perceptions of virtual interviews in GME.4  In this systematic review, we investigated the evidence base for different methods of conducting selection interviews in GME, with the goal of identifying best practices.

We conducted a systematic review to evaluate current interviewing practices and identify areas for future research.8  We chose a systematic review because it is a rigorous and comprehensive method to collect, analyze, and synthesize existing evidence on a topic. We developed a literature search strategy (provided as online supplementary data) with a Brown University health sciences librarian. We searched the Embase, ERIC, PubMed, and Web of Sciences databases from inception through September 30, 2022. Two investigators (J.L., D.H.) independently conducted title/abstract screening, full-text review, and data extraction in Covidence in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.9  We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI), which is a validated tool designed to assess the methodological quality of medical education studies across 6 domains: study design, sampling, type of data, validity of evaluation instrument, data analysis, and outcomes.10  In the literature, MERSQI scores over 12.5 have been considered higher-quality studies,7  although the MERSQI does not endorse a binary concept of study quality.10  Disagreements were resolved first by discussion with the 2 investigators, followed by mediation via the senior investigator (P.B.G.) if necessary. Cohen’s kappa was calculated to quantify interrater reliability. We used backward reference searching by reviewing the references of included articles to identify additional eligible studies.

We included studies that compared different methods of conducting an admissions interview in GME, including residency and fellowship programs for physician training after medical school. We excluded publications that: (1) were literature reviews, perspectives, or case reports; (2) were not related to GME (eg, in undergraduate medical education); (3) compared interview methods in applicants from different populations (eg, 2 different GME programs) or from different time periods (eg, 1989 vs 2001), as these differences could confound study findings. We included studies conducted at multiple institutions if all applicants were exposed to both the intervention and comparator.

Crossover studies were defined as those in which 2 or more different treatments were applied to subjects at different time periods.11  Parallel studies were defined as those in which 2 or more groups of subjects received different treatments.11  Structured interviews were defined as those in which interviewees responded to the same set of mandatory questions or prompts. Semistructured interviews utilized mandatory prompts but allowed interviewers to ask additional questions not provided by programs. Unstructured interviews were defined as those in which the interviewers were not required to ask specific prompts.

The research process included regular team reflections during data collection and analysis. We extracted data on institution, country, publication year, study design, interview methods, study outcomes, and key findings. We then compiled a narrative summary of our results based on the breadth and variety of methods and outcomes included in the review. We identified areas across interviewing that have received significant attention, as well as those that received less. This systematic review was conducted in accordance with the ethical principles of the Declaration of Helsinki.

In total, 39 studies met the inclusion criteria after duplicate removal, title/abstract screening, and full-text review (Figure).12-50  Backward reference searching did not identify additional eligible studies. In title/abstract screening, Cohen’s kappa was 0.39, and there was 7% disagreement (94 of 1387). In full-text review, Cohen’s kappa was 0.37, and there was 22% disagreement (12 of 55). All disagreements were resolved by discussion.

Figure

PRISMA Flow Diagram

Note: We incorporated studies examining various interview methodologies within graduate medical education (GME), encompassing both residency and fellowship programs post-medical school. Exclusions were: (1) literature reviews, perspectives, and case reports; (2) studies outside the GME purview such as undergraduate medical education; (3) those contrasting disparate applicant cohorts; and (4) studies that did not compare interventions. Notably, during our full-text review, a significant number of excluded studies were either single-arm investigations or pertained to undergraduate medical education.

Figure

PRISMA Flow Diagram

Note: We incorporated studies examining various interview methodologies within graduate medical education (GME), encompassing both residency and fellowship programs post-medical school. Exclusions were: (1) literature reviews, perspectives, and case reports; (2) studies outside the GME purview such as undergraduate medical education; (3) those contrasting disparate applicant cohorts; and (4) studies that did not compare interventions. Notably, during our full-text review, a significant number of excluded studies were either single-arm investigations or pertained to undergraduate medical education.

Close modal

Study Characteristics

The 39 included studies are described in Table 1. The studies were published from 1985 to 2022, with 37 (95%) studies published after 2000, and 32 (82%) studies published from 2010 to 2022. All but one were conducted in World Bank-classified high-income countries: 24 (62%) in the United States, 8 (21%) in Canada, 3 (8%) in Japan, and 1 (3% each) in Argentina, Denmark, Oman, and the Netherlands. Twenty-six studies (67%) were conducted at a single institution, 3 (8%) at multiple institutions, 2 (5%) did not report their institution, and 8 (21%) were general surveys sent to many institutions. Thirty-five studies (90%) evaluated residency programs, spanning over 19 distinct specialties. Four (10%) evaluated fellowship programs, covering 3 specialties. Additionally, 9 (23%) were crossover studies, 6 (15%) were parallel studies, 7 (18%) were post-interview surveys, 3 (8%) were post-application surveys, 3 (8%) were crossover studies with post-interview surveys, 2 (5%) were randomized controlled trials, and 1 (3%) was a pre-interview survey. The included studies were rated as moderately low quality by MERSQI (Table 2), averaging a mean (standard deviation [SD]) of 10.9 (1.5) out of a maximum of 18.

Table 1

Design and Details of Included Studies

Design and Details of Included Studies
Design and Details of Included Studies
Table 2

Findings and Outcomes of Included Studies

Findings and Outcomes of Included Studies
Findings and Outcomes of Included Studies

Thirty-four of the 39 studies (87%) reported their sample size. Of these, the mean sample was 187.7 (SD=338.1). Response rates of the 14 surveys (36%) ranged from 21% to 100%. Ten studies (26%) compared in-person to virtual interviews; 8 (21%) compared unstructured interviews to the multiple mini interviews (MMIs), which involve several short, separate interviews by different interviewers for each applicant20 ; 6 (15%) compared more than 2 interview methods, 4 (10%) compared different MMI formats, 4 (10%) compared structured to unstructured interviews, 3 (8%) compared closed-file to open-file interviews, and 2 (5%) compared semistructured to unstructured interviews. The remaining studies compared faculty to resident interviewers,16  2 to 3 interviewers,47  informal to formal dress codes,31  use of a pre-interview informational video to no pre-interview video,27  and a single-site interview fair to individual program interviews.28 

Twelve outcomes were identified from included studies: applicant costs,12,28,31,40,41  applicant perceptions,12,13,17-22,28,31,33-35,37,40,41,45,49  applicant time,27,38  concurrent validity,38  construct validity,36,43,44  institutional costs,15,28,39  interview bias,26,30,42  interview reliability,24,25,29,32,47,49,50  interviewer time,15,23,27,38,39  interviewer perceptions,19,28,41,49,50  PD perceptions,15,34,37,39,46  and predictive validity.14,16,48 

Study Outcomes

Applicant Preferences: In surveys sent to applicants by individual programs in gastroenterology, orthopedics, and urology, 67% of applicants preferred one-on-one (as opposed to 1:2, 1:3, 1:4) interviews,34  55 to 68% preferred 15-to-20 minute interviews,18,34  83% preferred 5- to 7-minute office setting faculty interviews,33  and 50 to 95% preferred interview days that lasted one-half to three-fourths of a workday.18-33  Ninety-six percent of orthopedic respondents wanted to be interviewed by current residents, and 94% wanted to have a social event in the interview process;18  88% of urology applicants wanted to interview with at least half of a program’s faculty.33  Of orthopedic applicants, 36% wanted skills tasks and 23% wanted knowledge tests during their interview.18  Ninety-seven percent of urology applicants believed that interview offers should be released on the same day.20 

In one study, 85% of applicants surveyed preferred an explicitly informal dress code over an unspoken formal dress code; 22% of respondents reported that this led them to rank the program higher and 28% reported it reduced their costs.31  Showing applicants a video about program information before the interview reduced time spent on program information during the interview.27  A post-interview survey conducted by Canadian urology programs showed that applicants and interviewers preferred a single-site, single-day interview fair with multiple programs present over individual interviews in multiple geographically distinct sites with each program/applicant; applicant costs were reduced from an estimated $2,065 to $367 for the fair, although institutional costs were slightly higher ($1,931 rather than an estimated $1,825).28 

Interviewer Practices: Interviewers rated applicants higher based on their grades, United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, school ranking, and application score, when available.42,44  Interviewers initially rated applicants who had rotated in their department higher, although this effect disappeared after they reviewed applicant files in one study.30  Interview ratings for the communication, collaboration, management, medical expertise, and scholarship domains had acceptable reliability (intraclass correlations: 0.900, 0.739, 0.585, and 0.585, respectively) in a Canadian ophthalmology program, but ratings for the professionalism and health advocacy domains did not.29 

Interview Format: In-Person vs Virtual: Ten of the 39 studies (26%) were conducted during or after the onset of the COVID-19 pandemic. Of these, 8 compared in-person versus virtual formats.

Applicants and PDs had mixed perceptions of virtual interviews. In 6 surveys of applicants, 1 reported that 80% of radiology applicants during the COVID-19 virtual interview season agreed that the benefits of the virtual interview season outweighed the drawbacks,37  1 reported that 73% of colorectal surgery applicants recommended virtual interviews,21  3 surveys reported that gastroenterology and urology applicants had mixed preferences,20,34,35  and 1 reported that virtual colorectal surgery applicants preferred virtual interviews and in-person applicants preferred in-person interviews.12  When allowed to choose an interview format, anesthesiology applicants who selected virtual interviews did so due to scheduling conflicts, distance, and costs, whereas applicants who selected in-person interviews did so for resident interactions, geographic proximity, and visiting campus.45  Of 4 general PD surveys, 2 reported that surgical PDs did not believe that virtual interviews were better than in-person interviews, 1 reported that gastroenterology PDs had mixed perceptions, and 1 reported that radiology PDs preferred virtual interviews. In a post-interview survey, virtual urology applicants and interviewers preferred continuing virtual interviews in addition to in-person interviews.41 

Applicants and PDs found virtual interviews more affordable than in-person interviews but disagreed on whether they saved time. Three general surveys found that applicants across all specialties on average spent less money for virtual interviews.12,40,41  On average, general surgery programs saved $6,462 after one year of virtual interviews. A general survey found that for urology applicants, 10% missed days at school with virtual interviews versus 30% with in-person interviews.41  Surgical PDs did not believe that virtual interviews were less time-consuming for programs.15,39 

Interview Format—The MMI: Applicants and interviewers reported to programs that they preferred the MMI over unstructured and other types of interviews. Most or all applicants preferred MMIs in 5 post-interview surveys in 7 specialties (100%,13  91%,17  78%,19  74%,22  100%50 ), and most interviewers preferred the MMI in 1 post-interview survey.19  In single studies, the MMI required less faculty time than panel interviews,23  required more faculty and resident time than individual interviews,38  extended interview day length by 15%,38  and doubled face-to-face interview time for applicants.38  Another study found that the MMI and unstructured interview were moderately correlated but disagreed on rank order lists, indicating that the MMI and unstructured interview measured different constructs.43 

In MMI stations, behavioral and situational questions demonstrated reliability per generalizability theory, though behavioral questions were preferred by interviewers, and they predicted 2-year faculty evaluations of residents.49,50  Applicants did not express a preference, but interviewers preferred behavioral over situational questions.49,50  Behavioral questions had lower interview reliability than situational questions,49  although both types had acceptable reliability (G=0.87 and 0.96, respectively).50  One study in neurology found that MMI station ratings were generally not significantly correlated,36  while another study in obstetrics and gynecology (OB/GYN) found that several MMI station ratings were correlated with each other.38  These results suggest that MMI stations overlap in their construct assessment. Behavioral questions were correlated with USMLE Step 2 scores; role play ratings were correlated with USMLE Step 1 scores; and social media ethics and surgical technical simulation ratings were correlated with male sex.38  Behavioral and situational questions were both correlated with 1-year faculty evaluations of residents in 3 specialties, but only behavioral questions were correlated after 2 years.48 

Interview Format—Structured (Non-MMI) vs Unstructured: Unstructured, semistructured, and non-MMI structured interviews mostly had similar interrater reliability. A study in orthopedics reported that there were no differences in interrater reliability between semistructured and unstructured interviews;25  another study in family medicine found that semistructured but not unstructured interviews were reliable in 1 out of 2 rounds.32  Interview reliability decreased slightly when 2, rather than 3, semistructured interview ratings were considered.47  However, interviewer training improved interrater reliability.24  Structured and unstructured interviews in anesthesiology were rated similarly.26 

The evidence was mixed regarding predictive validity of unstructured and structured interviews. Neither unstructured nor structured behavioral interviews predicted an applicant’s future clinical rating by their PD in one radiology residency program.14  Similarly, another study reported that different interview methods across residency programs were not correlated with subsequent PD satisfaction with their matched residents.46  In contrast, unstructured faculty and resident team interview scores were associated with future faculty and nursing evaluations in one OB/GYN residency program.16 

In this review of 39 studies of overall moderately low quality, we investigated different selection interview methods in GME to identify evidence-based practices. Applicants had mixed perceptions of virtual versus in-person interviews and generally preferred multiple one-on-one interviews and including interviews with current residents. Both applicants and PDs reported that virtual interviews saved costs. The MMI was preferred by applicants and interviewers over unstructured interviews and demonstrated construct validity, but it required more applicant and interviewer time than individual interviews. Aside from the MMI, adding structure to interviews did not improve reliability or predictive validity in this review. Masking interviewers to applicants’ academic records reduced the influence of academic performance on interview scores.

Implications for Interviewing in GME

As advised by the Association of American Medical Colleges, GME programs may consider conducting virtual interviews,51  which save time and money for applicants and programs and may help make the application process more equitable.51  Additionally, though there is a paucity of research on the reliability and validity of virtual MMIs in GME,4  virtual and in-person MMI scores were similar for medical and dental school admissions.52  Hence, a virtual MMI may be an option for programs seeking to utilize virtual interviews.

Within the 39 studies, conducted mostly in North American, university-based residency programs, the MMI had acceptable reliability, construct validity, predictive validity for 1- and 2-year faculty evaluations of trainees, and was rated favorably by applicants and interviewers in post-interview surveys. These findings align with a systematic review of admissions interviews in undergraduate medical education that determined that the MMI was reliable, unbiased, and predicted first-year performance on written examinations and Objective Structured Clinical Examinations.8  In one study, behavioral and situational MMI questions were both reliable and predictive of 1-year faculty evaluations of residents, and only behavioral MMI questions were predictive of 2-year faculty evaluations of residents.48  This information about the reliability and validity of behavioral MMI questions may be helpful when designing MMIs.48  Another study showed that the MMI and unstructured interviews produced different rank order lists.43  In comparison to the MMI, there was limited reliability and validity evidence for unstructured and structured interviews. However, potential drawbacks of the MMI include potential interviewer biases towards applicants based on age, rural background, and cultural and language barriers.53-56  Additionally, these findings may not be applicable to less-studied settings like community-based programs and populations like international medical graduates.

Although the evidence base was weak and meta-analysis was not performed, this review uncovered several general considerations for the conduct of GME interviews. Training interviewers may help ensure impartial, reliable interviewing. Resident team interviews can be predictive of faculty and nursing evaluations; indeed, applicants wanted to be interviewed by current residents. Interviewers provided with academic information before interviews, including USMLE scores and grades, may prejudge applicants accordingly. Additionally, providing basic program information before interviews may allow applicants and interviewers to discuss other information. When deciding upon an interview format, programs should also identify clear goals for the interview. For example, programs may seek to evaluate applicants’ interpersonal and problem-solving skills or assess their awareness and sensitivity toward different cultures and social backgrounds. Programs can also structure their interview day agenda to align with applicant preferences. Based on 3 surveys, these include holding 5 to 7 one-on-one interviews lasting 15 to 20 minutes, separate social events with only residents and with faculty, and an informal dress code. Applicants also preferred not to be tested on their skills or knowledge. However, the costs of implementation may render some of these ideas impractical for many programs.

This systematic review evaluated multiple interview methods in resident and fellow selection, building upon the work of prior reviews3,5  with a standardized literature search, adherence to a predefined protocol, use of PRISMA reporting guidelines, and study quality assessment with MERSQI. A 2015 literature review found that masking interviewers could reduce interviewer bias and had mixed findings regarding the predictive value of interviews for performance in residency.2  A 2022 systematic review on virtual interviewing identified applicant and interviewer satisfaction with virtual interviews as well as cost and time savings but did not investigate the predictive validity or reliability of a virtual MMI.4 

Limitations of the Evidence Base

Current studies on admissions interviewing in GME have several key limitations. There was a paucity of high-quality studies; only 2 randomized controlled trials were eligible for inclusion in this review, and 35 of the 39 included studies (90%) were rated as low quality by MERSQI. Of eligible studies, few were available for many outcomes of interest, and all had been published in or after 1985. Several studies did not report important information, such as institution and specialty. None of the studies compared patient outcomes for admitted residents based on their interview performance; instead, included studies used proxies such as faculty and nursing evaluations. Many studies did not describe their interview procedures, including how the interviewers were selected and trained. There was widespread heterogeneity in interview procedures, survey methodology, and outcome measurement that precluded meta-analysis. Some findings may not be transferable among specialties or institutions, as smaller departments may have fewer resources and faculty for interviews. Additionally, most relevant studies were conducted at academic institutions in our literature search. Hence, our findings may be less generalizable to GME programs—such as community-based programs57—underrepresented in published research.57  More research is needed to identify effective interview methods for these programs.

Moreover, the surveys used to evaluate interviewer and interviewee preferences were subject to several limitations. The program-specific surveys used in 11 (28%) of the included studies were vulnerable to response bias.58,59  Methods to minimize response bias include highlighting the scientific nature of the survey and its importance for future applicants, avoiding questions that may identify specific respondents, sending the survey after the interview process is completed and stating that responses will only be reviewed post-match.58-61  However, a general survey of colorectal surgery applicants found that applicants interviewed virtually preferred virtual interviews and that in-person interviewees preferred in-person interviews,12  which suggests that interview satisfaction surveys may be more broadly subject to response bias. While 29 (74%) of the included studies were conducted before the COVID-19 pandemic, 8 of the 10 studies comparing virtual versus in-person interviews were conducted during or shortly after the COVID-19 public health emergency ended. Applicant and interviewer preferences may shift over time as programs refine the virtual format and explore its implications for the trainee selection process.

Limitations of the Review

There are several limitations to this review. We may have omitted relevant articles. This could arise from errors in our search strategy, although we consulted a health sciences librarian to optimize our search, reviewed the reference lists of included articles, and searched 4 major health and education databases. This could also arise from investigator bias and errors, although 2 investigators independently conducted each step of study identification and data extraction with third party mediation. In addition, we limited the review to comparative studies, which may have omitted related studies but ensured that more rigorous evidence was included.10  Quantitative meta-analysis was not possible due to study heterogeneity. Finally, we did not study the weighting of the interview in developing rank-order lists in GME, which also impacts the trainee selection process.

Based on moderately low-quality evidence, this systematic review found several practices that should be considered in the GME selection interview, including using the MMI, conducting interviewer training, and providing applicants with information about a training program in advance.

1. 
National Resident Matching Program
.
Results of the 2022 NRMP Program Director Survey
.
2. 
Stephenson-Famy
A,
Houmard
BS,
Oberoi
S,
Manyak
A,
Chiang
S,
Kim
S.
Use of the interview in resident candidate selection: a review of the literature
.
J Grad Med Educ
.
2015
;
7
(
4
):
539
-
548
.
3. 
Kreiter
CD,
Yin
P,
Solow
C,
Brennan
RL.
Investigating the reliability of the medical school admissions interview
.
Adv Health Sci Educ Theory Pract
.
2004
;
9
(
2
):
147
-
159
.
4. 
Daniel
M,
Gottlieb
M,
Wooten
D,
et al.
Virtual interviewing for graduate medical education recruitment and selection: a BEME systematic review: BEME guide no. 80
.
Med Teach
.
2022
;
44
(
12
):
1313
-
1331
.
5. 
Leduc
J-M,
Rioux
R,
Gagnon
R,
Bourdy
C,
Dennis
A.
Impact of sociodemographic characteristics of applicants in multiple mini-interviews
.
Med Teach
.
2017
;
39
(
3
):
285
-
294
.
6. 
Chatterjee
A,
Greif
C,
Witzburg
R,
Henault
L,
Goodell
K,
Paasche-Orlow
MK.
US medical school applicant experiences of bias on the interview trail
.
J Health Care Poor Underserved
.
2020
;
31
(
1
):
185
-
200
.
7. 
Lin
JC,
Lokhande
A,
Margo
CE,
Greenberg
PB.
Best practices for interviewing applicants for medical school admissions: a systematic review
.
Perspect Med Educ
.
2022
;
11
(
5
):
239
-
246
.
8. 
McGaghie
WC.
Varieties of integrative scholarship: why rules of evidence, criteria, and standards matter
.
Acad Med
.
2015
;
90
(
3
):
294
-
302
.
9. 
Page
MJ,
McKenzie
JE,
Bossuyt
PM,
et al.
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
.
BMJ
.
2021
;
372
:
n71
.
10. 
Reed
DA,
Cook
DA,
Beckman
TJ,
Levine
RB,
Kern
DE,
Wright
SM.
Association between funding and quality of published medical education research
.
JAMA
.
2007
;
298
(
9
):
1002
-
1009
.
11. 
Louis
TA,
Lavori
PW,
Bailar
JC III,
Polansky
M.
Crossover and self-controlled designs in clinical research. In:
Bailar
JC III,
Mosteller
F
, eds.
Medical Uses of Statistics
. 2nd ed.
CRC Press
;
1992
:
83
-
104
.
12. 
Abelson
JS,
Al Jabri
A,
Kelley
SR,
D’Angelo
JD,
D’Angelo
AD,
Husain
S.
Applicants’ perspectives on virtual vs in-person colorectal residency interviews
.
Dis Colon Rectum
.
2022
;
65
(
10
):
1162
-
1166
.
13. 
Al Abri
R,
Mathew
J,
Jeyaseelan
L.
Multiple mini-interview consistency and satisfactoriness for residency program recruitment: Oman evidence
.
Oman Med J
.
2019
;
34
(
3
):
218
-
223
.
14. 
Altmaier
EM,
Smith
WL,
O’Halloran
CM,
Franken
EA
.
The predictive utility of behavior-based interviewing compared with traditional interviewing in the selection of radiology residents
.
Invest Radiol
.
1992
;
27
(
5
):
385
-
389
.
15. 
Asaad
M,
Elmorsi
R,
Ferry
AM,
Rajesh
A,
Maricevich
RS.
The experience of virtual interviews in resident selection: a survey of program directors in surgery
.
J Surg Res
.
2022
;
270
:
208
-
213
.
16. 
Beran
B,
Kaljo
K,
Narayan
R,
Lemen
P.
An analysis of obstetrics-gynecology residency interview methods in a single institution
.
J Surg Educ
.
2019
;
76
(
2
):
414
-
419
.
17. 
Burgos
LM,
Lima
AADE,
Parodi
J,
et al.
Reliability and acceptability of the multiple mini-interview for selection of residents in cardiology
.
J Adv Med Educ Prof
.
2020
;
8
(
1
):
25
-
31
.
18. 
Camp
CL,
Sousa
PL,
Hanssen
AD,
et al.
Orthopedic surgery applicants: what they want in an interview and how they are influenced by post-interview contact
.
J Surg Educ
.
2016
;
73
(
4
):
709
-
714
.
19. 
Campagna-Vaillancourt
M,
Manoukian
J,
Razack
S,
Nguyen
LH.
Acceptability and reliability of multiple mini interviews for admission to otolaryngology residency
.
Laryngoscope
.
2014
;
124
(
1
):
91
-
96
.
20. 
Carpinito
GP,
Khouri
RK,
Kenigsberg
AP,
et al.
The virtual urology residency match process: moving beyond the pandemic
.
Urology
.
2021
;
158
:
33
-
38
.
21. 
D’Angelo
AD,
D’Angelo
JD,
Beaty
JS,
et al.
Virtual interviews—utilizing technological affordances as a predictor of applicant confidence
.
Am J Surg
.
2021
;
222
(
6
):
1085
-
1092
.
22. 
Dore
KL,
Kreuger
S,
Ladhani
M,
et al.
The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions
.
Acad Med
.
2010
;
85
(
suppl 10
):
60
-
63
.
23. 
Finlayson
HC,
Townson
AF.
Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview
.
Am J Phys Med Rehabil
.
2011
;
90
(
4
):
330
-
335
.
24. 
Gardner
AK,
D’Onofrio
BC,
Dunkin
BJ.
Can we get faculty interviewers on the same page? An examination of a structured interview course for surgeons
.
J Surg Educ
.
2018
;
75
(
1
):
72
-
77
.
25. 
Gilbart
MK,
Cusimano
MD,
Regehr
G.
Evaluating surgical resident selection procedures
.
Am J Surg
.
2001
;
181
(
3
):
221
-
225
.
26. 
Gordon
EKB,
Clapp
JT,
Heins
SJ,
et al.
The role of the interview in residency selection: a mixed-methods study
.
Med Educ
.
2020
;
54
(
11
):
1029
-
1039
.
27. 
Greene
HL,
Beattie
HM,
Russon
AR,
Johnson
A,
Stickley
W,
Goldberg
R.
Impact of a residency program information videotape on resident selection interviewing as a time-saving strategy
.
J Med Educ
.
1985
;
60
(
4
):
335
-
337
.
28. 
Grober
ED,
Matsumoto
ED,
Jewett
MA,
Chin
JL.
The Canadian Urology Fair: a model for minimizing the financial and academic costs of the residency selection process
.
Can J Surg
.
2003
;
46
(
6
):
458
-
462
.
29. 
Hamel
P,
Boisjoly
H,
Corriveau
C,
et al.
Using the CanMEDS roles when interviewing for an ophthalmology residency program
.
Can J Ophthalmol
.
2007
;
42
(
2
):
299
-
304
.
30. 
Hauge
LS,
Stroessner
SJ,
Chowdhry
S,
Wool
NL.
Evaluating resident candidates: does closed file review impact faculty ratings?
Am J Surg
.
2007
;
193
(
6
):
761
-
765
.
31. 
Hern
HG
Wills
CP,
Johnson
B.
Change to an informal interview dress code improves residency applicant perceptions
.
West J Emerg Med
.
2015
;
16
(
1
):
127
-
132
.
32. 
Isaksen
JH,
Hertel
NT,
Kjær
NK.
Semi-structured interview is a reliable and feasible tool for selection of doctors for general practice specialist training
.
Dan Med J
.
2013
;
60
(
9
):
A4692
.
33. 
Jacobs
JC,
Guralnick
ML,
Sandlow
JI,
et al.
Senior medical student opinions regarding the ideal urology interview day
.
J Surg Educ
.
2014
;
71
(
6
):
878
-
882
.
34. 
Kamboj
AK,
Chandrasekhara
V,
Simonetto
DA,
Raffals
LE.
How we did it: virtual interviews with an eye toward the future
.
Am J Gastroenterol
.
2021
;
116
(
10
):
1972
-
1975
.
35. 
Kenigsberg
AP,
Khouri
RK
Kuprasertkul
A,
Wong
D,
Ganesan
V,
Lemack
GE.
Urology residency applications in the COVID-19 era
.
Urology
.
2020
;
143
:
55
-
61
.
36. 
Lubarsky
S,
Young
M.
Assessing the multiple mini interview (MMI) for use in selecting applicants to a neurology residency program
.
Neurology
.
2013
;
80
(
suppl 7
):
P04.253
.
37. 
Moran
SK,
Nguyen
JK,
Grimm
LJ,
et al.
Should radiology residency interviews remain virtual? Results of a multi-institutional survey inform the debate
.
Acad Radiol
.
2022
;
29
(
10
):
1595
-
1607
.
38. 
Ogunyemi
D,
Alexander
C,
Tangchitnob
E,
Kim
DS.
Mini surgical simulation, role play, and group and behavioral interviews in resident selection
.
J Grad Med Educ
.
2016
;
8
(
3
):
410
-
416
.
39. 
Rajesh
A,
Asaad
M,
Elmorsi
R,
Ferry
AM,
Maricevich
RS.
The virtual interview experience for Match 2021: a pilot survey of general surgery residency program directors
.
Am Surg
.
2023
;
89
(
4
):
720
-
725
.
40. 
Seifi
A,
Mirahmadizadeh
A,
Eslami
V.
Perception of medical students and residents about virtual interviews for residency applications in the United States
.
PloS One
.
2020
;
15
(
8
):
e0238239
.
41. 
Shah
SK,
Arora
S,
Skipper
B,
Kalishman
S,
Timm
TC,
Smith
AY.
Randomized evaluation of a web based interview process for urology resident selection
.
J Urol
.
2012
;
187
(
4
):
1380
-
1384
.
42. 
Shipper
ES,
Forrester
J,
Lau
JN,
Melcher
ML.
Evaluating the impact of blinded vs non-blinded interviews on the general surgery resident selection process
.
J Am Coll Surg
.
2017
;
225
(
4 suppl 1
):
174
-
175
.
43. 
Sklar
MC,
Eskander
A,
Dore
K,
Witterick
IJ.
Comparing the traditional and multiple mini interviews in the selection of post-graduate medical trainees
.
Can Med Educ J
.
2015
;
6
(
2
):
e6
-
e13
.
44. 
Swanson
WS,
Harris
MC,
Master
C,
Gallagher
PR,
Mauro
AE,
Ludwig
S.
The impact of the interview in pediatric residency selection
.
Ambul Pediatr
.
2005
;
5
(
4
):
216
-
220
.
45. 
Vadi
MG,
Malkin
MR,
Lenart
J,
Stier
GR,
Gatling
JW,
Applegate
RL
2nd. Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study
.
Int J Med Educ
.
2016
;
7
:
102
-
108
.
46. 
VanOrder
T,
Robbins
W,
Zemper
E.
Residency program directors’ interview methods and satisfaction with resident selection across multiple specialties
.
J Am Osteopath Assoc
.
2017
;
117
(
4
):
226
-
232
.
47. 
Vermeulen
MI,
Kuyvenhoven
MM,
Zuithoff
NP,
van der Graaf
Y,
Damoiseaux
RA.
Dutch postgraduate GP selection procedure; reliability of interview assessments
.
BMC Fam Pract
.
2013
;
14
:
43
.
48. 
Yamada
T,
Hiraoka
E,
Takemura
Y.
Predictive validity of past-behavioral versus situational questions in postgraduate medical admissions
.
Acad Med
.
2021
;
96
(
2
):
164
-
165
.
49. 
Yamada
T,
Sato
J,
Yoshimura
H,
et al.
Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission
.
BMC Med Educ
.
2017
;
17
(
1
):
57
.
50. 
Yoshimura
H,
Kitazono
H,
Fujitani
S,
et al.
Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison
.
BMC Med Educ
.
2015
;
15
:
75
.
51. 
Association of American Medical Colleges
.
Interviews in GME: where do we go from here?
52. 
Tiller
D,
O’Mara
D,
Rothnie
I,
Dunn
S,
Lee
L,
Roberts
C.
Internet-based multiple mini-interviews for candidate selection for graduate entry programmes
.
Med Educ
.
2013
;
47
(
8
):
801
-
810
.
53. 
Kelly
ME,
Dowell
J,
Husbands
A,
et al.
The fairness, predictive validity and acceptability of multiple mini interview in an internationally diverse student population—a mixed methods study
.
BMC Med Educ
.
2014
;
14
:
267
.
54. 
Kumar
K,
Roberts
C,
Rothnie
I,
du Fresne
C,
Walton
M.
Experiences of the multiple mini-interview: a qualitative analysis
.
Med Educ
.
2009
;
43
(
4
):
360
-
367
.
55. 
Jerant
A,
Griffin
E,
Rainwater
J,
et al.
Does applicant personality influence multiple mini-interview performance and medical school acceptance offers?
Acad Med
.
2012
;
87
(
9
):
1250
-
1259
.
56. 
Reiter
HI,
Lockyer
J,
Ziola
B,
Courneya
CA,
Eva
K.
Should efforts in favor of medical student diversity be focused during admissions or farther upstream?
Acad Med
.
2012
;
87
(
4
):
443
-
448
.
57. 
Levine
RB,
Hebert
RS,
Wright
SM.
Resident research and scholarly activity in internal medicine residency training programs
.
J Gen Intern Med
.
2005
;
20
(
2
):
155
-
159
.
58. 
Krumpal
I.
Determinants of social desirability bias in sensitive surveys: a literature review
.
Qual Quant
.
2013
;
47
(
4
):
2025
-
2047
.
59. 
Kelley
K,
Clark
B,
Brown
V,
Sitzia
J.
Good practice in the conduct and reporting of survey research
.
Int J Qual Health Care
.
2003
;
15
(
3
):
261
-
266
.
60. 
Presser
S,
Couper
MP,
Lessler
JT,
et al.
Methods for testing and evaluating survey questions. In:
Presser
S,
Rothgeb
JM,
Couper
MP,
et al
, eds.
Methods for Testing and Evaluating Survey Questionnaires. Wiley & Sons
;
2004
:
1
-
22
.
61. 
Rudiawarni
FA,
Narsa
IM,
Tjahjadi
B.
Are emotions exacerbating the recency bias?: An experimental study
.
Int J Trade Glob Mark
.
2020
;
13
(
1
):
61
-
70
.

The online supplementary data contains the literature search strategy used in the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: Paul B. Greenberg, MD, MPH, is a former ex-officio member of the Accreditation Council for Graduate Medical Education Medically Underserved Areas and Population Advisory Group (MUA/P) in 2022-2023 and received accommodations for the MUA/P meeting. Dr Greenberg is also a council member for the National Board of Medical Examiners (NBME) and received accommodations and meals for the NBME annual meeting.

Disclaimer: The views expressed here are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs or the US government.

Supplementary data