Background

Although the resident candidate interview is costly and time-consuming for both applicants and programs, it is considered critically important for resident selection. Noncognitive attributes, including communication skills and professionalism, can be assessed by the personal interview.

Objective

We conducted a review of the literature on the residency interview to identify the interview characteristics used for resident selection and to ascertain to what extent the interview yields information that predicts future performance.

Methods

We searched PubMed and Scopus using the following search terms: residency, internship, interview, selection, and performance. We extracted information on characteristics of the interview process, including type of interview format, measures taken to minimize bias by interviewers, and testing of other clinical/surgical skills.

Results

We identified 104 studies that pertained to the resident selection interview, with highly varied interview formats and assessment tools. A positive correlation was demonstrated between a medical school academic record and the interview, especially for unblinded interview formats. A total of 34 studies attempted to correlate interview score with performance in residency, with mixed results. We also identified a number of studies that included personality testing, clinical skills testing, or surgical skills testing.

Conclusions

Our review identified a wide variety of approaches to the selection interview and a range of factors that have been studied to assess its effectiveness. More research needs to be done not only to address and ascertain appropriate interview formats that predict positive performance in residency, but also to determine interview factors that can predict both residents' “success” and program attrition.

Editor's Note: The online version of this article contains a list of the data extracted for each article.

The year 2014 marked a record high of 26 678 postgraduate year (PGY)-1 positions offered in the United States. To achieve a Match rate of 96%, 17 374 US medical school seniors ranked a median of 11.5 programs, and often ranked more than 1 specialty.1  Residency programs screen large numbers of applications for a limited number of interview slots, and the residency selection process creates a significant expense for both applicants and programs. A recent survey of plastic surgery applicants found that interview costs ranged from $2,500 to $10,000,2  excluding the academic “cost” of lost days of medical education. The cost to residency programs is multifactorial, with the mean cost of recruiting 1 PGY-1 position estimated at $9,899.3 

The screening, interview, and ranking processes are critical, as applicant selection has enduring consequences for the programs. Ideally, an applicant should be a good “fit” for the program, with a high likelihood of success and a low likelihood of problems. Poor performance in residency may require remediation, and resident attrition has been reported to be as high as 22% to 27% in general surgery programs,46  which can create significant workflow issues, reduce morale, and have a negative impact on future recruitment.

Academic qualifications of candidates are accessible via the electronic residency application service (ERAS) and include United States Medical Licensing Examination (USMLE) scores, clerkship and preclinical grades, and class rank. Although readily available, there is wide variation in grading, class rank, and academic honors among medical schools. Studies of the use of academic data in predicting future performance have produced mixed results. A recent meta-analysis of factors showed that examination-based selection strategies (eg, USMLE Step 1) had a strong positive association with in-service training examinations, whereas medical school grades had a less robust association with subjective outcomes such as resident performance evaluations.7  USMLE Part 1 may even have a negative correlation with future clinical performance and professionalism.8  Attributes such as leadership and professionalism are not assessable through standardized testing, but may be evident in a review of the curriculum vitae, personal statement, dean's letter, and letters of recommendation. A combination of cognitive and noncognitive factors was shown in 1979 by Keck et al9  to be the most predictive of postgraduate clinical performance compared with any individual variable, and it is unlikely that this paradigm has changed 36 years later.

The interview process allows for assessment of noncognitive factors, such as interpersonal and communication skills, maturity, interest in the field, dependability, and honesty.10,11  Program directors reported that the interview is the most important factor in determining resident selection.1120  The interview may also identify negative applicant characteristics such as anxiety or aggression.21  Candidates value the interview process highly,22,23  and cite the quality of interactions with the program director24  and, most important, with the residents2527  during the interview as important in their decision making. Applicants value the interaction with residents during an interview to assess resident morale,28  esprit de corps,29  and program “red flags” as perceived by applicants.30  Ultimately, program directors and residents have reported that their “gut feeling” is important in creating their respective rank lists.29,31 

The resident selection interview also has been criticized for its “dubious value,”32  due to the lack of a standardized approach and low interrater reliability.27  During the interview, there is a strong potential for a “halo effect,” in which interviewers' prior knowledge about an applicant's academic record (grades, test scores) affect the outcome.33  Additionally, the interview can be a venue for unethical questioning regarding applicants' marital status, reproductive plans, and health, especially by faculty without adequate training.3436 

Given the high cost of the resident interview and its importance in resident selection, this review attempts to identify data-driven strategies to optimize resident interview processes. The objectives were (1) to identify interview characteristics utilized by residency programs to evaluate candidates for selection; (2) to establish a relationship between the interview score and applicant characteristics and rank-order position in blinded and unblinded interview formats; and (3) to examine associations between applicant interviews and trainee/physician performance.

Identifying and reviewing articles that met inclusion criteria involved 2 phases. During the first phase, we conducted a pilot search of articles to develop a data abstraction tool. This initial search yielded 107 articles in PubMed, using medical subject heading (MeSH) terms such as internship and residency, interviews as topic/methods, and personnel selection covering the period between 1966 and 2013. The tool was developed by a research team member (S.K.), with 1 revision after the pilot phase. A kappa value of 0.80 (for accuracy of data abstraction) was attained across 3 research team members (S.K., S.O., A.M.) who reviewed 5 articles. Subsequently, the team reviewed all 30 articles and completed the data abstraction tool with the addition of 2 added researchers (B.H., S.C.). A list of the data extracted for each article is provided as online supplemental material.

The second review phase involved an expanded search of articles with consultation from a health sciences librarian. A search for English-language articles from 1966 through October 2014 in PubMed used the following terms: (1) internship and residency (MeSH terms), (2) interview, and (3) selection or performance. The search strategy was repeated in Scopus, and we also reviewed references of identified articles. Figure 1 illustrates the article selection process. A full-text review of 264 articles was performed, which included the initial 30 studies. We excluded review articles, surveys of program directors or applicants, letters to the editor, and commentaries. Forty-seven additional articles focused on other aspects of resident selection and did not include data on the residency interview. A total of 104 articles contained program-level data on the interview processes for resident selection or as a predictor of future performance. The lead author (A.S-F.) completed the review of all 104 articles using the review process established during phase 1. Data analysis was performed using Stata version 13 (StataCorp LP) to generate descriptive statistics.

For the type of interview structure, we defined traditional or unstructured interviewing as the use of academic criteria and curricula vitae to generate questions that the interviewer determined to be relevant to the applicants' credentials. In contrast, structured interviews used predefined questions established by the selection committee prior to the interview and posed to all applicants. Behavioral interviews (also referred to as accomplishment interviews) are a type of structured interview in which applicants are asked to describe a past experience to provide an example of when they demonstrated characteristics desirable to the residency program. Multiple mini interviews include a timed circuit of stations to assess skills, including interpersonal communication, professionalism, and ethics.

Interview Characteristics

Of the 104 articles, 91 (88%) were from the United States or Canada; the remainder were from the United Kingdom (n = 8), Australia, Ireland, Thailand, and the United Arab Emirates. The specialties studied in the reviewed articles are shown in figure 2. Table 1 reports the key results of studies. Studies were predominantly single institution studies. Less than half (46%, 48 of 104) of the articles reported the interview format. Of the studies that provided data on the interview format, 54% (26 of 48) described using web- or video-based skills testing or multiple mini formats, 31% (15 of 48) reported a structured (including behavioral) interview, and 15% (7 of 48) used a traditional or unstructured format.

We found that most studies lacked substantial information regarding the scope and the format of the interview, the steps taken to reduce possible interview biases, and the psychometric evidence associated with the interview instrument, such as reliability and validity. Only one-third of studies tracked resident performance over time. The type and number of interviewers were not consistently reported. For studies that included this information, in addition to physician faculty, interviewers comprised of residents (n = 20), PhD faculty (n = 2), psychologists (n = 3), or other health professionals (n = 4).

Correlational Studies

Studies frequently reported on the relationship of the interview score with academic criteria (USMLE, transcript, class rank, Alpha Omega Alpha Honor Medical Society, research publications), final rank, or match results. The cognitive and noncognitive criteria most commonly cited in studies as components of the final rank are shown in table 2. Many studies demonstrated a positive correlation among interview score, academic data, and final rank.32,3743  One study found that the interview could so closely correlate with academic variables that the R2 values for final rank and preinterview academic rank were unchanged whether the interview score was added to the model or not.44  Three studies showed that the interview significantly changed the rank of some applicants, moving them both higher and lower (> 10 positions) than their preinterview rank.37,38,45  In a study using an interview format based on the CanMEDS competency-based framework (Communicator, Collaborator, Manager, Professional, Health Advocate, Medical Expert, Scholar), poor correlation was found between the interview and academic record, suggesting the interview was measuring factors other than academic performance.46 

Blinding Interviewers

Studies that evaluated the effect of blinding of interviewers to reduce bias consistently showed that an open or unblinded file resulted in higher interviewer ratings.33,42,4749  Unblinded interviews correlated more closely to USMLE Step 1 scores,33,42,48  whereas 2 studies where interviewers were blinded to the applicant's academic record found a negative correlation with USMLE scores.33,42  One study found that 30% of the variance in interview scores was due to interviewer bias and concluded that unblinded interview scores were influenced significantly by other variables (USMLE scores, transcript, class rank, Alpha Omega Alpha), minimizing the weight of the interview in resident selection.47 

Performance

Thirty-four studies attempted to evaluate whether the interview predicted performance. Outcomes studied included (1) clinical evaluations during residency by program directors, faculty, and/or chief residents; (2) global evaluation or ranking of residents; (3) in-training examination results; (4) attrition; or (5) problems with professionalism in residency or subsequently in their careers. The duration of follow-up ranged from 7 to 12 months for those outcomes that were assessed during the course of internship,5053  and from 36 to 60 months for outcomes in residency. Results from these studies varied. Of the 34 studies (table 3), 17 showed that the interview did not predict subsequent clinical performance (weak positive, negative, or no correlation) in internship or residency, particularly with a traditional or unstructured interview format.41,50,5367  Eleven studies showed a positive correlation between interview and subsequent performance.8,51,52,6875  Performance metrics with a positive correlation included clinical evaluations, in-training examinations, licensing board examinations, and a composite score or rank of resident performance. Attrition from residency was not consistently predicted by the interview process in the 6 studies that assessed this variable.46,43,55,56  In a case control study of psychiatry residents, minor or major problems during residency may have only been associated with negative comments in the dean's letter and were not picked up in the interview.76  A longitudinal study from 1965 to 1999 showed that neither the residency interview nor the performance evaluations in a psychiatry program predicted long-term professionalism issues, resulting in referral to an impaired physician program.77 

The behavioral or accomplishment interview was more predictive of subsequent residency performance evaluation than the traditional interview or other academic variables.54  Although 1 study did not find a correlation between the overall behavioral interview score and resident performance, a subscore for suitability/trainability was negatively associated with attrition.55  In another study, components of the accomplishment interview correlated with specific resident behaviors such as confidence with interpersonal skills (r = 0.38) or recognition of personal limits (r = −0.46).75 

Clinical and Surgical Skills

The 7 studies that included an assessment of surgical skills as part of the interview or selection process showed mixed results.7884  A soap carving task as part of an otolaryngology residency interview was not predictive of cognitive knowledge, visuospatial ability, manual dexterity, decision making, or overall resident performance.82  The use of surgical skills testing at an otolaryngology residency interview did not correlate with USMLE Step 1 scores,78  although in a larger cohort, it was predictive of faculty performance evaluation.79 

Assessment of clinical skills in the form of an objective structured clinical examination or a multiple mini interview (MMI) to assess competencies desired by the program has been described for individual programs and regional assessment centers for selection of residents. Many of the studies examining validity evidence for the MMIs or multiple station interviews were identified in this review (MMI70,8591  and competency center71,92–98). Several studies showed a positive correlation between multiple station interviews and future clinical examination testing.70,71,95  A study of the use of the MMI for emergency medicine interns at 3 institutions found that the results did not correlate with clerkship grades, medical school quality, or USMLE scores, but did correlate with emergency medicine clerkship grades. This suggests that the MMI may measure abilities that contribute to success in the specialty.91 

Personality Testing

Eight studies explored the use of personality99103  or emotional intelligence104106  testing as part of resident selection. Use of the Myers-Briggs personality inventory for interviewers and applicants demonstrated that clinician faculty ranked candidates more favorably when they shared certain personality styles.103  Bohm et al106  evaluated a moral reasoning assessment tool in orthopedic resident applicants and found no association between moral reasoning and resident rank or USMLE scores.

Our comprehensive review shows the complexity of the interview process in resident selection. The scope of this review, which included a broad range of medical and surgical specialties, provides an overview of the diverse characteristics of the interview in historical and contemporary resident selection. It suggests that the interview process varies greatly among programs and specialties, with regard to the assessment of interview format, logistics, characteristics of interviewers, and noncognitive skills.

Although 34 studies attempted to establish a relationship between the interview and future performance, the results are mixed as to whether the interview itself, or the interview combined with other data used in the application, predicted future “success” or “problems” for residency applicants. It was not possible to determine whether a specific interview format is superior for predicting performance.

It would be informative if selection strategies could predict subsequent resident performance, but quality data are limited and mixed. This is partly due to the lack of useful performance outcomes and limited power in small studies. Thus, longitudinal and multi-institution studies of performance during and after residency with clinically meaningful performance measures are needed to fully assess the predictability of various interview techniques or academic records. With improved specialty-specific trainee assessment tools, program directors may have a common language to assess overall performance and design interview processes that could predict more long-term physician success.

Gordon and Lincoln27  criticized the traditional interview for poor inter-interviewer reliability, content that varies from applicant to applicant, interviewer bias from stereotypes or idiosyncratic rapport, and the fact that questions may be anticipated or rehearsed by the applicant, resulting in a skewed view of an applicant's values and motivation. The box shows a series of recommendations to improve the reliability and value of the resident interview as a selection tool.

box Attributes of the Resident Interview That Improve Reliability
  • 1

    Explicit written description of the desired traits in an applicant/resident

  • 2

    Standardized questions to every applicant

  • 3

    Provision of behavior-specific anchors for rating scales for interviewers and using a scoring rubric to improve interrater and intrarater scoring

  • 4

    Use of multiple observers rather than a single interviewer

  • 5

    Training of interviewers in the format and scoring and including unethical and “illegal” question rules

  • 6

    Blinding of the interviewer to cognitive application data to minimize bias107 

In the future, if the regional interview fair108  or web-based conference interview109,110  has additional evidence of validity, this may diminish the expense of interviewing to both programs and applicants. The major drawback will be the lack of interactions with resident and staff that an onsite interview provides.

This review has 2 limitations. The first is the heterogeneity of the source articles, which do not consistently report many of the variables of interest to the authors, limiting the ability to make interpretation of findings across studies. Additionally, a single author reviewed all included studies, which could have introduced bias in the data abstraction and conclusions raised in this study.

The interview in resident selection often is used to assess noncognitive factors and can provide information to assess the mutual fit of the applicant and the program. Ideally, the interview contributes to the final rank list by systematically assessing specialty-specific personal qualities, skills, and competencies. However, the selection interview does not predict clinical performance, problems with professionalism, or resident attrition. The predictive value of the interview may be limited due to “halo effects,” as interviewers base their assessment of candidates on academic criteria, especially in unblinded formats. Although there is insufficient evidence to recommend an optimal interview format to predict future performance, use of the unstructured, unblinded interview should be replaced with a more rigorous interview strategy.

1
National Resident Matching Program
.
Charting outcomes in the match
.
2014
.
5th ed
. .
2
Wood
JS
,
David
LR.
Outcome analysis of factors impacting the plastic surgery match
.
Ann Plast Surg
.
2010
;
64
(
6
):
770
774
.
3
Brummond
A
,
Sefcik
S
,
Halvorsen
AJ
,
Chaudhry
S
,
Arora
V
,
Adams
M
,
et al
.
Resident recruitment costs: a national survey of internal medicine program directors
.
Am J Med
.
2013
;
126
(
7
):
646
653
.
4
Alterman
DM
,
Jones
TM
,
Heidel
RE
,
Daley
BJ
,
Goldman
MH.
The predictive value of general surgery application data for future resident performance
.
J Surg Educ
.
2011
;
68
(
6
):
513
518
.
5
Kelz
RR
,
Mullen
JL
,
Kaiser
LR
,
Pray
LA
,
Shea
GP
,
Drebin
JA
,
et al
.
Prevention of surgical resident attrition by a novel selection strategy
.
Ann Surg
.
2010
;
252
(
3
):
537
541
;
discussion 541–543
.
6
Naylor
RA
,
Reisch
JS
,
Valentine
RJ.
Factors related to attrition in surgery residency based on application data
.
Arch Surg
.
2008
;
143
(
7
):
647
651
;
discussion 651–652
.
7
Kenny
S
,
McInnes
M
,
Singh
V.
Associations between residency selection strategies and doctor performance: a meta-analysis
.
Med Educ
.
2013
;
47
(
8
):
790
800
.
8
Brothers
TE
,
Wetherholt
S.
Importance of the faculty interview during the resident application process
.
J Surg Educ
.
2007
;
64
(
6
):
378
385
.
9
Keck
JW
,
Arnold
L
,
Willoughby
L
,
Calkins
V.
Efficacy of cognitive/noncognitive measures in predicting resident-physician performance
.
J Med Educ
.
1979
;
54
(
10
):
759
765
.
10
LaGrasso
JR
,
Kennedy
DA
,
Hoehn
JG
,
Ashruf
S
,
Przybyla
AM.
Selection criteria for the integrated model of plastic surgery residency
.
Plast Reconstr Surg
.
2008
;
121
(
3
):
121e
125e
.
11
Wagoner
NE
,
Suriano
JR
,
Stoner
JA.
Factors used by program directors to select residents
.
J Med Educ
.
1986
;
61
(
1
):
10
21
.
12
Wagoner
NE
,
Gray
GT.
Report on a survey of program directors regarding selection factors in graduate medical education
.
J Med Educ
.
1979
;
54
(
6
):
445
452
.
13
Wagoner
NE
,
Suriano
JR.
Program directors' responses to a survey on variables used to select residents in a time of change
.
Acad Med
.
1999
;
74
(
1
):
51
58
.
14
Melendez
MM
,
Xu
X
,
Sexton
TR
,
Shapiro
MJ
,
Mohan
EP.
The importance of basic science and clinical research as a selection criterion for general surgery residency programs
.
J Surg Educ
.
2008
;
65
(
2
):
151
154
.
15
Hirthler
MA
,
Glick
PL
,
Hassett
JM
Jr,
Cooney
DR.
Evaluation of the pediatric surgical matching program by the directors of pediatric surgical training programs
.
J Pediatr Surg
.
1994
;
29
(
10
):
1370
1374
.
16
Makdisi
G
,
Takeuchi
T
,
Rodriguez
J
,
Rucinski
J
,
Wise
L.
How we select our residents—a survey of selection criteria in general surgery residents
.
J Surg Educ
.
2011
;
68
(
1
):
67
72
.
17
DeLisa
JA
,
Jain
SS
,
Campagnolo
DI.
Factors used by physical medicine and rehabilitation residency training directors to select their residents
.
Am J Phys Med Rehabil
.
1994
;
73
(
3
):
152
156
.
18
Galazka
SS
,
Kikano
GE
,
Zyzanski
S.
Methods of recruiting and selecting residents for US family practice residencies
.
Acad Med
.
1994
;
69
(
4
):
304
306
.
19
Nallasamy
S
,
Uhler
T
,
Nallasamy
N
,
Tapino
PJ
,
Volpe
NJ.
Ophthalmology resident selection: current trends in selection criteria and improving the process
.
Ophthalmology
.
2010
;
117
(
5
):
1041
1047
.
20
Taylor
CA
,
Weinstein
L
,
Mayhew
HE.
The process of resident selection: a view from the residency director's desk
.
Obstet Gynecol
.
1995
;
85
(
2
):
299
303
.
21
Al Khalili
K
,
Chalouhi
N
,
Tjoumakaris
S
,
Gonzalez
LF
,
Starke
RM
,
Rosenwasser
R
,
et al
.
Programs selection criteria for neurological surgery applicants in the United States: a national survey for neurological surgery program directors
.
World Neurosurg
.
2014
;
81
(
3–4
):
473
477.e2
.
22
Puscas
L
,
Sharp
SR
,
Schwab
B
,
Lee
WT.
Qualities of residency applicants: comparison of otolaryngology program criteria with applicant expectations
.
Arch Otolaryngol Head Neck Surg
.
2012
;
138
(
1
):
10
14
.
23
Simmonds
AC
,
Robbins
JM
,
Brinker
MR
,
Rice
JC
,
Kerstein
MD.
Factors important to students in selecting a residency program
.
Acad Med
.
1990
;
65
(
10
):
640
643
.
24
Deiorio
NM
,
Yarris
LM
,
Gaines
SA.
Emergency medicine residency applicant views on the interview day process
.
Acad Emerg Med
.
2009
;
16
(
suppl 2
):
67
70
.
25
Schlitzkus
LL
,
Schenarts
PJ
,
Schenarts
KD.
It was the night before the interview: perceptions of resident applicants about the preinterview reception
.
J Surg Educ
.
2013
;
70
(
6
):
750
757
.
26
Love
JN
,
Howell
JM
,
Hegarty
CB
,
McLaughlin
SA
,
Coates
WC
,
Hopson
LR
,
et al
.
Factors that influence medical student selection of an emergency medicine residency program: implications for training programs
.
Acad Emerg Med
.
2012
;
19
(
4
):
455
460
.
27
Gordon
MJ
,
Lincoln
JA.
Family practice resident selection: value of the interview
.
J Fam Pract
.
1976
;
3
(
2
):
175
177
.
28
Aagaard
EM
,
Julian
K
,
Dedier
J
,
Soloman
I
,
Tillisch
J
,
Perez-Stable
EJ.
Factors affecting medical students' selection of an internal medicine residency program
.
J Natl Med Assoc
.
2005
;
97
(
9
):
1264
1270
.
29
Parker
AM
,
Petroze
RT
,
Schirmer
BD
,
Calland
JF.
Surgical residency market research—what are applicants looking for?
J Surg Educ
.
2013
;
70
(
2
):
232
236
.
30
Yousuf
SJ
,
Kwagyan
J
,
Jones
LS.
Applicants' choice of an ophthalmology residency program
.
Ophthalmology
.
2013
;
120
(
2
):
423
427
.
31
Otero
HJ
,
Erturk
SM
,
Ondategui-Parra
S
,
Ros
PR.
Key criteria for selection of radiology residents: results of a national survey
.
Acad Radiol
.
2006
;
13
(
9
):
1155
1164
.
32
Prager
JD
,
Myer
CM
4th,
Hayes
KM
,
Myer
CM
3rd,
Pensak
ML.
Improving methods of resident selection
.
Laryngoscope
.
2010
;
120
(
12
):
2391
2398
.
33
Smilen
SW
,
Funai
EF
,
Bianco
AT.
Residency selection: should interviewers be given applicants' board scores?
Am J Obstet Gynecol
.
2001
;
184
(
3
):
508
513
.
34
Slone
RM.
Resident selection: Part 3. The interview
.
Invest Radiol
.
1991
;
26
(
4
):
396
399
.
35
Hern
HG
Jr,
Alter
HJ
,
Wills
CP
,
Snoey
ER
,
Simon
BC.
How prevalent are potentially illegal questions during residency interviews?
Acad Med
.
2013
;
88
(
8
):
1116
1121
.
36
Thurman
RJ
,
Katz
E
,
Carter
W
,
Han
J
,
Kayala
E
,
McCoin
N
,
et al
.
Emergency medicine residency applicant perceptions of unethical recruiting practices and illegal questioning in the match
.
Acad Emerg Med
.
2009
;
16
(
6
):
550
557
.
37
Christakis
PG
,
Christakis
TJ
,
Dziura
J
,
Christakis
JT.
Role of the interview in admissions at the University of Toronto ophthalmology program
.
Can J Ophthalmol
.
2010
;
45
(
5
):
527
530
.
38
Collins
M
,
Curtis
A
,
Artis
K
,
Staib
L
,
Bokhari
J.
Comparison of two methods for ranking applicants for residency
.
J Am Coll Radiol
.
2010
;
7
(
12
):
961
966
.
39
Curtis
DJ
,
Riordan
DD
,
Cruess
DF
,
Brower
AC.
Selecting radiology resident candidates
.
Invest Radiol
.
1989
;
24
(
4
):
324
330
.
40
Davis
JL
,
Platt
LD
,
Sandhu
M
,
Shapiro
F.
Evaluating factors in the selection of residents
.
Acad Med
.
1995
;
70
(
3
):
176
177
.
41
Kandler
H
,
Plutchik
R
,
Conte
H
,
Siegel
B.
Prediction of performance of psychiatric residents: a three-year follow-up study
.
Am J Psychiatry
.
1975
;
132
(
12
):
1286
1290
.
42
Miles
WS
,
Shaw
V
,
Risucci
D.
The role of blinded interviews in the assessment of surgical residency candidates
.
Am J Surg
.
2001
;
182
(
2
):
143
146
.
43
Scott-Conner
CEH
,
Rhodes
RS
,
Aggarwal
AK
,
Bass
J.
Selecting residents for a surgical residency: Are we doing what we say we are doing?
Current Surgery
.
1997
;
54
(
1
):
42
46
.
44
Young
MJ
,
Woolliscroft
JO
,
Holloway
JJ.
Determining the policies of a residency selection committee
.
J Med Educ
.
1986
;
61
(
10
):
835
837
.
45
Gong
H
Jr,
Parker
NH
,
Apgar
FA
,
Shank
C.
Influence of the interview on ranking in the residency selection process
.
Med Educ
.
1984
;
18
(
5
):
366
369
.
46
Hamel
P
,
Boisjoly
H
,
Corriveau
C
,
Fallaha
N
,
Lahoud
S
,
Luneau
K
,
et al
.
Using the CanMEDS roles when interviewing for an ophthalmology residency program
.
Can J Ophthalmol
.
2007
;
42
(
2
):
299
304
.
47
Robin
AP
,
Bombeck
CT
,
Pollak
R
,
Nyhus
LM.
Introduction of bias in residency-candidate interviews
.
Surgery
.
1991
;
110
(
2
):
253
258
.
48
Swanson
DB
,
Sawhill
A
,
Holtzman
KZ
,
Bucak
SD
,
Morrison
C
,
Hurwitz
S
,
et al
.
Relationship between performance on part I of the American Board of Orthopaedic Surgery certifying examination and scores on USMLE steps 1 and 2
.
Acad Med
.
2009
;
84
(
suppl 10
):
21
24
.
49
Hauge
LS
,
Stroessner
SJ
,
Chowdhry
S
,
Wool
NL
;
Association for Surgical Education
.
Evaluating resident candidates: does closed file review impact faculty ratings? Am J Surg
.
2007
;
193
(
6
):
761
765
.
50
Part
HM
,
Markert
RJ.
Predicting the first-year performances of international medical graduates in an internal medicine residency
.
Acad Med
.
1993
;
68
(
11
):
856
858
.
51
Khongphatthanayothin
A
,
Chongsrisawat
V
,
Wananukul
S
,
Sanpavat
S.
Resident recruitment: what are good predictors for performance during pediatric residency training?
J Med Assoc Thai
.
2002
;
85
(
suppl 1
):
302
311
.
52
Olawaiye
A
,
Yeh
J
,
Withiam-Leitch
M.
Resident selection process and prediction of clinical performance in an obstetrics and gynecology program
.
Teach Learn Med
.
2006
;
18
(
4
):
310
315
.
53
Komives
E
,
Weiss
ST
,
Rosa
RM.
The applicant interview as a predictor of resident performance
.
J Med Educ
.
1984
;
59
(
5
):
425
426
.
54
Altmaier
EM
,
Smith
WL
,
O'Halloran
CM
,
Franken
EA
Jr.
The predictive utility of behavior-based interviewing compared with traditional interviewing in the selection of radiology residents
.
Invest Radiol
.
1992
;
27
(
5
):
385
389
.
55
Strand
EA
,
Moore
E
,
Laube
DW.
Can a structured, behavior-based interview predict future resident success?
Am J Obstet Gynecol
.
2011
;
204
(
5
):
446.e1
446.e13
.
56
Dubovsky
SL
,
Gendel
MH
,
Dubovsky
AN
,
Levin
R
,
Rosse
J
,
House
R.
Can admissions interviews predict performance in residency?
Acad Psychiatry
.
2008
;
32
(
6
):
498
503
.
57
Bell
JG
,
Kanellitsas
I
,
Shaffer
L.
Selection of obstetrics and gynecology residents on the basis of medical school performance
.
Am J Obstet Gynecol
.
2002
;
186
(
5
):
1091
1094
.
58
Bent
JP
,
Colley
PM
,
Zahtz
GD
,
Smith
RV
,
Parikh
SR
,
Schiff
B
,
et al
.
Otolaryngology resident selection: do rank lists matter?
Otolaryngol Head Neck Surg
.
2011
;
144
(
4
):
537
541
.
59
Borowitz
SM
,
Saulsbury
FT
,
Wilson
WG.
Information collected during the residency match process does not predict clinical performance
.
Arch Pediatr Adolesc Med
.
2000
;
154
(
3
):
256
260
.
60
Chole
RA
,
Ogden
MA.
Predictors of future success in otolaryngology residency applicants
.
Arch Otolaryngol Head Neck Surg
.
2012
;
138
(
8
):
707
712
.
61
Daly
KA
,
Levine
SC
,
Adams
GL.
Predictors for resident success in otolaryngology
.
J Am Coll Surg
.
2006
;
202
(
4
):
649
654
.
62
Dawkins
K
,
Ekstrom
RD
,
Maltbie
A
,
Golden
RN.
The relationship between psychiatry residency applicant evaluations and subsequent residency performance
.
Acad Psychiatry
.
2005
;
29
(
1
):
69
75
.
63
Oldfield
Z
,
Beasley
SW
,
Smith
J
,
Anthony
A
,
Watt
A.
Correlation of selection scores with subsequent assessment scores during surgical training
.
ANZ J Surg
.
2013
;
83
(
6
):
412
416
.
64
Metro
DG
,
Talarico
JF
,
Patel
RM
,
Wetmore
AL.
The resident application process and its correlation to future performance as a resident
.
Anesth Analg
.
2005
;
100
(
2
):
502
505
.
65
McGreevy
JM
,
Kollmorgen
DR.
Medical school performance does not predict success in a surgery residency
.
Curr Surg
.
1997
;
54
(
7
):
438
441
.
66
Fryer
JP
,
Corcoran
N
,
George
B
,
Wang
E
,
Darosa
D.
Does resident ranking during recruitment accurately predict subsequent performance as a surgical resident?
J Surg Educ
.
2012
;
69
(
6
):
724
730
.
67
George
JM
,
Young
D
,
Metz
EN.
Evaluating selected internship candidates and their subsequent performances
.
Acad Med
.
1989
;
64
(
8
):
480
482
.
68
Geissler
J
,
VanHeest
A
,
Tatman
P
,
Gioe
T.
Aggregate Interview Method of ranking orthopedic applicants predicts future performance
.
Orthopedics
.
2013
;
36
(
7
):
e966
e970
.
69
Grewal
SG
,
Yeung
LS
,
Brandes
SB.
Predictors of success in a urology residency program
.
J Surg Educ
.
2013
;
70
(
1
):
138
143
.
70
Eva
KW
,
Reiter
HI
,
Trinh
K
,
Wasi
P
,
Rosenfeld
J
,
Norman
GR.
Predictive validity of the multiple mini-interview for selecting medical trainees
.
Med Educ
.
2009
;
43
(
8
):
767
775
.
71
Thordarson
DB
,
Ebramzadeh
E
,
Sangiorgio
SN
,
Schnall
SB
,
Patzakis
MJ.
Resident selection: how we are doing and why?
Clin Orthop Relat Res
.
2007
(
459
):
255
259
.
72
Ozuah
PO.
Predicting residents' performance: a prospective study
.
BMC Med Educ
.
2002
;
2
:
7
.
73
Shiroma
PR
,
Alarcon
RD.
Selection factors among international medical graduates and psychiatric residency performance
.
Acad Psychiatry
.
2010
;
34
(
2
):
128
131
.
74
Warrick
SS
,
Crumrine
RS.
Predictors of success in an anesthesiology residency
.
J Med Educ
.
1986
;
61
(
7
):
591
595
.
75
Wood
PS
,
Smith
WL
,
Altmaier
EM
,
Tarico
VS
,
Franken
EA
Jr.
A prospective study of cognitive and noncognitive selection criteria as predictors of resident performance
.
Invest Radiol
.
1990
;
25
(
7
):
855
859
.
76
Brenner
AM
,
Mathai
S
,
Jain
S
,
Mohl
PC.
Can we predict “problem residents”?
Acad Med
.
2010
;
85
(
7
):
1147
1151
.
77
Dubovsky
SL
,
Gendel
M
,
Dubovsky
AN
,
Rosse
J
,
Levin
R
,
House
R.
Do data obtained from admissions interviews and resident evaluations predict later personal and practice problems?
Acad Psychiatry
.
2005
;
29
(
5
):
443
447
.
78
Carlson
ML
,
Archibald
DJ
,
Sorom
AJ
,
Moore
EJ.
Under the microscope: assessing surgical aptitude of otolaryngology residency applicants
.
Laryngoscope
.
2010
;
120
(
6
):
1109
1113
.
79
Moore
EJ
,
Price
DL
,
Van Abel
KM
,
Carlson
ML.
Still under the microscope: can a surgical aptitude test predict otolaryngology resident performance?
Laryngoscope
.
2015
;
125
(
2
):
e57
e61
.
80
Krespi
YP
,
Levine
TM
,
Einhorn
RK
,
Mitrani
M.
Surgical aptitude test for otolaryngology-head and neck surgery resident applicants
.
Laryngoscope
.
1986
;
96
(
11
):
1201
1206
.
81
Panait
L
,
Larios
JM
,
Brenes
RA
,
Francher
TT
,
Ajemian
MS
,
Dudrick
SJ
,
et al
.
Surgical skills assessment of applicants to general surgery residency
.
J Surg Res
.
2011
;
170
(
2
):
189
194
.
82
Tang
CG
,
Hilsinger
RL
Jr,
Cruz
RM
,
Schloegel
LJ
,
Byl
FM
Jr,
Rasgon
BM.
Manual dexterity aptitude testing: a soap carving study
.
JAMA Otolaryngol Head Neck Surg
.
2014
;
140
(
3
):
243
249
.
83
Gallagher
AG
,
Leonard
G
,
Traynor
OJ.
Role and feasibility of psychomotor and dexterity testing in selection for surgical training
.
ANZ J Surg
.
2009
;
79
(
3
):
108
113
.
84
Gallagher
AG
,
Neary
P
,
Gillen
P
,
Lane
B
,
Whelan
A
,
Tanner
WA
,
et al
.
Novel method for assessment and selection of trainees for higher surgical training in general surgery
.
ANZ J Surg
.
2008
;
78
(
4
):
282
290
.
85
Ahmed
A
,
Qayed
KI
,
Abdulrahman
M
,
Tavares
W
,
Rosenfeld
J.
The multiple mini-interview for selecting medical residents: first experience in the Middle East region
.
Med Teach
.
2014
;
36
(
8
):
703
709
.
86
Campagna-Vaillancourt
M
,
Manoukian
J
,
Razack
S
,
Nguyen
LH.
Acceptability and reliability of multiple mini interviews for admission to otolaryngology residency
.
Laryngoscope
.
2014
;
124
(
1
):
91
96
.
87
Dore
KL
,
Kreuger
S
,
Ladhani
M
,
Rolfson
D
,
Kurtz
D
,
Kulasegaram
K
,
et al
.
The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions
.
Acad Med
.
2010
;
85
(
suppl 10
):
60
63
.
88
Finlayson
HC
,
Townson
AF.
Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview
.
Am J Phys Med Rehabil
.
2011
;
90
(
4
):
330
335
.
89
Hofmeister
M
,
Lockyer
J
,
Crutcher
R.
The acceptability of the multiple mini interview for resident selection
.
Fam Med
.
2008
;
40
(
10
):
734
740
.
90
Hofmeister
M
,
Lockyer
J
,
Crutcher
R.
The multiple mini-interview for selection of international medical graduates into family medicine residency education
.
Med Educ
.
2009
;
43
(
6
):
573
579
.
91
Hopson
LR
,
Burkhardt
JC
,
Stansfield
RB
,
Vohra
T
,
Turner-Lawrence
D
,
Losman
ED.
The multiple mini-interview for emergency medicine resident selection
.
J Emerg Med
.
2014
;
46
(
4
):
537
543
.
92
Brown
CA
,
Wakefield
SE
,
Bullock
AD.
The selection of GP trainees in the West Midlands: audit of assessment centre scores by ethnicity and country of qualification
.
Med Teach
.
2001
;
23
(
6
):
605
609
.
93
Brown
CA
,
Wakefield
SE
,
Bullock
AD.
The selection of GP trainees in the West Midlands: second audit of assessment centre scores by ethnicity and country of qualification
.
Med Teach
.
2003
;
25
(
6
):
649
653
.
94
Onyon
C
,
Wall
D
,
Goodyear
HM.
Reliability of multi-station interviews in selection of junior doctors for specialty training
.
Med Teach
.
2009
;
31
(
7
):
665
667
.
95
Randall
R
,
Davies
H
,
Patterson
F
,
Farrell
K.
Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre
.
Arch Dis Child
.
2006
;
91
(
5
):
444
448
.
96
Randall
R
,
Stewart
P
,
Farrell
K
,
Patterson
F.
Using an assessment centre to select doctors for postgraduate training in obstetrics and gynaecology
.
Obstet Gynaecol
.
2006
;
8
(
4
):
257
262
.
97
Rao
R.
The Structured Clinically Relevant Interview for Psychiatrists in Training (SCRIPT): a new standardized assessment tool for recruitment in the UK
.
Acad Psychiatry
.
2007
;
31
(
6
):
443
446
.
98
Smith
PE
,
Dunstan
FD
,
Wiles
CM.
Selecting specialist registrars by station interview
.
Clin Med
.
2006
;
6
(
3
):
279
280
.
99
Tulsky
A
,
Aultman
JM
,
Zirwas
MJ.
Personality testing in resident selection
.
Virtual Mentor
.
2012
;
14
(
12
):
937
945
.
100
Merlo
LJ
,
Matveevskii
AS.
Personality testing may improve resident selection in anesthesiology programs
.
Med Teach
.
2009
;
31
(
12
):
e551
e554
.
101
Bell
RM
,
Fann
SA
,
Morrison
JE
,
Lisk
JR.
Determining personal talents and behavioral styles of applicants to surgical training: a new look at an old problem, part II
.
J Surg Educ
.
2012
;
69
(
1
):
23
29
.
102
Bell
RM
,
Fann
SA
,
Morrison
JE
,
Lisk
JR.
Determining personal talents and behavioral styles of applicants to surgical training: a new look at an old problem, part I
.
J Surg Educ
.
2011
;
68
(
6
):
534
541
.
103
Quintero
AJ
,
Segal
LS
,
King
TS
,
Black
KP.
The personal interview: assessing the potential for personality similarity to bias the selection of orthopaedic residents
.
Acad Med
.
2009
;
84
(
10
):
1364
1372
.
104
Lin
DT
,
Kannappan
A
,
Lau
JN.
The assessment of emotional intelligence among candidates interviewing for general surgery residency
.
J Surg Educ
.
2013
;
70
(
4
):
514
521
.
105
Talarico
JF
,
Varon
AJ
,
Banks
SE
,
Berger
JS
,
Pivalizza
EG
,
Medina-Rivera
G
,
et al
.
Emotional intelligence and the relationship to resident performance: a multi-institutional study
.
J Clin Anesth
.
2013
;
25
(
3
):
181
187
.
106
Bohm
KC
,
Van Heest
T
,
Gioe
TJ
,
Agel
J
,
Johnson
TC
,
Van Heest
A.
Assessment of moral reasoning skills in the orthopaedic surgery resident applicant
.
J Bone Joint Surg Am
.
2014
;
96
(
17
):
e151
.
107
Lee
AG
,
Golnik
KC
,
Oetting
TA
,
Beaver
HA
,
Boldt
HC
,
Olson
R
,
et al
.
Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement
.
Surv Ophthalmol
.
2008
;
53
(
2
):
164
176
.
108
Grober
ED
,
Matsumoto
ED
,
Jewett
MA
,
Chin
JL
,
Canadian Urology Program Directors. The Canadian Urology Fair: a model for minimizing the financial and academic costs of the residency selection process
.
Can J Surg
.
2003
;
46
(
6
):
458
462
.
109
Pasadhika
S
,
Altenbernd
T
,
Ober
RR
,
Harvey
EM
,
Miller
JM.
Residency interview video conferencing
.
Ophthalmology
.
2014
;
119
(
2
):
426
426.e5
.
110
Shah
SK
,
Arora
S
,
Skipper
B
,
Kalishman
S
,
Timm
TC
,
Smith
AY.
Randomized evaluation of a web based interview process for urology resident selection
.
J Urol
.
2012
;
187
(
4
):
1380
1384
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

This study was presented at the Council on Resident Education in Obstetrics and Gynecology and Association of Professors of Gynecology and Obstetrics Annual Meeting, in San Antonio, Texas, March 9–12, 2011.

Supplementary data