ABSTRACT

Background

Residency applicants feel increasing pressure to maximize their chances of successfully matching into the program of their choice, and are applying to more programs than ever before.

Objective

In this narrative review, we examined the most common and highly rated factors used to select applicants for interviews. We also examined the literature surrounding those factors to illuminate the advantages and disadvantages of using them as differentiating elements in interviewee selection.

Methods

Using the 2018 NRMP Program Director Survey as a framework, we examined the last 10 years of literature to ascertain how residency directors are using these common factors to grant residency interviews, and whether these factors are predictive of success in residency.

Results

Residency program directors identified 12 factors that contribute substantially to the decision to invite applicants for interviews. Although United States Medical Licensing Examination (USMLE) Step 1 is often used as a comparative factor, most studies do not demonstrate its predictive value for resident performance, except in the case of test failure. We also found that structured letters of recommendation from within a specialty carry increased benefit when compared with generic letters. Failing USMLE Step 1 or 2 and unprofessional behavior predicted lower performance in residency.

Conclusions

We found that the evidence basis for the factors most commonly used by residency directors is decidedly mixed in terms of predicting success in residency and beyond. Given these limitations, program directors should be skeptical of making summative decisions based on any one factor.

Introduction

All medical students wonder what is needed to receive an interview invitation to the residency program of their choice. In the last 2 decades there has been a sustained increase in the number of applications per student, probably due to attempts to ensure interview invitations.1  As application numbers continue to rise, it is essential that programs have a strong sense of which factors should be considered and why. In this narrative review, we will explore how programs use certain factors to select students for residency interviews and the evidence for the utility of these factors as predictors of success in residency and beyond. We aim to illuminate the advantages and disadvantages of using those markers as differentiating elements in selecting candidates to interview.

Methods

As a basis for this narrative review, we examined what residency directors say about the factors most commonly used to select applicants for residency interviews, using the 2018 National Resident Matching Program (NRMP) Program Director Survey.2  In answering the question, “What factors are most important for obtaining residency interviews?” we found this survey to be the most comprehensive guide available.

The factors program directors ranked in the 2018 NRMP Program Director Survey as most important for selecting applicants to interview are listed in the table. We defined factors as “most important” if at least 70% of program directors said they used them or if they had an average rating greater than 4.3.

table

Factors in Selecting Applicants to Interviewa

Factors in Selecting Applicants to Interviewa
Factors in Selecting Applicants to Interviewa

We searched PubMed, Web of Science, Ovid, and Scopus for articles published from 2008 to 2018 that contained key terms related to residency application as well as key terms for the factors identified in the NRMP Program Director Survey. Article abstracts were reviewed by the authors for pertinence to the topics at hand. Articles underwent analysis for relevance to the topic and quality of the study design. As strong evidence for these factors was sparse in some cases, we included studies that evaluated final applicant ranking as well as invitations for interviews, emphasizing those that examined the predictive power of these factors.

For context, the authors are 3 emergency physicians on faculty at Wake Forest School of Medicine with extensive experience in the areas of resident selection and student advising. N.D.H. is an assistant residency director with more than 5 years of experience, C.W.L. is a residency program director with more than 10 years of residency leadership experience, and D.E.M. has 25 years of experience ranging from clerkship director to associate dean for medical education. All authors actively participate in the annual process of selecting candidates for interviews and also advise students entering our specialty.

Results

USMLE Step 1 and 2

In addition to determining qualification for licensure, United States Medical Licensing Examination (USMLE) test results are often used in screening applicants for residency.3,4  In the surgical literature, USMLE Step 1 and Step 2 Clinical Knowledge (CK) ranking quartiles have been found to be predictive of program ranking, with higher test scores associated with higher probability of applicant ranking.5  Although Step 1 was not designed to be a primary determinant of the likelihood of success in residency, low scores on USMLE Step 1 correlate with failure on specialty in-training and certification examinations.6  USMLE Step 1 scores demonstrate a moderate correlation with performance on the specialty-specific internal medicine and emergency medicine board examinations.7,8  Some investigations have found a strong association between the scores and the success of a physician after residency,9  while most show that Step 1 performance has no correlation to resident quality or success, even if it matches in-service scores.1013 

Letters of Recommendation

Eighty-six percent of program directors cited letters of recommendation as a factor in selecting applicants to interview. Recent initiatives to standardize these documents and move away from “recommendation” to “evaluation” are gaining a foothold in multiple medical specialties, including emergency medicine, dermatology, otolaryngology, and internal medicine.14–17 These standardized letters often convey more actionable information and comparative data than a freeform letter. When assessing standardized letters, internal medicine residency directors value depth of familiarity with the applicant as well as quantitative comparison with other applicants.18 

Medical Student Performance Evaluation

The Medical Student Performance Evaluation (MSPE), often referred to as the “Dean's letter,” has long served as a rather opaque tool for evaluating applicants. In the past, studies have found that negative academic information is routinely suppressed in these documents.19  MSPEs are often not written according to the 2002 MSPE guidelines, with medical schools inconsistently providing graphic comparative data.20  They also rarely provide comparative professionalism data.21  The terminology used in the MSPE can be difficult to decode. In one study, 72 distinct key words were used to describe performance and 27 of those were assigned to the top category. The median percentage of medical students who received the top category was 24%, ranging from 1% to 60%. Ten percent of schools using key words did not provide distribution data and another 17% who used key words gave no data on how to interpret them.22  These key words, or “final adjectives,” actually have a greater impact on perceived desirability than any other aspect of the letter.23  These words usually relate to a measurement of class rank or percentile.

Personal Statement

The personal statement is the ideal place to express an applicant's personal beliefs and journey. It is also an opportunity for an applicant to demonstrate a commitment to the specialty, which program directors cite as an important factor. There is little evidence to show exactly what should or should not be included in a personal statement or how these compositions relate to future success. Medical schools are starting to develop writing seminars for the personal statement.24  Themes for personal statements can also display gender differences.25,26 

Grades in Required Clerkships

Grades in required third-year clerkships help determine class rank, and may be individually important for a specialty if represented in the core clerkships. Survey data outside of NRMP confirm the importance placed on core clerkship grades in fields such as otolaryngology.11  Variability in the grading systems and the percentage of students who achieve the highest performance levels can make these grades difficult for program directors to interpret, and the predictive value is unclear.27,28 

Failure of USMLE Step 1 and Step 2

Thirty percent of program directors stated that they would never interview a student with a Step 1 failure (an additional 58% “seldom” would) and 35% would not interview a student with a Step 2 failure. A Step 1 failure is a predictor of subsequent clerkship and National Board of Medical Examiners shelf examination grades.29,30  Students who fail Step 1 on their first attempt are less likely to match overall, but a majority will obtain a residency spot. Failure of Step 1 also predicts lower performance on the in-training examinations in internal medicine,6,31  emergency medicine,32  orthopedic surgery,33  and obstetrics and gynecology.34 

Class Ranking

Residency programs often consider class rank in the interview process. In line with the Association of American Medical Colleges' initiative on competency, many medical schools have abandoned class ranking in favor of a dichotomous determination of competence (pass/fail). No clear relationship between class rank and residency performance has been shown, and there is little evidence that a lower class rank without systemic failure portends inability to match.35 

Perceived Commitment to a Specialty (Audition Rotations)

Perceived commitment to a specialty is cited by 69% of program directors as a factor in choosing interviewees. There are 3 main ways to demonstrate this commitment in the application: by previous experience, in the personal statement, and by completing rotations at other institutions within the field, usually generating letters of recommendation. Audition rotation completion was a measured factor on the NRMP survey and did not meet the criteria for inclusion in this review. The impact of audition rotations on applicants' match potential is dependent on specialty.36,37 There is no quality evidence regarding the relationship between perceived commitment to specialty and later success.

Grades in Clerkship in Desired Specialty

Some specialties include the grade received in the clerkship in a standard letter of evaluation along with other quantitative and comparative assessments of the applicant, making that piece of data particularly useful in the evaluation process.13,14  It is important that the grade distribution be expressly stated so the reviewer can place the applicants' scores in appropriate context.

Evidence of Professionalism, Ethics, and Match Violations

Professionalism among applicants is of substantial interest to program directors because professional challenges during medical school endure into and beyond residency training.3840  Social networking sites, or more specifically the lack of professionalism displayed on them, may adversely affect a candidate's application.41  Programs and students should be familiar with the social media landscape and the privacy issues therein.42  Match violation citations likely raise professionalism concerns, leading to some degree of overlap in these factors.

Discussion

Our findings in this review demonstrate that many of the most commonly used factors are not necessarily predictive of an applicant's future performance or have scant evidence basis. The literature is more robust around factors that are easy to quantify, such as USMLE scores, and less clear about factors such as the personal statement and commitment to the specialty. The weight of the literature regarding predictive value of Step 1 performance shows that there is little to no correlation with resident quality or success, beyond a correlation with later specialty board passage. The use of the Step 1 score (other than failure) as a discriminator for whom to offer a residency interview is not well supported and its impact on selection should be markedly lessened. Evidence supports the use of structured letters of recommendation in applicant selection. Use of a structured letter has become a “best practice” employed in multiple specialties: emergency medicine, otolaryngology, dermatology, and internal medicine.

Two specific negative findings in an application appear as predictors of future poor performance: unprofessional behavior and USMLE examination failure. The finding of unprofessional behavior in medical school correlates with similar behavior in residency and beyond, as best demonstrated in the internal medicine literature. Failure of Step 1 or Step 2 is a predictor of poor performance on specialty examinations in multiple specialties.

The literature is less clear around factors such as the personal statement and commitment to the specialty. There is no evidence as to what should or should not be in a personal statement. This lack of evidence may explain a tendency toward emphasizing quantifiable, comparable factors over others. However, quantitative factors, such as USMLE scores, do not appear to predict success in residency. Thus, graduate medical education leaders should promote use of other selection factors and be cautious about using any single factor to exclude applicants from consideration.

Important limitations to this study include the low response rate to the NRMP survey (29.2% in this iteration and 39.9% in 2016). However, the remarkable degree of stability of its findings from year to year argues for some credibility that these factors are the most frequently reviewed and thus relevant.43  Additionally, separate studies, including specialty-specific studies, have affirmed that many of these factors are important to residency program directors.44,45  Current literature on this issue, including the NRMP survey, is limited to some extent by the heterogeneity of selection processes, heterogeneity of the literature itself, and difficulty in defining outcomes such as “success in residency.” With varying outcomes used to define success in residency it is difficult to determine the predictive value of individual application factors. The use of these factors, and their relative importance, seems to vary significantly by specialty, which further limits a general review.

Potential next steps include focused research on the predictive value of these factors. Several specialties have investigated the residency application process to better identify applicants who would be suited to both the specialty and the specific residency program.4649  These initiatives, cited here from general surgery, orthopedics, and family medicine, share common themes: de-emphasis of Step 1 scores, standardized evaluation letters, improvements in the MSPE to include student progress in various competencies, holistic file review, and evaluating for personal characteristics thought to be important to the specialty in question.

Conclusion

Program directors use a variety of factors to select applicants for residency. Some of these factors, such as USMLE scores and the MSPE, have little supporting evidence in predicting future applicant performance in residency. New initiatives, such as structured letters of recommendation, may provide better predictive value yet require further study. The limitations of the current evidence challenge the notion that any single factor should be used as a “red flag.”

References

References
1
Weissbart
SJ,
Kim
SJ,
Feinn
RS,
Stock
JA.
Relationship between the number of residency applications and the yearly match rate: time to start thinking about an application limit?
J Grad Med Educ
.
2015
;
7
(
1
):
81
85
. doi:.
2
National Resident Matching Program
.
Results of the 2018 NRMP Program Director Survey. Washington, DC
:
National Resident Matching Program; 2018
.
3
Green
M,
Jones P, Thomas JX Jr. Selection criteria for residency: results of a national program directors survey
.
Acad Med
.
2009
;
84
(
3
):
362
367
. doi:.
4
Schrock
JB,
Kraeutler
MJ,
Dayton
MR,
McCarty
EC.
A cross-sectional analysis of minimum USMLE Step 1 and 2 criteria used by orthopaedic surgery residency programs in screening residency applications
.
J Am Acad Orthop Surg
.
2017
;
25
(
6
):
464
468
. doi:.
5
Dort
JM,
Trickey
AW,
Kallies
KJ,
Joshi
AR,
Sidwell
RA,
Jarman
BT.
Applicant characteristics associated with selection for ranking at independent surgery residency programs
.
J Surg Educ
.
2015
;
72
(
6
):
e123
e129
. doi:. 2015.04.021.
6
Hamdy
H,
Prasad
K,
Anderson
MB,
Scherpbier
A,
Williams
R,
Zwierstra
R,
et al.
BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice
.
Med Teach
.
2006
;
28
(
2
):
103
116
. doi:.
7
Kay
C,
Jackson
JL,
Frank
M.
The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination
.
Acad Med
.
2015
;
90
(
1
):
100
104
. doi:.
8
Harmouche
E,
Goyal
N,
Pinawin
A,
Nagarwala
J,
Bhat
R.
USMLE scores predict success in ABEM initial certification: a multicenter study
.
West J Emerg Med
.
2017
;
18
(
3
):
544
549
. doi:. 2016.12.32478.
9
Kenny
S,
McInnes
M,
Singh
V.
Associations between residency selection strategies and doctor performance: a meta-analysis
.
Med Educ
.
2013
;
47
(
8
):
790
800
. doi:.
10
Burish
MJ,
Fredericks
CA,
Engstrom
JW,
Tateo
VL,
Josephson
SA.
Predicting success: what medical student measures predict resident performance in neurology?
Clin Neurol Neurosurg
.
2015
;
135
:
69
72
. doi:.
11
Hauser
LJ,
Gebhard
GM,
Blumhagen
R,
Carlson
NE,
Cabrera-Muffly
C.
Applicant characteristics associated with successful matching into otolaryngology
.
Laryngoscope
.
2017
;
127
(
5
):
1052
1057
. doi:.
12
Porter
SE,
Graves
M.
Resident selection beyond the United States medical licensing examination
.
J Am Acad Orthop Surg
.
2017
;
25
(
6
):
411
415
. doi:.
13
Prober
CG,
Kolars
JC,
First
LR,
Melnick
DE.
A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection
.
Acad Med
.
2016
;
91
(
1
):
12
15
. doi:.
14
Love
JN,
Deiorio
NM,
Ronan-Bentle
S,
Howell
JM,
Doty
CI,
Lane
DR,
et al.
Characterization of the Council of Emergency Medicine Residency Directors' standardized letter of recommendation in 2011–2012
.
Acad Emerg Med
.
2013
;
20
(
9
):
926
932
. doi:.
15
Love
JN,
Ronan-Bentle
SE,
Lane
DR,
Hegarty
CB.
The standardized letter of evaluation for postgraduate training: a concept whose time has come?
Acad Med
.
2016
;
91
(
11
):
1480
1482
. doi:.
16
Perkins
JN,
Liang
C,
McFann
K,
Abaza
MM,
Streubel
SO,
Prager
JD.
Standardized letter of recommendation for otolaryngology residency selection
.
Laryngoscope
.
2013
;
123
(
1
):
123
133
. doi:.
17
Kaffenberger
JA,
Mosser
J,
Lee
G,
Pootrakul
L,
Harfmann
K,
Fabbro
S,
et al.
A retrospective analysis comparing the new standardized letter of recommendation in dermatology with the classic narrative letter of recommendation
.
J Clin Asthet Dermatol
.
2016
;
9
(
9
):
36
42
.
18
DeZee
KJ,
Thomas
MR,
Mintz
M,
Durning
SJ:
Letters of recommendation: rating, writing, and reading by clerkship directors of internal medicine
.
Teach Learn Med
.
2009
;
21
(
2
):
153
158
. doi:.
19
Edmond
M,
Roberson
M,
Hasan
N.
The dishonest dean's letter: an analysis of 532 dean's letters from 99 US medical schools
.
Acad Med
.
1999
;
74
(
9
):
1033
1035
.
20
Boysen-Osborn
M,
Mattson
J,
Yanuck
J,
Anderson
C,
Tekian
A,
Fox
JC,
et al.
Ranking practice variability in the medical student performance evaluation: so bad, it's “good.”
Acad Med
.
2016
;
91
(
11
):
1540
1545
. doi:.
21
Boysen-Osborn
M,
Yanuck
J,
Mattson
J,
Toohey
S,
Wray
A,
Wiechmann
W,
et al.
Who to interview? Low adherence by US medical schools to medical student performance evaluation format makes resident selection difficult
.
West J Emerg Med
.
2017
;
18
(
1
):
50
55
. doi:.
22
Hom
J,
Richman
I,
Hall
P,
Ahuja
N,
Harman
S,
Harrington
R,
et al.
The state of medical student performance evaluations: improved transparency or continued obfuscation?
Acad Med
.
2016
;
91
(
11
):
1534
1539
. doi:.
23
Ward
MA,
Palazzi
DL,
Lorin
MI,
Agrawal
A,
Frankenthal
H,
Turner
TL.
Impact of the final adjective in the Medical Student Performance Evaluation on determination of applicant desirability
.
Med Educ Online
.
2018
;
23
(
1
):
1542922
. doi:.
24
Campbell
BH,
Havas
N,
Derse
AR,
Holloway
RL.
Creating a residency application personal statement writers workshop: fostering narrative, teamwork, and insight at a time of stress
.
Acad Med
.
2016
;
91
(
3
):
371
375
. doi:.
25
Ostapenko
L,
Schonhardt-Bailey
C,
Sublette
JW,
Smink
DS,
Osman
NY.
Textual analysis of general surgery residency personal statements: topics and gender differences
.
J Surg Educ
.
2018
;
75
(
3
):
573
581
. doi:.
26
Max
BA,
Gelfand
B,
Brooks
MR,
Beckerly
R,
Segal
S.
Have personal statements become impersonal? An evaluation of personal statements in anesthesiology residency applications
.
J Clin Anesth
.
2010
;
22
(
5
):
346
51
. doi:.
27
Takayama
H,
Grinsell
R,
Brock
D,
Foy
H,
Pellegrini
C,
Horvath
K.
Is it appropriate to use core clerkship grades in the selection of residents?
Curr Surg
.
2005
;
63
(
6
):
391
396
. doi:.
28
Alexander
EK,
Osman
NY,
Walling
JL,
Mitchell
VG.
Variation and imprecision of clerkship grading in US medical schools
.
Acad Med
.
2012
;
87
(
8
):
1070
1076
. doi:.
29
Myles
TD.
United States Medical Licensure Examination Step 1 scores and obstetrics-gynecology clerkship final examination
.
Obstet Gynecol
.
1999
;
94
(
6
):
1049
1051
.
30
Myles
T,
Galvez-Myles
R. USMLE
Step 1 and 2 scores correlate with family medicine clinical and examination scores
.
Fam Med
.
2003
;
35
(
7
):
510
513
.
31
Perez
JA
Jr,
Greer
S.
Correlation of United States Medical Licensing Examination and Internal Medicine In-Training Examination performance
.
Adv Health Sci Educ Theory Pract
.
2009
;
14
(
5
):
753
758
. doi:.
32
Thundiyil
JG,
Modica
RF,
Silvestri
S,
Papa
L.
Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents?
J Emerg Med
.
2010
;
38
(
1
):
65
69
. doi:.
33
Carmichael
KD,
Westmoreland
JB,
Thomas
JA,
Patterson
RM.
Relation of residency selection factors to subsequent orthopedic in-training examination performance
.
South Med J
.
2005
;
98
(
5
):
528
532
. doi:.
34
Armstrong
A,
Alvero
R,
Nielsen
P,
Deering
S,
Robinson
R,
Frattarelli
J,
et al.
Do US Medical Licensure Examination Step 1 scores correlate with Council on Resident Education in Obstetrics and Gynecology in-training examination scores and American Board of Obstetrics and Gynecology written examination performance?
Mil Med
.
2007
;
172
(
6
):
640
643
.
35
Puscas
L.
Viewpoint from a program director: they can't all walk on water
.
J Grad Med Educ
.
2016
;
8
(
3
):
314
316
. doi:.
36
Higgins
E,
Newman
L,
Haligan
K,
Miller
M,
Schwab
S,
Kosowicz
L.
Do audition electives impact match success?
Med Educ Online
.
2016
;
21
(
10
):
1
4
. doi:.
37
Camp
CL,
Sousa
PL,
Hanssen
AD,
Karam
MD,
Haidukewych
GJ,
Oakes
DA,
et al.
The cost of getting into orthopedic residency: analysis of applicant demographics, expenditures, and the value of away rotations
.
J Surg Educ
.
2016
;
73
(
5
):
886
891
. doi:.
38
Papadakis
M,
Hodgson
C,
Teherani
A,
Kohatsu
N.
Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board
.
Acad Med
.
2004
;
79
(
3
):
244
249
.
39
Papadakis
M,
Teherani
A,
Banach
M,
Knettler
TR,
Rattner
SL,
Stern
DT,
et al.
Disciplinary action by medical boards and prior behavior in medical school
.
N Eng J Med
.
2005
;
353
(
25
):
2673
2682
. doi:.
40
Papadakis
M,
Arnold
GK,
Blank
LL,
Holmboe
ES,
Lipner
RS.
Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards
.
Ann Intern Med
.
2008
;
148
(
11
):
869
876
.
41
Schulman
CI,
Kuchkarian
FM,
Withum
KF,
Boecker
FS,
Graygo
JM.
Influence of social networking websites on medical school and residency selection process
.
Postgrad Med J
.
2013
;
89
(
1049
):
126
130
. doi:.
42
Wells
DM.
When faced with Facebook: what role should social media play in selecting residents?
J Grad Med Educ
.
2015
;
7
(
1
):
14
15
. doi:.
43
National Resident Matching Program
.
Results of the 2016 NRMP Program Director Survey. Washington, DC
:
National Resident Matching Program; 2016
.
44
Bhat
R,
Takenaka
K,
Levine
B,
Goyal
N,
Garg
M,
Visconti
A,
et al.
Predictors of a top performer during emergency medicine residency
.
J Emerg Med
.
2015
;
49
(
4
):
505
512
. doi:.
45
King
K,
Kass
D.
What do they want from us? A survey of EM program directors on EM application criteria
.
West J Emerg Med
.
2017
;
18
(
1
):
126
128
. doi:.
46
Joshi
ART,
Vargo
D,
Mathis
A,
Love
JN,
Dhir
T,
Termuhlen
PM.
Surgical residency recruitment-opportunities for improvement
.
J Surg Educ
.
2016
;
73
(
6
):
e104
e110
. doi:.
47
Katsufrakis
PJ,
Uhler
TA,
Jones
LD.
The residency application process: pursuing improved outcomes through better understanding of the issues
.
Acad Med
.
2016
;
91
(
11
):
1483
1487
. doi:.
48
Martin
M,
Salzberg
L.
Resident characteristics to evaluate during recruitment and interview: a Delphi study
.
Educ Prim Care
.
2017
;
28
(
2
):
81
85
. doi:.
49
Schenker
ML,
Baldwin
KD,
Israelite
CL,
Levin
LS,
Mehta
S,
Ahn
J.
Selecting the best and brightest: a structured approach to orthopedic resident selection
.
J Surg Educ
.
2016
;
73
(
5
):
879
885
. doi:.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.