A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews

All medical students wonder what is needed to receive an interview invitation to the residency program of their choice. In the last 2 decades there has been a sustained increase in the number of applications per student, probably due to attempts to ensure interview invitations.¹  As application numbers continue to rise, it is essential that programs have a strong sense of which factors should be considered and why. In this narrative review, we will explore how programs use certain factors to select students for residency interviews and the evidence for the utility of these factors as predictors of success in residency and beyond. We aim to illuminate the advantages and disadvantages of using those markers as differentiating elements in selecting candidates to interview.

As a basis for this narrative review, we examined what residency directors say about the factors most commonly used to select applicants for residency interviews, using the 2018 National Resident Matching Program (NRMP) Program Director Survey.²  In answering the question, “What factors are most important for obtaining residency interviews?” we found this survey to be the most comprehensive guide available.

The factors program directors ranked in the 2018 NRMP Program Director Survey as most important for selecting applicants to interview are listed in the table. We defined factors as “most important” if at least 70% of program directors said they used them or if they had an average rating greater than 4.3.

We searched PubMed, Web of Science, Ovid, and Scopus for articles published from 2008 to 2018 that contained key terms related to residency application as well as key terms for the factors identified in the NRMP Program Director Survey. Article abstracts were reviewed by the authors for pertinence to the topics at hand. Articles underwent analysis for relevance to the topic and quality of the study design. As strong evidence for these factors was sparse in some cases, we included studies that evaluated final applicant ranking as well as invitations for interviews, emphasizing those that examined the predictive power of these factors.

For context, the authors are 3 emergency physicians on faculty at Wake Forest School of Medicine with extensive experience in the areas of resident selection and student advising. N.D.H. is an assistant residency director with more than 5 years of experience, C.W.L. is a residency program director with more than 10 years of residency leadership experience, and D.E.M. has 25 years of experience ranging from clerkship director to associate dean for medical education. All authors actively participate in the annual process of selecting candidates for interviews and also advise students entering our specialty.

In addition to determining qualification for licensure, United States Medical Licensing Examination (USMLE) test results are often used in screening applicants for residency.^3,4  In the surgical literature, USMLE Step 1 and Step 2 Clinical Knowledge (CK) ranking quartiles have been found to be predictive of program ranking, with higher test scores associated with higher probability of applicant ranking.⁵  Although Step 1 was not designed to be a primary determinant of the likelihood of success in residency, low scores on USMLE Step 1 correlate with failure on specialty in-training and certification examinations.⁶  USMLE Step 1 scores demonstrate a moderate correlation with performance on the specialty-specific internal medicine and emergency medicine board examinations.^7,8  Some investigations have found a strong association between the scores and the success of a physician after residency,⁹  while most show that Step 1 performance has no correlation to resident quality or success, even if it matches in-service scores.^10–13 

Eighty-six percent of program directors cited letters of recommendation as a factor in selecting applicants to interview. Recent initiatives to standardize these documents and move away from “recommendation” to “evaluation” are gaining a foothold in multiple medical specialties, including emergency medicine, dermatology, otolaryngology, and internal medicine.^14–17 These standardized letters often convey more actionable information and comparative data than a freeform letter. When assessing standardized letters, internal medicine residency directors value depth of familiarity with the applicant as well as quantitative comparison with other applicants.¹⁸ 

The Medical Student Performance Evaluation (MSPE), often referred to as the “Dean's letter,” has long served as a rather opaque tool for evaluating applicants. In the past, studies have found that negative academic information is routinely suppressed in these documents.¹⁹  MSPEs are often not written according to the 2002 MSPE guidelines, with medical schools inconsistently providing graphic comparative data.²⁰  They also rarely provide comparative professionalism data.²¹  The terminology used in the MSPE can be difficult to decode. In one study, 72 distinct key words were used to describe performance and 27 of those were assigned to the top category. The median percentage of medical students who received the top category was 24%, ranging from 1% to 60%. Ten percent of schools using key words did not provide distribution data and another 17% who used key words gave no data on how to interpret them.²²  These key words, or “final adjectives,” actually have a greater impact on perceived desirability than any other aspect of the letter.²³  These words usually relate to a measurement of class rank or percentile.

The personal statement is the ideal place to express an applicant's personal beliefs and journey. It is also an opportunity for an applicant to demonstrate a commitment to the specialty, which program directors cite as an important factor. There is little evidence to show exactly what should or should not be included in a personal statement or how these compositions relate to future success. Medical schools are starting to develop writing seminars for the personal statement.²⁴  Themes for personal statements can also display gender differences.^25,26 

Grades in required third-year clerkships help determine class rank, and may be individually important for a specialty if represented in the core clerkships. Survey data outside of NRMP confirm the importance placed on core clerkship grades in fields such as otolaryngology.¹¹  Variability in the grading systems and the percentage of students who achieve the highest performance levels can make these grades difficult for program directors to interpret, and the predictive value is unclear.^27,28 

Thirty percent of program directors stated that they would never interview a student with a Step 1 failure (an additional 58% “seldom” would) and 35% would not interview a student with a Step 2 failure. A Step 1 failure is a predictor of subsequent clerkship and National Board of Medical Examiners shelf examination grades.^29,30  Students who fail Step 1 on their first attempt are less likely to match overall, but a majority will obtain a residency spot. Failure of Step 1 also predicts lower performance on the in-training examinations in internal medicine,^6,31  emergency medicine,³²  orthopedic surgery,³³  and obstetrics and gynecology.³⁴ 

Residency programs often consider class rank in the interview process. In line with the Association of American Medical Colleges' initiative on competency, many medical schools have abandoned class ranking in favor of a dichotomous determination of competence (pass/fail). No clear relationship between class rank and residency performance has been shown, and there is little evidence that a lower class rank without systemic failure portends inability to match.³⁵ 

Perceived commitment to a specialty is cited by 69% of program directors as a factor in choosing interviewees. There are 3 main ways to demonstrate this commitment in the application: by previous experience, in the personal statement, and by completing rotations at other institutions within the field, usually generating letters of recommendation. Audition rotation completion was a measured factor on the NRMP survey and did not meet the criteria for inclusion in this review. The impact of audition rotations on applicants' match potential is dependent on specialty.^36,37 There is no quality evidence regarding the relationship between perceived commitment to specialty and later success.

Some specialties include the grade received in the clerkship in a standard letter of evaluation along with other quantitative and comparative assessments of the applicant, making that piece of data particularly useful in the evaluation process.^13,14  It is important that the grade distribution be expressly stated so the reviewer can place the applicants' scores in appropriate context.

Professionalism among applicants is of substantial interest to program directors because professional challenges during medical school endure into and beyond residency training.^38–40  Social networking sites, or more specifically the lack of professionalism displayed on them, may adversely affect a candidate's application.⁴¹  Programs and students should be familiar with the social media landscape and the privacy issues therein.⁴²  Match violation citations likely raise professionalism concerns, leading to some degree of overlap in these factors.

Our findings in this review demonstrate that many of the most commonly used factors are not necessarily predictive of an applicant's future performance or have scant evidence basis. The literature is more robust around factors that are easy to quantify, such as USMLE scores, and less clear about factors such as the personal statement and commitment to the specialty. The weight of the literature regarding predictive value of Step 1 performance shows that there is little to no correlation with resident quality or success, beyond a correlation with later specialty board passage. The use of the Step 1 score (other than failure) as a discriminator for whom to offer a residency interview is not well supported and its impact on selection should be markedly lessened. Evidence supports the use of structured letters of recommendation in applicant selection. Use of a structured letter has become a “best practice” employed in multiple specialties: emergency medicine, otolaryngology, dermatology, and internal medicine.

Two specific negative findings in an application appear as predictors of future poor performance: unprofessional behavior and USMLE examination failure. The finding of unprofessional behavior in medical school correlates with similar behavior in residency and beyond, as best demonstrated in the internal medicine literature. Failure of Step 1 or Step 2 is a predictor of poor performance on specialty examinations in multiple specialties.

The literature is less clear around factors such as the personal statement and commitment to the specialty. There is no evidence as to what should or should not be in a personal statement. This lack of evidence may explain a tendency toward emphasizing quantifiable, comparable factors over others. However, quantitative factors, such as USMLE scores, do not appear to predict success in residency. Thus, graduate medical education leaders should promote use of other selection factors and be cautious about using any single factor to exclude applicants from consideration.

Important limitations to this study include the low response rate to the NRMP survey (29.2% in this iteration and 39.9% in 2016). However, the remarkable degree of stability of its findings from year to year argues for some credibility that these factors are the most frequently reviewed and thus relevant.⁴³  Additionally, separate studies, including specialty-specific studies, have affirmed that many of these factors are important to residency program directors.^44,45  Current literature on this issue, including the NRMP survey, is limited to some extent by the heterogeneity of selection processes, heterogeneity of the literature itself, and difficulty in defining outcomes such as “success in residency.” With varying outcomes used to define success in residency it is difficult to determine the predictive value of individual application factors. The use of these factors, and their relative importance, seems to vary significantly by specialty, which further limits a general review.

Potential next steps include focused research on the predictive value of these factors. Several specialties have investigated the residency application process to better identify applicants who would be suited to both the specialty and the specific residency program.^46–49  These initiatives, cited here from general surgery, orthopedics, and family medicine, share common themes: de-emphasis of Step 1 scores, standardized evaluation letters, improvements in the MSPE to include student progress in various competencies, holistic file review, and evaluating for personal characteristics thought to be important to the specialty in question.

Program directors use a variety of factors to select applicants for residency. Some of these factors, such as USMLE scores and the MSPE, have little supporting evidence in predicting future applicant performance in residency. New initiatives, such as structured letters of recommendation, may provide better predictive value yet require further study. The limitations of the current evidence challenge the notion that any single factor should be used as a “red flag.”