Medical students today submit more residency applications than ever before. This trend is costly for students and imposes a substantial burden on program directors. Yet, despite a steady increase in the number of applications submitted per applicant, overall match rates have not improved.1  Put another way, applicants could collectively apply to fewer programs than they do now—and enjoy essentially the same overall match rate.

To assist students in determining the optimal number of residency programs to which they should apply, the Association of American Medical Colleges (AAMC) unveiled Apply Smart in 2016. The website notes that “there is a point where the relationship between the number of applications submitted and the likelihood of entry into a residency changes,” and suggests that students should consider limiting their applications at this point of diminishing returns.2  Responses to Apply Smart have been positive, with deans and program directors praising the tools and encouraging their use in counseling medical students.36 

At first glance, the Apply Smart analyses seem highly informative. Yet, closer inspection reveals methodologic issues that introduce bias and suggest the need for improvements.

The Apply Smart analyses include data and graphics for 19 medical specialties, broken down by applicant type (US MD, US DO, and both US and non-US citizen international medical graduates). For simplicity and consistency, this article will only consider the analyses of MD graduates of US medical schools.

On each graphic, the x-axis represents the number of residency programs in that specialty to which a group of applicants applied. The y-axis indicates the probability of entering a program in that specialty. Three curves appear on each graphic. These were fitted with spline regression and correspond to the probability of entering that specialty for applicants within the top, middle, or bottom tertile of United States Medical Licensing Examination Step 1 scores.

The curves for each specialty have a similar form. Students who apply to few programs have a low probability of entering a residency program in that specialty. Applicants who apply to more programs have a greater probability of matching, up to a “point of diminishing returns” (Table 1). The point of diminishing returns is set as the first knot in the spline regression model and reflects the point at which the likelihood of entering a program in that specialty does not significantly increase when compared to applicants who submitted fewer applications.

Table 1

Mean Applications Submitted, “Point of Diminishing Returns,” and Ranks Required for > 90% Match Rate by Specialty

Mean Applications Submitted, “Point of Diminishing Returns,” and Ranks Required for > 90% Match Rate by Specialty
Mean Applications Submitted, “Point of Diminishing Returns,” and Ranks Required for > 90% Match Rate by Specialty

Close inspection of the Apply Smart graphics reveals several surprising features. For instance, the probability of successfully entering a residency program for applicants who apply to fewer than 5 to 10 programs is strikingly low—sometimes just 20% to 40%. Given the high cost of not securing a residency position, it is curious that so many candidates would “underapply,” especially since the Electronic Residency Application Service (ERAS) charges a flat rate for the first 10 applications within a specialty.7  What type of student would apply to less than 10 programs and accept such a low probability of success? The likely answer is one who is not especially interested in matching to that specialty in the first place.

Importantly, although the y-axes of the Apply Smart graphics are labeled “probability of entering a residency program,” the analyses are specialty-specific. That is, applicants are considered to have entered a residency program only if they enter the specialty shown in the graphic. Applicants who enter another specialty are considered as not having entered a residency program. Yet many medical students apply to multiple specialties. For instance, in 2018, 83% of applicants whose preferred specialty was dermatology applied to at least one other specialty, even though the vast majority of these applicants (82%) ultimately matched to dermatology.8 

Similarly, the Apply Smart graphics show a maximal likelihood of entering a residency program that is curiously low, even for those submitting the ideal number of applications. For instance, the graphic for anesthesiology shows an asymptote around 75%—even though 96% of US seniors who preferred anesthesiology successfully matched in 2018.8  Notably, the National Resident Matching Program (NRMP) Match data are limited to an applicant's “preferred specialty,” as determined by their first-ranked program. Therefore, for every specialty, the Apply Smart probability of entering a residency program is lower than the corresponding match rate (Table 2).

Table 2

Apply Smart Probability of Entering a Residency Program Versus Match Rate by Specialty

Apply Smart Probability of Entering a Residency Program Versus Match Rate by Specialty
Apply Smart Probability of Entering a Residency Program Versus Match Rate by Specialty

For most specialties, the Apply Smart “point of diminishing returns” is lower than the mean number of applications submitted by applicants applying in that field (Table 1). But will Apply Smart actually encourage fewer applications?

Predicting the overall effect of Apply Smart requires some knowledge of the overall distribution of applications submitted by medical students. While the AAMC does not publicly report this information, several studies show a right-tailed distribution, with a relatively small number of candidates applying to the greatest number of programs. For instance, approximately 25% of internal medicine residents who graduated from US medical schools submitted fewer than 15 applications, while only approximately 7% submitted more than 75 applications.10  Similarly, in general surgery, the median number of applications submitted by US medical graduates was 26 versus a mean of 41 and a point of diminishing returns ranging from 36 to 49.6,9,11 

Unfortunately, by failing to exclude applicants who are applying to backup specialties, the Apply Smart analyses present biased estimates of the probability of residency entry. How these probabilities are likely to be interpreted by students should be carefully considered. For instance, it is debatable whether Apply Smart data will successfully convince medical students applying to the greatest number of programs that they should instead apply only to the point of diminishing returns—especially when doing so appears to confer only a 60% to 70% probability of entering a residency program in that specialty. On the other hand, for students applying to fewer programs, the interpretation of the Apply Smart analyses is unambiguous: they should apply to more programs, up to the point of diminishing returns, to avoid the seemingly low probability of success associated with submitting fewer applications. Yet, given that the number of applications submitted has a right-tailed distribution, more candidates may find themselves in the latter category than the former. Thus, in its current form, Apply Smart may actually cause an increase in overall applications.

How can the AAMC provide higher quality information to inform medical students' decision-making? There are 2 broad possibilities.

The first is to partner with the NRMP and limit the Apply Smart analyses to a student's preferred specialty. Forging such a partnership may be administratively challenging, but would permit analyses that are unbiased by candidates applying to backup specialties. Yet such a solution still fails to address a fundamental problem with Apply Smart: the probability of entering a residency program is not primarily determined by the number of programs to which an applicant applies. The nature of those programs and the competitiveness of the applicant matter more.

Therefore, a second strategy would be to leverage ERAS data to provide applicants with more informative statistics. The AAMC could do this by focusing not on the number of applications submitted, but on the probability that a given application will result in an interview offer.

Unlike the relationship between the number of applications submitted and residency entry, the number of ranked programs needed to successfully match is quite predictable. Through its Charting Outcomes in the Match reports, the NRMP reports the probability of matching by the number of contiguous ranks submitted (ie, the number of programs an individual ranked within one specialty before ranking a program in another specialty). For almost every specialty, ranking 5 to 10 programs is associated with a better than 90% chance of Match success (Table 1). Because applicants may not rank all programs at which they interview, the number of ranked programs does not always equal the number of interviews completed. Yet the former is a reasonable surrogate for the latter: in 2019, the median number of interviews applicants attended was the same as the median number of programs ranked for almost all specialties.12  Thus, instead of encouraging applicants to apply haphazardly to a particular number of programs in the hope of generating enough interviews to successfully match, a better strategy may be to help applicants specifically select a group of programs where their application is likely to result in an interview.

With the support of residency programs, the AAMC could use multivariable logistic regression to estimate the probability that a given applicant would be offered an interview at a particular program, based on interview decision data from recent application cycles and readily accessible ERAS data (standardized test scores, citizenship status, geographic proximity, honor society membership, etc). This information could be incorporated into the existing Residency Explorer tool and would provide more individualized and interpretable information to students than the general statistics that this website currently contains. If students know that they have a better than 90% (or less than 10%) chance of being offered an interview at a certain program, they may adjust their application strategy accordingly.

Overapplication is costly, for applicants and programs alike. Given the incentives for medical students to overapply, it is unclear whether informational strategies alone can curtail overapplication. Until graduate medical education leaders are willing to support application caps or a fundamental restructuring of the Match to better allow signaling between applicants and programs,13  it is imperative that informational strategies present unbiased data that can aid students in applying to an appropriate number of programs. We cannot be satisfied for students to Apply Smart—we need to help them apply smarter.

1
Weissbart
SJ,
Kim
SJ,
Feinn
RS,
Stock
JA.
Relationship between the number of residency applications and the yearly Match rate: time to start thinking about an application limit?
J Grad Med Educ
.
2015
;
7
(
1
):
81
85
. doi:.
2
Association of American Medical Colleges
.
Apply Smart: Data to Consider When Applying for Residency
. ,
2019
.
3
Andolsek
KM.
One small step for Step 1
.
Acad Med
.
2019
;
94
(
3
):
309
313
. doi:.
4
Gliatto
P,
Karani
R.
Viewpoint from 2 undergraduate medical education deans
.
The residency application process: working well, needs fixing, or broken beyond repair? J Grad Med Educ
.
2016
;
8
(
3
):
307
310
. doi:.
5
Liao
NN,
Mahan
JD,
Scherzer
R.
The pediatric match frenzy: an overview and an approach for mentoring medical students
[published online ahead of print June 12,
2019]
.
Acad Pediatr.
doi:.
6
Joshi
ART,
Choi
J,
Terhune
K. ERAS
and the NRMP and their roles in residency recruitment—a primer for surgical program directors
.
J Surg Educ
.
2019
;
76
(
5
):
1163
1166
. doi:.
7
Association of American Medical Colleges
.
Fees for ERAS Residency Applications
. ,
2019
.
8
National Resident Matching Program
.
Charting Outcomes in the Match: US Allopathic Seniors: Characteristics of US Allopathic Seniors Who Matched to Their Preferred Specialty in the 2018 Main Residency Match
. ,
2019
.
9
Association of American Medical Colleges
.
Table C-3: Residency Applicants to ACGME-Accredited Programs by Specialty and Medical School Type, 2018–2019
. ,
2019
.
10
Angus
SV,
Williams
CM,
Kwan
B,
Vu
TR,
Harris
L,
Muntz
M,
et al.
Drivers of application inflation: a national survey of internal medicine residents
.
Am J Med
.
2018
;
131
(
4
):
447
452
. doi:.
11
Association of American Medical Colleges
.
Apply Smart: Data to Consider When Applying to Residency
.
US MD, General Surgery.
,
2019
.
12
National Resident Matching Program
.
Results of the 2019 NRMP applicant survey
. ,
2019
.
13
Coles P, Cawley J, Levine PB, Niederle M, Roth AE, Siegfried JJ
.
The job market for new economists: a market design perspective
.
J Econ Perspect
.
2010
;
24
(
4
):
187
206
.