Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

Calls for substantive reforms to the US resident selection process are growing, given increasing competition and inefficiencies of the current system.¹  Over the last decade, applicants have doubled the number of applications they submit.^2–4  Inundated by applications, programs are increasingly reliant on filters such as United States Medical Licensing Examination (USMLE) Step 1 scores for screening, despite its bias against minorities and poor prediction of residency performance.^5–8  With Step 1 transitioning to pass/fail in 2022, programs may utilize Step 2 Clinical Knowledge (CK) for screening in the absence of other reforms.⁹ 

Even if programs could conduct a holistic review of all applications, limited standardized data are available for review: clerkship grading distributions vary between institutions,¹⁰  Medical Student Performance Evaluations (MSPEs) lack standardized objective measures,^11,12  and narrative letters of recommendation are subjective and omit applicant shortcomings.¹³  Once interview offers are extended, a “first-come, first-served” frenzy occurs among applicants to secure an interview,^4,14  often with competitive applicants hoarding interviews only to cancel last minute.^14,15  Couples, osteopathic (DO) graduates, and international medical graduates (IMG) face additional barriers.¹⁶ 

Numerous reforms have been proposed to improve the efficiency, equity, and integrity of the resident selection process. However, many proposals are published in specialty-specific journals, and a comprehensive compilation of all reforms is currently lacking.^4,17,18  Therefore, this scoping review was conducted to characterize systems-level reforms to the resident selection process. We aim to provide educational leaders with a clear framework and consistent language to facilitate national discussions.

The protocol was drafted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) and prospectively registered with the Open Science Framework on February 20, 2020.¹⁹ 

Designed by a health science librarian, comprehensive searches of the Embase, MEDLINE, Scopus, and Web of Science databases were conducted in February 2020 for articles published from January 2005 to February 2020 (provided as online supplementary data). In an attempt to capture all reforms presented in editorials, commentaries, and letters, the table of contents of the following undergraduate and graduate medical education journals were manually searched: Academic Medicine, Journal of Graduate Medical Education, Advances in Health Sciences Education, BMC Medical Education, Teaching and Learning in Medicine, and Medical Teacher. Furthermore, bibliographies of included articles were manually searched for additional relevant articles. All records were imported into Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia), and duplicates were removed.

English language articles proposing systems-level reforms to the US residency application, selection, and match process were included if applicable to all applicants, medical schools, or residency programs, regardless of specialty. Articles were excluded if no reform was proposed, a previously proposed reform was mentioned without explicit endorsement, the reform referenced a resident selection process outside the United States, or the reform was not generalizable beyond the individual program (eg, program-specific resident selection protocols, hiring independent contractors to assess program culture). Articles proposing reforms to the fellowship match process were also excluded.

References were independently screened by 2 authors (R.Z., D.L., or J.BR.) for inclusion based on their title and abstract. Articles then underwent full-text review by 2 reviewers (R.Z., D.L.). A third reviewer (J.BR.) reconciled discordant cases at screening and full-text review.

Two reviewers (R.Z., D.L.) extracted data in duplicate. Extracted data included the type of reform, implementation strategies, cited advantages and disadvantages of the proposed reform, and the specialty targeted by the reform as suggested by the journal in which the article was published or authors' affiliations. Articles were classified as “not specialty specific” if authors were from multiple specialties or if the proposed reform was explicitly applicable to multiple specialties.

Qualitative content analysis using an inductive approach with grounded theory was performed to categorize reforms.²⁰  First, 2 reviewers independently read a subset of included articles and created preliminary codes. Reviewers then reconciled their findings, refined preliminary codes, and generated a codebook. The process was repeated for subsequent articles in multiple groupings, creating new codes as needed. Finally, similar codes were categorized into higher-order categories and themes. ATLAS.ti 8 (ATLAS.ti Scientific Software Development, Berlin, Germany) was used to aid in higher-level category creation and assessment of code co-occurrences.

Of the 10 407 unique references identified, 225 proceeded to full-text review and 116 were included in the scoping review (Figure). Though many references were not specialty-specific (n = 38), otolaryngology (n = 22), orthopaedic surgery (n = 16), general surgery (n = 8), emergency medicine (n = 7), and plastic surgery (n = 5) were the specialties most frequently proposing reforms (Figure provided as online supplementary data). Over the study period, there was a steady increase in the number of articles proposing reforms each year (Figure provided as online supplementary data).

Our inductive approach generated 34 codes that were grouped into 14 categories and then organized according to broad stages of the resident selection process: application submission, application review, interviews, and the Match (Table 1). Additionally, the pros and cons of each reform, as reported by the reviewed articles, were compiled and summarized (Table 2).

Twenty-eight articles endorsed an application cap.^{4,15,18,21–45}  This was the most frequently proposed reform and implementation strategies varied. Many recommended fixed caps for all specialties, but there was no consensus on the optimal cap number.^{4,15,18,21–41}  Others suggested variable caps based on the specific-specialty and/or applicant metrics,^41–43  or “soft” caps imposed by resource-intensive supplemental applications or higher Electronic Residency Application Service (ERAS) fees, without explicitly setting limits.^41,44,45  The cited advantages of application caps include cost-savings to applicants and decreased application volume for programs, presumably facilitating holistic review.^{5,24,26–28,39}  Challenges include determining the optimal cap and limited program data currently available to inform applicant decisions.⁴⁰ 

Eleven articles promoted supplemental applications to express program preference, including a program-specific paragraph in the personal statement, written or video statements of interest, or secondary applications.^{4,18,26–28,40,41,46–49}  This proposal frequently co-occurred with application caps (n = 7) as a mechanism for reducing application volume through their time-intensive nature.^24,50,51  However, requiring applicants to submit these materials results in an increase in resources needed for application review.⁵²  Similarly, 5 articles endorsed the creation of a “signaling” system within ERAS or a separate, third-party platform, allowing applicants to designate a specified number of “preferred” programs.^{24,44,50–52}  Such limited preference signaling makes applicant interest explicit and may facilitate holistic review by residency programs,⁴⁴  but may increase applicant costs via third-party servicing fees.⁵⁰ 

Twenty-one articles proposed the creation of a database with standardized program information^{4,5,15,17,18,24,27–29,33,41–43,50,53–60}  beyond data currently captured in the American Medical Association FREIDA Tool⁶¹  and Association of American Medical Colleges (AAMC) Residency Explorer Tool.⁶²  Data captured may include program information (eg, curriculum, case logs, research opportunities, graduate outcomes), screening criteria (eg, USMLE scores, AOA status, DO/IMG status, publications), and metrics of previously matched applicants (eg, National Resident Matching Program [NRMP] statistical profiles). This proposal frequently co-occurred with application caps (n = 10) and supplemental applications (n = 4), as these reforms likely assist applicants in identifying suitable programs. Additionally, 6 articles suggested residency programs establish a “brand” (language used in the articles) that conveys the culture, desired attributes of residents, and ideal career path for graduates, thereby informing applicants of the type of candidate likely to thrive in that culture.^{13,17,33,48,63,64}  This reform may enhance the compatibility between applicants and programs, improving resident satisfaction, and decreasing residents leaving the program.⁶⁴ 

Twelve articles advocated for holistic review with balanced consideration of academic performance, extracurricular activities, and personal experiences.^{18,40,41,60,65–72}  One article promoted blinding Step 1 scores,⁶⁸  and another encouraged the removal of specialty board passage rates as part of the Accreditation Council for Graduate Medical Education accreditation of residency programs to facilitate holistic review.¹⁸  Additionally, 2 articles encouraged the creation of national norms to holistically quantify and compare applicant accomplishments.^13,65  Holistic review may increase diversity and improve compatibility between applicants and programs, but requires additional resources and is subjective.^40,60,70  To mitigate these issues, 3 articles suggested standardized holistic applicant scoring generated from a weighted rubric.^13,73,74 

Two articles recommended nationally standardized grading to address grade inflation and facilitate applicant comparison.^10,75  However, uniform grading is difficult to achieve across all institutions given differing grading schemas (eg, pass/fail or graded) and inconsistent language between—and even within—institutions depending upon the rotation or course.^10,75  Four articles also proposed the creation of new medical school assessments, including competency-based evaluations, simulations, entrustable professional activities, and gateway exercises that serve as knowledge and skill checkpoints.^4,18,66,76  These assessments permit longitudinal evaluation of applicant performance, identify applicant strengths and weaknesses, and provide a common framework for applicant assessments.^47,66  However, concerns regarding validity coupled with variability in assessment utilization, learning objectives, and grading between institutions may preclude direct applicant comparisons.⁶⁶  Finally, one article advocated for new medical school metrics capturing personal and professional experiences in a standardized manner to permit easier comparison of applicants.⁵ 

One article promoted the creation of a general residency selection examination, separate from the USMLE series, that is intentionally designed to assess the knowledge and skills necessary for residency.¹⁷  Similarly, 3 articles endorsed the creation of new specialty-specific examinations and metrics generated from customized National Board of Medical Examiners specialty tests or aptitude tests to capture specialty knowledge and skills.^65,68,77  Though a new residency selection examination or novel specialty-specific examinations/metrics may facilitate holistic review by deemphasizing Step 1 and Step 2 CK, they subject applicants to additional tests and require costly development and validation.⁶⁸ 

Twenty-three articles promoted introduction of noncognitive assessments in resident selection, including personality assessments (n = 10),^{17,72,78–85}  and other psychological assessments including situational judgement tests (n = 5),^{17,64,72,83,85}  emotional intelligence tests (n = 2),^83,86  grit assessments (n = 2),^84,87  moral reasoning tests (n = 1),⁶⁷  and unspecified behavior-based questionnaires (n = 4).^13,88–90  These measures may provide insight into applicants' future behavior in residency and are customizable, allowing programs to identify applicants with the best compatibility for their unique culture, thereby reducing resident attrition.^{78–80,82–84}  However, these assessments may be costly to validate and implement, and results may be skewed by social desirability bias and the Hawthorne effect.^78,82,86 

Twelve articles proposed requiring objective data in the MSPE, particularly class rank, clerkship grades, and shelf examination scores.^{4,11,12,18,31,39,59,91–95}  Proponents also sought disclosure of professionalism issues, academic difficulties, and leaves of absence. Additionally, 8 articles endorsed standardization of the MSPE structure, content, and language.^{11,12,59,60,70,93–95}  These 2 reforms co-occurred 5 times, with the common feature being facilitation of efficient applicant comparisons. Other modifications included third-party MSPE review to ensure institutional compliance with AAMC guidelines as these are currently unenforced.⁵ 

Twenty articles supported standardized letters of recommendation (SLOR) or evaluation (SLOE) to uniformly summarize applicants' academic and professional potential for a given specialty.^{11,13,17,27,31,48,60,90,96–107}  Despite their interpretability, comparability, and objectivity, SLORs/SLOEs may be undermined by grade inflation.^{27,48,90,99,101,104}  Consequently, 3 articles proposed the creation of nationally normed SLOR rubrics and a database comparing letter writer and program performance to national standards to curtail inflation.^27,90,106  Similarly, one article advocated for published letter writer guidelines detailing the preparation of SLORs/SLOEs, particularly the assignment of appropriate grading or ranking.¹⁰⁷  Finally, one article endorsed visual letters of recommendation (eg, word clouds), utilizing terms from the SLOR to enhance review efficiency.¹⁰⁸ 

Thirteen articles proposed pre-interview screening via standardized video interviews,^{66,69,88,109,110}  preliminary videoconference or telephone interviews,^{21,26,57,64,68,76,111}  or regional interviews^29,112  prior to on-site interviews. Such screening practices enable early assessment of applicant/program compatibility and reduce the interview cohort size, delivering cost-savings to applicants and programs alike.^{57,64,109,112}  However, concerns exist regarding the scalability and increased administrative burden of pre-interview screening.^88,109 

Seven articles suggested date standardization for interview offers with an acceptance window, creating a predictable timeline for applicants and minimizing clinical distractions.^{4,13,22,53,113,114}  Alternatively, one article promoted online interview scheduling to preserve “first-come, first-served” acceptances,⁵⁷  and one article proposed an interview match in which applicants and programs submit rank lists for interviews and are then “matched” to interview.¹¹⁵  More radically, 2 articles proposed an interview lottery, with one assigning interview dates from a rank lists of preferred dates,¹¹⁴  and the other randomly filling 50% of interview positions with applicants meeting minimum criteria.³¹ 

Additionally, 6 articles supported an interview cap that limits the number of interviews an applicant can accept and attend.^{21,29,76,113,116,117}  An interview cap may facilitate more equitable interview allocation and decrease costs, but the optimal limit is unclear given variable applicant circumstances and specialty competitiveness.^14,76,117  Five articles promoted date standardization for interviews by specialty or region to decrease costs and serve as a de facto interview cap that limits interview overlap by the same top-tier applicants.^{29,59,76,116,118} 

Eleven articles promoted structured interviews utilizing standardized job-related and behavior-based questions that are scored with a rubric.^{13,17,18,63,64,73,85,86,89,119,120}  Structured interviews enable multidimensional assessment and improve efficiency and interrater reliability, but require interviewer training and the development of validated questions and scoring rubrics.^13,86,119  Additionally, one article proposed virtual reality scenarios involving multiple applicants to assess real-time communication and problem-solving skills,⁸⁹  and another promoted the use of skills-based simulations on interview day to uniformly assess technical abilities and knowledge base.¹²¹ 

Six studies supported a ban on post-interview communication (eg, calls, emails, and “second looks”) and creation of an anonymous reporting system for violations.^{56,113,117,122–124}  This ban may enhance the integrity of the resident selection process and minimize undue influence on applicant rank lists.^14,122,124  Recognizing the challenges of a moratorium, one article suggested allowing post-interview commitments with the requirement of written documentation to improve accountability.¹²⁵ 

Eight articles proposed multiple match rounds with varying application caps per round.^{18,22–24,34,35,126,127}  By limiting the number of applications received, this approach facilitates an in-depth review of truly interest applicants and may allocate interviews more equitably, as early matching of competitive applicants affords interview opportunities for other applicants in subsequent rounds.^24,34  Six articles supported an early assurance match, including guaranteed residency positions as a condition of medical school acceptance, “pre-matching” to home programs or programs where subinternships are performed, or allowing early acceptance to a consortium of institutions.^{17,29,47,70,126,127}  This reform emphasizes knowledge and skill acquisition during medical school rather than residency securement and may improve the undergraduate to graduate medical education transition.^4,29,70,126  However, moving up the resident selection decision may unnecessarily increase emphasis on the medical school an applicant attends.^70,126  Finally, one study endorsed a “free market” approach in which graduation-eligible applicants interview and accept residency positions on a rolling basis.¹²⁸  This approach may expedite medical training, but a continual reassessment of graduation readiness is burdensome, and rolling offers may pressure applicants into making decisions with incomplete knowledge.^128,129 

Calls for resident selection reform grew over the past 15 years, particularly among competitive specialties such as otolaryngology and orthopaedic surgery. This is likely in response to increasing application volume and applicant competitiveness with limited comparative metrics. Many popular reforms, including application caps, supplemental applications, and standardized letters of evaluation, seem to benefit both applicants and programs via application reduction and efficient applicant comparisons, facilitating holistic review.

Prior to the 2020–2021 application cycle, specialties implemented several reforms identified in this scoping review. For instance, emergency medicine instituted the SLOE in 1997 for efficient applicant comparison.⁹¹  Though initially hamstrung by grade inflation, a concerted effort to create a national cohort of experienced authors addressed this limitation, and the SLOE remains a mainstay of the emergency medicine resident selection process.¹⁰⁶  Likewise, emergency medicine partnered with the AAMC from 2017–2020 to pilot the standardized video interview, an asynchronous online interview that assesses applicants' professionalism and interpersonal communication skills.¹¹⁰  However, concerns regarding its validity, cost, and negative applicant perceptions prompted discontinuation of the standardized video interview.¹³⁰  Additionally, otolaryngology mandated a program-specific paragraph in the personal statement in 2015 to gauge applicant interest and deter application inflation.⁴⁶  Program-specific paragraphs were found to be generically crafted,⁴⁶  and seemingly exacerbated a downturn in the number of otolaryngology applicants, resulting in them now being optional.^131,132 

The COVID-19 pandemic has affected the residency application process in numerous ways,^133–136  accelerating calls for disruptive innovation and affording opportunities for novel reform.^1,137–141  Application caps were the most frequently proposed reform (n = 28 articles), and many specialties have recommended limits for the 2020–2021 cycle.^138,142  Some have suggested deriving application caps from the AAMC Apply Smart tool, which correlates application data with specialty entrance rates to identify the point of diminishing returns for application submissions stratified by applicant type (MD/DO/IMG) and Step 1 score tertile.¹⁴³  However, several methodologic concerns exist regarding the calculation and use of specialty entrance rates rather than match rates as well as the applicability of the data to individual applicants.¹⁴⁴  Moreover, the acceptability of application caps varies by applicant and specialty,^24,145  and unenforced recommendations are unlikely to alter applicant behavior.

In contrast to applications, an applicant's number of contiguous ranks, a proxy for the number of interviews attended, correlates well with their match rate.¹⁴⁶  A small cohort of highly competitive applicants accept and attend a disproportionate number of interviews, and virtual interviews are likely to exacerbate this maldistribution by removing cost and travel constraints.¹⁴⁷  Calls for interview caps are growing,^148,149  and ophthalmology utilized a centralized scheduling platform to limit applicants to 20 virtual inteviews.¹⁵⁰  Likewise, obstetrics and gynecology, orthopaedic surgery, and dermatology implemented standardized interview offer dates with acceptance windows, creating a predictable timeline.^151–153 

In addition, otolaryngology implemented preference signaling, permitting applicants 5 signals each to explicitly designate their interest in programs.¹⁵⁴  Coupling preference signaling with a comprehensive database of residency program information, as is underway by ophthalmology and obstetrics and gynecology, assists applicants in identifying suitable programs.^150,151  Other proposed strategies such as pre-interview screening provide innovative approaches for narrowing the applicant pool before full virtual interviews, but risk introducing an unvalidated metric susceptible to implicit bias. Finally, 9 specialties promoted holistic review, which is easy to suggest, but difficult to achieve given that graduate medical education programs are under-resourced for the current application volume.¹⁴¹  Obstetrics and gynecology is further promoting holistic review via development of new application review metrics, an applicant compatibility index, and an early match program.¹⁵¹ 

Despite myriad proposed reforms, changes to the resident selection process have occurred piecemeal in single specialties. Articles in specialty-specific journals and lack of a common language impede widespread change. The fragmented nature of graduate medical education, both within and across specialties, further hinders progress. Additionally, multiple stakeholders (eg, AAMC, NRMP, San Francisco Match, Urology Match) are involved in the resident selection process and their agreement is requisite for national change. The Coalition for Physician Accountability has convened a cross-organization committee for this purpose, with recommendations expected in Spring 2021.¹⁵⁵  Careful examination of applicant and program experiences as well as match outcomes following implementation of these reforms is imperative to inform future directions.

This scoping review has several limitations, namely the potential exclusion of reforms published in editorials or commentaries without a title or abstract, rendering them difficult to identify via database queries. This likelihood was minimized by manual searching of leading undergraduate and graduate medical education journals and bibliographies of included studies. Additionally, articles describing novel reforms that lacked generalizability were excluded.

This scoping review characterized proposed reforms to the US resident selection process, developing a common language and framework to facilitate national conversations and change. The COVID-19 pandemic prompted many specialties to implement novel reforms identified in this review.

The authors would like to thank Lily Martin, MLIS, Icahn School of Medicine at Mount Sinai, for her assistance in designing the search strategies.