ABSTRACT

Background

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

Objective

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

Methods

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

Results

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

Conclusions

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Introduction

Calls for substantive reforms to the US resident selection process are growing, given increasing competition and inefficiencies of the current system.1  Over the last decade, applicants have doubled the number of applications they submit.24  Inundated by applications, programs are increasingly reliant on filters such as United States Medical Licensing Examination (USMLE) Step 1 scores for screening, despite its bias against minorities and poor prediction of residency performance.58  With Step 1 transitioning to pass/fail in 2022, programs may utilize Step 2 Clinical Knowledge (CK) for screening in the absence of other reforms.9 

Even if programs could conduct a holistic review of all applications, limited standardized data are available for review: clerkship grading distributions vary between institutions,10  Medical Student Performance Evaluations (MSPEs) lack standardized objective measures,11,12  and narrative letters of recommendation are subjective and omit applicant shortcomings.13  Once interview offers are extended, a “first-come, first-served” frenzy occurs among applicants to secure an interview,4,14  often with competitive applicants hoarding interviews only to cancel last minute.14,15  Couples, osteopathic (DO) graduates, and international medical graduates (IMG) face additional barriers.16 

Numerous reforms have been proposed to improve the efficiency, equity, and integrity of the resident selection process. However, many proposals are published in specialty-specific journals, and a comprehensive compilation of all reforms is currently lacking.4,17,18  Therefore, this scoping review was conducted to characterize systems-level reforms to the resident selection process. We aim to provide educational leaders with a clear framework and consistent language to facilitate national discussions.

Methods

The protocol was drafted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) and prospectively registered with the Open Science Framework on February 20, 2020.19 

Search Strategy

Designed by a health science librarian, comprehensive searches of the Embase, MEDLINE, Scopus, and Web of Science databases were conducted in February 2020 for articles published from January 2005 to February 2020 (provided as online supplementary data). In an attempt to capture all reforms presented in editorials, commentaries, and letters, the table of contents of the following undergraduate and graduate medical education journals were manually searched: Academic Medicine, Journal of Graduate Medical Education, Advances in Health Sciences Education, BMC Medical Education, Teaching and Learning in Medicine, and Medical Teacher. Furthermore, bibliographies of included articles were manually searched for additional relevant articles. All records were imported into Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia), and duplicates were removed.

Inclusion and Exclusion Criteria

English language articles proposing systems-level reforms to the US residency application, selection, and match process were included if applicable to all applicants, medical schools, or residency programs, regardless of specialty. Articles were excluded if no reform was proposed, a previously proposed reform was mentioned without explicit endorsement, the reform referenced a resident selection process outside the United States, or the reform was not generalizable beyond the individual program (eg, program-specific resident selection protocols, hiring independent contractors to assess program culture). Articles proposing reforms to the fellowship match process were also excluded.

Screening and Full-Text Review

References were independently screened by 2 authors (R.Z., D.L., or J.BR.) for inclusion based on their title and abstract. Articles then underwent full-text review by 2 reviewers (R.Z., D.L.). A third reviewer (J.BR.) reconciled discordant cases at screening and full-text review.

Data Extraction and Synthesis

Two reviewers (R.Z., D.L.) extracted data in duplicate. Extracted data included the type of reform, implementation strategies, cited advantages and disadvantages of the proposed reform, and the specialty targeted by the reform as suggested by the journal in which the article was published or authors' affiliations. Articles were classified as “not specialty specific” if authors were from multiple specialties or if the proposed reform was explicitly applicable to multiple specialties.

Qualitative content analysis using an inductive approach with grounded theory was performed to categorize reforms.20  First, 2 reviewers independently read a subset of included articles and created preliminary codes. Reviewers then reconciled their findings, refined preliminary codes, and generated a codebook. The process was repeated for subsequent articles in multiple groupings, creating new codes as needed. Finally, similar codes were categorized into higher-order categories and themes. ATLAS.ti 8 (ATLAS.ti Scientific Software Development, Berlin, Germany) was used to aid in higher-level category creation and assessment of code co-occurrences.

Results

Of the 10 407 unique references identified, 225 proceeded to full-text review and 116 were included in the scoping review (Figure). Though many references were not specialty-specific (n = 38), otolaryngology (n = 22), orthopaedic surgery (n = 16), general surgery (n = 8), emergency medicine (n = 7), and plastic surgery (n = 5) were the specialties most frequently proposing reforms (Figure provided as online supplementary data). Over the study period, there was a steady increase in the number of articles proposing reforms each year (Figure provided as online supplementary data).

Figure

Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Flow Diagram of Search Strategy

Figure

Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Flow Diagram of Search Strategy

Our inductive approach generated 34 codes that were grouped into 14 categories and then organized according to broad stages of the resident selection process: application submission, application review, interviews, and the Match (Table 1). Additionally, the pros and cons of each reform, as reported by the reviewed articles, were compiled and summarized (Table 2).

Table 1

Proposed Systems-Level Reforms to the US Resident Selection Process From 116 Articles Meeting Inclusion Criteria (January 2005–February 2020)

Proposed Systems-Level Reforms to the US Resident Selection Process From 116 Articles Meeting Inclusion Criteria (January 2005–February 2020)
Proposed Systems-Level Reforms to the US Resident Selection Process From 116 Articles Meeting Inclusion Criteria (January 2005–February 2020)
Table 2

Summation of Pros and Cons of Proposed Systems-Level Reforms to US Resident Selection Process, As Reported by Authors of 116 Articles (January 2005–February 2020)

Summation of Pros and Cons of Proposed Systems-Level Reforms to US Resident Selection Process, As Reported by Authors of 116 Articles (January 2005–February 2020)
Summation of Pros and Cons of Proposed Systems-Level Reforms to US Resident Selection Process, As Reported by Authors of 116 Articles (January 2005–February 2020)

Reforms to the Application Submission Process

Application Cap: 

Twenty-eight articles endorsed an application cap.4,15,18,2145  This was the most frequently proposed reform and implementation strategies varied. Many recommended fixed caps for all specialties, but there was no consensus on the optimal cap number.4,15,18,2141  Others suggested variable caps based on the specific-specialty and/or applicant metrics,4143  or “soft” caps imposed by resource-intensive supplemental applications or higher Electronic Residency Application Service (ERAS) fees, without explicitly setting limits.41,44,45  The cited advantages of application caps include cost-savings to applicants and decreased application volume for programs, presumably facilitating holistic review.5,24,2628,39  Challenges include determining the optimal cap and limited program data currently available to inform applicant decisions.40 

Signaling Program Preference: 

Eleven articles promoted supplemental applications to express program preference, including a program-specific paragraph in the personal statement, written or video statements of interest, or secondary applications.4,18,2628,40,41,4649  This proposal frequently co-occurred with application caps (n = 7) as a mechanism for reducing application volume through their time-intensive nature.24,50,51  However, requiring applicants to submit these materials results in an increase in resources needed for application review.52  Similarly, 5 articles endorsed the creation of a “signaling” system within ERAS or a separate, third-party platform, allowing applicants to designate a specified number of “preferred” programs.24,44,5052  Such limited preference signaling makes applicant interest explicit and may facilitate holistic review by residency programs,44  but may increase applicant costs via third-party servicing fees.50 

Standardized Program Database: 

Twenty-one articles proposed the creation of a database with standardized program information4,5,15,17,18,24,2729,33,4143,50,5360  beyond data currently captured in the American Medical Association FREIDA Tool61  and Association of American Medical Colleges (AAMC) Residency Explorer Tool.62  Data captured may include program information (eg, curriculum, case logs, research opportunities, graduate outcomes), screening criteria (eg, USMLE scores, AOA status, DO/IMG status, publications), and metrics of previously matched applicants (eg, National Resident Matching Program [NRMP] statistical profiles). This proposal frequently co-occurred with application caps (n = 10) and supplemental applications (n = 4), as these reforms likely assist applicants in identifying suitable programs. Additionally, 6 articles suggested residency programs establish a “brand” (language used in the articles) that conveys the culture, desired attributes of residents, and ideal career path for graduates, thereby informing applicants of the type of candidate likely to thrive in that culture.13,17,33,48,63,64  This reform may enhance the compatibility between applicants and programs, improving resident satisfaction, and decreasing residents leaving the program.64 

Reforms to the Application Review Process

Holistic Review: 

Twelve articles advocated for holistic review with balanced consideration of academic performance, extracurricular activities, and personal experiences.18,40,41,60,6572  One article promoted blinding Step 1 scores,68  and another encouraged the removal of specialty board passage rates as part of the Accreditation Council for Graduate Medical Education accreditation of residency programs to facilitate holistic review.18  Additionally, 2 articles encouraged the creation of national norms to holistically quantify and compare applicant accomplishments.13,65  Holistic review may increase diversity and improve compatibility between applicants and programs, but requires additional resources and is subjective.40,60,70  To mitigate these issues, 3 articles suggested standardized holistic applicant scoring generated from a weighted rubric.13,73,74 

Medical School Grading, Examinations, and Metrics: 

Two articles recommended nationally standardized grading to address grade inflation and facilitate applicant comparison.10,75  However, uniform grading is difficult to achieve across all institutions given differing grading schemas (eg, pass/fail or graded) and inconsistent language between—and even within—institutions depending upon the rotation or course.10,75  Four articles also proposed the creation of new medical school assessments, including competency-based evaluations, simulations, entrustable professional activities, and gateway exercises that serve as knowledge and skill checkpoints.4,18,66,76  These assessments permit longitudinal evaluation of applicant performance, identify applicant strengths and weaknesses, and provide a common framework for applicant assessments.47,66  However, concerns regarding validity coupled with variability in assessment utilization, learning objectives, and grading between institutions may preclude direct applicant comparisons.66  Finally, one article advocated for new medical school metrics capturing personal and professional experiences in a standardized manner to permit easier comparison of applicants.5 

Residency Evaluations and Metrics: 

One article promoted the creation of a general residency selection examination, separate from the USMLE series, that is intentionally designed to assess the knowledge and skills necessary for residency.17  Similarly, 3 articles endorsed the creation of new specialty-specific examinations and metrics generated from customized National Board of Medical Examiners specialty tests or aptitude tests to capture specialty knowledge and skills.65,68,77  Though a new residency selection examination or novel specialty-specific examinations/metrics may facilitate holistic review by deemphasizing Step 1 and Step 2 CK, they subject applicants to additional tests and require costly development and validation.68 

Noncognitive Assessments: 

Twenty-three articles promoted introduction of noncognitive assessments in resident selection, including personality assessments (n = 10),17,72,7885  and other psychological assessments including situational judgement tests (n = 5),17,64,72,83,85  emotional intelligence tests (n = 2),83,86  grit assessments (n = 2),84,87  moral reasoning tests (n = 1),67  and unspecified behavior-based questionnaires (n = 4).13,8890  These measures may provide insight into applicants' future behavior in residency and are customizable, allowing programs to identify applicants with the best compatibility for their unique culture, thereby reducing resident attrition.7880,8284  However, these assessments may be costly to validate and implement, and results may be skewed by social desirability bias and the Hawthorne effect.78,82,86 

MSPE: 

Twelve articles proposed requiring objective data in the MSPE, particularly class rank, clerkship grades, and shelf examination scores.4,11,12,18,31,39,59,9195  Proponents also sought disclosure of professionalism issues, academic difficulties, and leaves of absence. Additionally, 8 articles endorsed standardization of the MSPE structure, content, and language.11,12,59,60,70,9395  These 2 reforms co-occurred 5 times, with the common feature being facilitation of efficient applicant comparisons. Other modifications included third-party MSPE review to ensure institutional compliance with AAMC guidelines as these are currently unenforced.5 

Standardized Letters of Recommendation or Evaluation: 

Twenty articles supported standardized letters of recommendation (SLOR) or evaluation (SLOE) to uniformly summarize applicants' academic and professional potential for a given specialty.11,13,17,27,31,48,60,90,96107  Despite their interpretability, comparability, and objectivity, SLORs/SLOEs may be undermined by grade inflation.27,48,90,99,101,104  Consequently, 3 articles proposed the creation of nationally normed SLOR rubrics and a database comparing letter writer and program performance to national standards to curtail inflation.27,90,106  Similarly, one article advocated for published letter writer guidelines detailing the preparation of SLORs/SLOEs, particularly the assignment of appropriate grading or ranking.107  Finally, one article endorsed visual letters of recommendation (eg, word clouds), utilizing terms from the SLOR to enhance review efficiency.108 

Reforms to the Interview Process

Pre-Interview Screening: 

Thirteen articles proposed pre-interview screening via standardized video interviews,66,69,88,109,110  preliminary videoconference or telephone interviews,21,26,57,64,68,76,111  or regional interviews29,112  prior to on-site interviews. Such screening practices enable early assessment of applicant/program compatibility and reduce the interview cohort size, delivering cost-savings to applicants and programs alike.57,64,109,112  However, concerns exist regarding the scalability and increased administrative burden of pre-interview screening.88,109 

Interview Allocation and Scheduling: 

Seven articles suggested date standardization for interview offers with an acceptance window, creating a predictable timeline for applicants and minimizing clinical distractions.4,13,22,53,113,114  Alternatively, one article promoted online interview scheduling to preserve “first-come, first-served” acceptances,57  and one article proposed an interview match in which applicants and programs submit rank lists for interviews and are then “matched” to interview.115  More radically, 2 articles proposed an interview lottery, with one assigning interview dates from a rank lists of preferred dates,114  and the other randomly filling 50% of interview positions with applicants meeting minimum criteria.31 

Additionally, 6 articles supported an interview cap that limits the number of interviews an applicant can accept and attend.21,29,76,113,116,117  An interview cap may facilitate more equitable interview allocation and decrease costs, but the optimal limit is unclear given variable applicant circumstances and specialty competitiveness.14,76,117  Five articles promoted date standardization for interviews by specialty or region to decrease costs and serve as a de facto interview cap that limits interview overlap by the same top-tier applicants.29,59,76,116,118 

Interview Day: 

Eleven articles promoted structured interviews utilizing standardized job-related and behavior-based questions that are scored with a rubric.13,17,18,63,64,73,85,86,89,119,120  Structured interviews enable multidimensional assessment and improve efficiency and interrater reliability, but require interviewer training and the development of validated questions and scoring rubrics.13,86,119  Additionally, one article proposed virtual reality scenarios involving multiple applicants to assess real-time communication and problem-solving skills,89  and another promoted the use of skills-based simulations on interview day to uniformly assess technical abilities and knowledge base.121 

Post-Interview Communication: 

Six studies supported a ban on post-interview communication (eg, calls, emails, and “second looks”) and creation of an anonymous reporting system for violations.56,113,117,122124  This ban may enhance the integrity of the resident selection process and minimize undue influence on applicant rank lists.14,122,124  Recognizing the challenges of a moratorium, one article suggested allowing post-interview commitments with the requirement of written documentation to improve accountability.125 

Reforms to the Match Process

Eight articles proposed multiple match rounds with varying application caps per round.18,2224,34,35,126,127  By limiting the number of applications received, this approach facilitates an in-depth review of truly interest applicants and may allocate interviews more equitably, as early matching of competitive applicants affords interview opportunities for other applicants in subsequent rounds.24,34  Six articles supported an early assurance match, including guaranteed residency positions as a condition of medical school acceptance, “pre-matching” to home programs or programs where subinternships are performed, or allowing early acceptance to a consortium of institutions.17,29,47,70,126,127  This reform emphasizes knowledge and skill acquisition during medical school rather than residency securement and may improve the undergraduate to graduate medical education transition.4,29,70,126  However, moving up the resident selection decision may unnecessarily increase emphasis on the medical school an applicant attends.70,126  Finally, one study endorsed a “free market” approach in which graduation-eligible applicants interview and accept residency positions on a rolling basis.128  This approach may expedite medical training, but a continual reassessment of graduation readiness is burdensome, and rolling offers may pressure applicants into making decisions with incomplete knowledge.128,129 

Discussion

Calls for resident selection reform grew over the past 15 years, particularly among competitive specialties such as otolaryngology and orthopaedic surgery. This is likely in response to increasing application volume and applicant competitiveness with limited comparative metrics. Many popular reforms, including application caps, supplemental applications, and standardized letters of evaluation, seem to benefit both applicants and programs via application reduction and efficient applicant comparisons, facilitating holistic review.

Prior to the 2020–2021 application cycle, specialties implemented several reforms identified in this scoping review. For instance, emergency medicine instituted the SLOE in 1997 for efficient applicant comparison.91  Though initially hamstrung by grade inflation, a concerted effort to create a national cohort of experienced authors addressed this limitation, and the SLOE remains a mainstay of the emergency medicine resident selection process.106  Likewise, emergency medicine partnered with the AAMC from 2017–2020 to pilot the standardized video interview, an asynchronous online interview that assesses applicants' professionalism and interpersonal communication skills.110  However, concerns regarding its validity, cost, and negative applicant perceptions prompted discontinuation of the standardized video interview.130  Additionally, otolaryngology mandated a program-specific paragraph in the personal statement in 2015 to gauge applicant interest and deter application inflation.46  Program-specific paragraphs were found to be generically crafted,46  and seemingly exacerbated a downturn in the number of otolaryngology applicants, resulting in them now being optional.131,132 

The COVID-19 pandemic has affected the residency application process in numerous ways,133136  accelerating calls for disruptive innovation and affording opportunities for novel reform.1,137141  Application caps were the most frequently proposed reform (n = 28 articles), and many specialties have recommended limits for the 2020–2021 cycle.138,142  Some have suggested deriving application caps from the AAMC Apply Smart tool, which correlates application data with specialty entrance rates to identify the point of diminishing returns for application submissions stratified by applicant type (MD/DO/IMG) and Step 1 score tertile.143  However, several methodologic concerns exist regarding the calculation and use of specialty entrance rates rather than match rates as well as the applicability of the data to individual applicants.144  Moreover, the acceptability of application caps varies by applicant and specialty,24,145  and unenforced recommendations are unlikely to alter applicant behavior.

In contrast to applications, an applicant's number of contiguous ranks, a proxy for the number of interviews attended, correlates well with their match rate.146  A small cohort of highly competitive applicants accept and attend a disproportionate number of interviews, and virtual interviews are likely to exacerbate this maldistribution by removing cost and travel constraints.147  Calls for interview caps are growing,148,149  and ophthalmology utilized a centralized scheduling platform to limit applicants to 20 virtual inteviews.150  Likewise, obstetrics and gynecology, orthopaedic surgery, and dermatology implemented standardized interview offer dates with acceptance windows, creating a predictable timeline.151153 

In addition, otolaryngology implemented preference signaling, permitting applicants 5 signals each to explicitly designate their interest in programs.154  Coupling preference signaling with a comprehensive database of residency program information, as is underway by ophthalmology and obstetrics and gynecology, assists applicants in identifying suitable programs.150,151  Other proposed strategies such as pre-interview screening provide innovative approaches for narrowing the applicant pool before full virtual interviews, but risk introducing an unvalidated metric susceptible to implicit bias. Finally, 9 specialties promoted holistic review, which is easy to suggest, but difficult to achieve given that graduate medical education programs are under-resourced for the current application volume.141  Obstetrics and gynecology is further promoting holistic review via development of new application review metrics, an applicant compatibility index, and an early match program.151 

Despite myriad proposed reforms, changes to the resident selection process have occurred piecemeal in single specialties. Articles in specialty-specific journals and lack of a common language impede widespread change. The fragmented nature of graduate medical education, both within and across specialties, further hinders progress. Additionally, multiple stakeholders (eg, AAMC, NRMP, San Francisco Match, Urology Match) are involved in the resident selection process and their agreement is requisite for national change. The Coalition for Physician Accountability has convened a cross-organization committee for this purpose, with recommendations expected in Spring 2021.155  Careful examination of applicant and program experiences as well as match outcomes following implementation of these reforms is imperative to inform future directions.

This scoping review has several limitations, namely the potential exclusion of reforms published in editorials or commentaries without a title or abstract, rendering them difficult to identify via database queries. This likelihood was minimized by manual searching of leading undergraduate and graduate medical education journals and bibliographies of included studies. Additionally, articles describing novel reforms that lacked generalizability were excluded.

Conclusions

This scoping review characterized proposed reforms to the US resident selection process, developing a common language and framework to facilitate national conversations and change. The COVID-19 pandemic prompted many specialties to implement novel reforms identified in this review.

The authors would like to thank Lily Martin, MLIS, Icahn School of Medicine at Mount Sinai, for her assistance in designing the search strategies.

References

1. 
Hammoud
MM,
Standiford
T,
Carmody
B.
Potential implications of COVID-19 for the 2020-2021 residency application cycle
.
JAMA
.
2020
;
324
(1)
:
29
30
.
2. 
National Resident Matching Program.
Results of the 2011 NRMP applicant survey by preferred specialty and applicant type
.
2021
.
3. 
National Resident Matching Program.
Results of the 2019 NRMP applicant survey by preferred specialty and applicant type
.
2021
.
4. 
Pereira
AG,
Chelminski
PR,
Chheda
SG,
et al
Application inflation for internal medicine applicants in the match: drivers, consequences, and potential solutions
.
Am J Med
.
2016
;
129
(8)
:
885
891
.
5. 
Katsufrakis
PJ,
Chaudhry
HJ.
Improving residency selection requires close study and better understanding of stakeholder needs
.
Acad Med
.
2019
;
94
(3)
:
305
308
.
6. 
Williams
M,
Kim
EJ,
Pappas
K,
et al
The impact of United States Medical Licensing Exam (USMLE) Step 1 cutoff scores on recruitment of underrepresented minorities in medicine: a retrospective cross-sectional study
.
Health Sci Rep
.
2020
;
3
(2)
:
1
8
.
7. 
Wagner
JG,
Schneberk
T,
Zobrist
M,
et al
Association between resident performance, rank list position, and United States Medical Licensing Exam Step 1 scores
.
J Emerg Med
.
2017
;
52
(3)
:
332
340
.
8. 
Harfmann
KL,
Zirwas
MJ.
Can performance in medical school predict performance in residency? A compilation and review of correlative studies
.
J Am Acad Dermatol
.
2011
;
65
(5)
:
1010
1022
.
9. 
Makhoul
AT,
Pontell
ME,
Kumar
NG,
Drolet
BC.
Objective measures needed—program directors' perspectives on a pass/fail USMLE Step 1
.
N Engl J Med
.
2020
;
382
(25)
:
2389
2392
.
10. 
Westerman
ME,
Boe
C,
Bole
R,
et al
Evaluation of medical school grading variability in the United States: are all honors the same?
Acad Med
.
2019
;
94
(12)
:
1939
1945
.
doi:10.1097/ACM. 0000000000002843
11. 
Andolsek
KM.
Improving the medical student performance evaluation to facilitate resident selection
.
Acad Med
.
2016
;
91
(11)
:
1475
1479
.
12. 
Hom
J,
Richam
I,
Hall
P,
et al
The state of medical student performance evaluations: improved transparency or continued obfuscation
.
Acad Med
.
2016
;
91
(11)
:
1534
1539
.
13. 
Lee
AG,
Golnik
KC,
Oetting
TA,
et al
Re-engineering the resident application selection process in ophthalmology: a literature review and recommendations for improvement
.
Surv Ophthalmol
.
2008
;
53
(2)
:
164
176
.
14. 
Frush
BW,
Byerley
J.
High-value interviewing: a call for quality improvement in the match process
.
Acad Med
.
2019
;
94
(3)
:
324
327
.
15. 
Christophel
JJ,
Levin
PA.
Too much of a good thing
.
JAMA Otolaryngol Head Neck Surg.
2014
;
140
(4)
291
292
.
16. 
Cummings
M.
Osteopathic students' graduate medical education aspirations versus realities: the relationship of osteopathic medicine and primary care
.
Acad Med
.
2016
;
91
(1)
:
36
41
.
17. 
Andolsek
KM.
One small step for Step 1
.
Acad Med
.
2019
;
94
(3)
:
309
313
.
18. 
Berger
JS,
Cioletti
A.
Viewpoint from 2 graduate medical education deans: application overload in the residency match process
.
J Grad Med Educ
.
2016
;
8
(3)
:
317
321
.
19. 
Tricco
AC,
Lillie
E,
Zarin
W,
et al
PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation
.
Ann Intern Med
.
2018
;
169
(7)
:
467
473
.
20. 
Cho
JY,
Lee
EH.
Reducing confusion about grounded theory and qualitative content analysis: similarities and differences
.
Qual Rep
.
2014
;
19
(32)
:
1
20
.
21. 
Gardner
AK,
Smink
DS,
Scott
BG,
Korndorffer
JR
Jr,
Harrington
D,
Ritter
EM.
How much are we spending on resident selection?
J Surg Educ
.
2018
;
75
(6)
:
e85
e90
.
22. 
Arnold
L,
Sullivan
C,
Okah
FA.
A free-market approach to the Match: a proposal whose time has not yet come
.
Acad Med
.
2018
;
93
(1)
:
16
19
.
23. 
Hueston
WJ.
A proposal to address the increasing number of residency applications
.
Acad Med
.
2017
;
92
(7)
:
896
897
.
24. 
Ward
M,
Pingree
C,
Laury
AM,
Bowe
SN.
Applicant perspectives on the otolaryngology residency application process
.
JAMA Otolaryngol Head Neck Surg
.
2017
;
143
(8)
:
782
787
.
25. 
Nasreddine
AY,
Gallo
R.
Applying to orthopaedic residency and matching rates: analysis and review of the past 25 years
.
J Bone Joint Surg Am
.
2019
;
101
(24)
:
e134
.
26. 
Li
NY,
Gruppuso
PA,
Kalagara
S,
Eltorai
AEM,
DePasse
JM,
Daniels
AH.
Critical assessment of the contemporary orthopaedic surgery residency application process
.
J Bone Joint Surg Am
.
2018
;
101
(21)
:
e114
.
27. 
Trikha
R,
Keswani
A,
Ishmael
CR,
Greig
D,
Kelley
BV,
Bernthal
NM.
Current trends in orthopaedic surgery residency applications and match rates
.
J Bone Joint Surg Am
.
2020
;
102
(6)
:
e24
.
28. 
Naclerio
RM,
Pinto
JM,
Baroody
FM.
Drowning in applications for residency training: a program's perspective and simple solutions
.
JAMA Otolaryngol Head Neck Surg
.
2014
;
140
(8)
:
695
696
.
29. 
Agarwal
N,
Choi
PA,
Okonkwo
DO,
Barrow
DL,
Friedlander
RM.
Financial burden associated with the residency match in neurological surgery
.
J Neurosurg
.
2017
;
126
(1)
:
184
190
.
30. 
Kraeutler
MJ.
It is time to change the status quo: limiting orthopedic surgery residency applications
.
Orthopedics
.
2017
;
40
(5)
:
267
268
.
31. 
Bernstein
J.
Not the last word: roll them bones—selecting orthopaedic surgery residents by lottery
.
Clin Orthop Relat Res
.
2019
;
477
(12)
:
2635
2638
.
32. 
Weissbart
SJ,
Kim
SJ,
Feinn
RS,
Stock
JA.
Relationship between the number of residency applications and the yearly match rate: time to start thinking about an application limit?
J Grad Med Educ
.
2015
;
7
(1)
:
81
85
.
33. 
Zhao
H,
Freedman
A,
Lerman
S.
Reforming the urology match application process: a role for the residency programs
.
J Urol
.
2020
;
203
(1)
:
44
45
.
34. 
Monir
JG.
Reforming the Match: a proposal for a new 3-phase system
.
J Grad Med Educ
.
2020
;
12
(1)
:
7
9
.
35. 
London
DA.
SOAP for everyone: an evolutionary development of the Match
.
Acad Med
.
2017
;
92
(6)
:
730
.
36. 
Burbano
FM,
Yao
A,
Burish
N,
et al
Solving congestion in the plastic surgery match: a game theory analysis
.
Plast Reconstr Surg
.
2019
;
143
(2)
:
634
639
.
37. 
Baroody
FM,
Pinto
JM,
Naclerio
RM.
Otolaryngology (urban) legend: the more programs to which you apply, the better the chances of matching
.
Arch Otolaryngol Head Neck Surg
.
2008
;
134
(10)
:
1038
.
38. 
Weissbart
SJ,
Hall
SJ,
Fultz
BR,
Stock
JA.
The urology match as prisoner's dilemma: a game theory perspective
.
Urology
.
2013
;
82
(4)
:
791
797
.
39. 
Weissbart
SJ,
Stock
JA,
Wein
AJ.
Challenges facing program directors in the urology match
.
Urol Pract
.
2016
;
3
(6)
:
486
492
.
40. 
Putnam-Pite
D.
Viewpoint from a former medical student/now intern playing the game—balancing numbers and intangibles in the orthopedic surgery match
.
J Grad Med Educ
.
2016
;
8
(3)
:
311
313
.
41. 
Katsufrakis
PJ,
Uhler
TA,
Jones
LD.
The residency application process: pursuing improved outcomes through better understanding of the issues
.
Acad Med
.
2016
;
91
(11)
:
1483
1487
.
42. 
Kaplan
AB,
Riedy
KN,
Grundfast
KM.
Increasing competitiveness for an otolaryngology residency: where we are and concerns about the future
.
Otolaryngol Head Neck Surg
.
2015
;
153
(5)
:
699
701
.
43. 
Deng
F,
Chen
JX,
Wesevich
A.
More transparency is needed to curb excessive residency applications
.
Acad Med
.
2017
;
92
(7)
:
895
896
.
44. 
Whipple
ME,
Law
AB,
Bly
RA.
A computer simulation model to analyze the application process for competitive residency programs
.
J Grad Med Educ
.
2019
;
11
(1)
:
30
35
.
45. 
Stoddard
HA.
More about the role of USMLE Step 1 scores in resident selection
.
Acad Med
.
2016
;
91
(11)
:
1469
1470
.
46. 
Stepan
KO,
Kaul
VF,
Raquib
AR,
et al
An evaluation of the program-specific paragraph in the otolaryngology residency application
.
Laryngoscope
.
2018
;
128
(11)
:
2508
2513
.
47. 
Pereira
AG,
Williams
CM,
Angus
SV.
Disruptive innovation and the residency Match: the time is now
.
J Grad Med Educ
.
2019
;
11
(1)
:
36
38
.
48. 
Porter
SE,
Razi
AE,
Ramsey
TB.
Novel strategies to improve resident selection by improving cultural fit: AOA critical issues
.
J Bone Joint Surg Am
.
2017
;
99
(22)
:
e120
.
49. 
Puscas
L,
Esclamado
R.
Use of a secondary essay in the residency application process
.
JAMA Otolaryngol Head Neck Surg
.
2015
;
141
(7)
:
591
592
.
50. 
Bernstein
J.
Not the last word: want to match in an orthopaedic surgery residency? Send a rose to the program director
.
Clin Orthop Relat Res
.
2017
;
475
(12)
:
2845
2849
.
51. 
Chen
JX,
Deng
F,
Gray
ST.
Preference signaling in the national resident matching program
.
JAMA Otolaryngol Head Neck Surg
.
2018
;
144
(10)
:
951
.
52. 
Salehi
PP,
Benito
D,
Michaelides
E.
A novel approach to the national resident matching program—the star system
.
JAMA Otolaryngol Head Neck Surg
.
2018
;
144
(5)
:
397
398
.
53. 
Kozin
ED,
Setthi
RV,
Lehmann
A,
et al
Analysis of an online match discussion board: improving the otolaryngology-head and neck surgery match
.
Otolaryngol Head Neck Surg
.
2015
;
152
(3)
:
458
464
.
54. 
Cabrera-Muffly
C,
Chang
CWD,
Puscas
L.
Current interview trail metrics in the otolaryngology match
.
Otolaryngol Head Neck Surg
.
2017
;
156
(6)
:
1097
1103
.
55. 
Gupta
D,
Kumar
S.
ERAS: can it be revamped? One point of view
.
J Grad Med Educ
.
2016
;
8
(3)
:
467
.
56. 
Bernstein
J.
Not the last word: agonizing appropriately over the residency match rank list
.
Clin Orthop Relat Res
.
2018
;
476
(12)
:
2309
2312
.
57. 
Hariton
E,
Bortoletto
P,
Ayogu
N.
Residency interviews in the 21st century
.
J Grad Med Educ
.
2016
;
8
(3)
:
322
324
.
58. 
Susarla
SM,
Swanson
EW,
Slezak
S,
Lifchez
SD,
Redett
RJ.
The perception and costs of the interview process for plastic surgery residency programs: can the process be streamlined?
Plast Reconst Surg
.
2017
;
139
(1)
:
302
309
.
59. 
Aagaard
EM,
Abaza
M.
The residency application process—burden and consequences
.
N Engl J Med
.
2016
;
374
(4)
:
303
305
.
60. 
Gliatto
P,
Karani
R.
Viewpoint from 2 undergraduate medical education deans the residency application process: working well, needs fixing, or broken beyond repair?
J Grad Med Educ
.
2016
;
8
(3)
:
307
310
.
61. 
American Medical Association.
FREIDA
.
https://freida.ama-assn.org/. Accessed March 16,
2021
.
62. 
Association of American Medical Colleges.
Residency Explorer Tool
.
2021
.
63. 
Lee
WT,
Esclamado
RM,
Puscas
L.
Selecting among otolaryngology residency applicants to train as tomorrow's leaders
.
JAMA Otolaryngol Head Neck Surg
.
2013
;
139
(8)
:
770
771
.
64. 
Lyons
J,
Bingmer
K,
Ammori
J,
Marks
J.
Utilization of a novel program-specific evaluation tool results in a decidedly different interview pool than traditional application review
.
J Surg Educ
.
2019
;
76
(6)
:
e110
e117
.
65. 
Prober
CG,
Kolars
JC,
First
LR,
Melnick
DE.
A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection
.
Acad Med
.
2016
;
91
(1)
:
12
15
.
66. 
Radabaugh
CL,
Hawkins
RE,
Welcher
CM,
et al
Beyond the United States Medical Licensing Examination Score: assessing competence for entering residency
.
Acad Med
.
2019
;
94
(7)
:
983
989
.
67. 
Bohm
KC,
Van Heest
T,
Gioe
TJ,
Agel
J,
Johnson
TC,
Van Heest
A.
Assessment of moral reasoning skills in the orthopaedic surgery resident applicant
.
J Bone Joint Surg Am.
2014
;
96
(17)
e:151.
68. 
Chen
A,
Shinkai
K.
Rethinking how we select dermatology applicants—turning the tide
.
JAMA Dermatol
.
2017
;
153
(3)
:
259
260
.
69. 
King
A,
Mayer
C,
Starnes
C,
Barringer
K,
Beier
L,
Sule
H.
Using the Association of American Medical Colleges standardized video interview in a holistic residency application review
.
Cureus
.
2017
;
9
(12)
:
e1913
.
70. 
Moynahan
KF.
The current use of United States Medical Licensing Examination Step 1 scores: holistic admissions and student well-being are in the balance
.
Acad Med
.
2018
;
93
(7)
:
963
965
.
71. 
Jones
RL,
Burk-Rafel
J.
The Match: a numbers game
.
Acad Med
.
2017
;
92
(6)
:
731
.
72. 
Bowe
SN,
Schmalbach
CE,
Laury
AM.
The state of the otolaryngology match: a review of applicant trends, “impossible” qualifications, and implications
.
Otolaryngol Head Neck Surg
.
2017
;
156
(6)
:
985
990
.
73. 
Martin
M,
Salzberg
L.
Resident characteristics to evaluate during recruitment and interview. A Delphi study
.
Educ Prim Care
.
2017
;
28
(2)
:
81
85
.
74. 
Turner
NS,
Shaughnessy
WJ,
Berg
EJ,
Larson
DR,
Hanssen
AD.
A quantitative composite scoring tool for orthopaedic residency screening and selection
.
Clin Orthop Relat Res
.
2006
;
449
:
50
55
.
75. 
Weissman
S.
Why reforms must be made to make the dean's letter useful
.
Acad Med
.
2013
;
88
(4)
:
434
.
76. 
Gruppuso
PA,
Adashi
EY.
Residency placement fever: is it time for a reevaluation?
Acad Med
.
2017
;
92
(7)
:
923
926
.
77. 
Bernstein
J.
Not the last word: ending the residency application arms race—starting with the USMLE
.
Clin Orthop Relat Res
.
2016
;
474
(12)
:
2571
2576
.
78. 
Lubelski
D,
Healy
AT,
Friedman
A,
Ferraris
D,
Benzel
EC,
Schlenk
R.
Correlation of personality assessments with standard selection criteria for neurosurgical residency applicants
.
J Neurosurg
.
2016
;
125
(4)
:
986
994
.
79. 
Tornetta
P
III,
Jacobs
JJ,
Sterling
RS,
Kogan
M,
Fletcher
KA,
Friedman AMl. Personality assessment in orthopaedic surgery. AOA critical issues
.
J Bone Joint Surg Am
.
2019
;
101
(4)
:
e13
.
80. 
Phillips
D,
Egol
KA,
Maculatis
MC,
et al
Personality factors associated with resident performance: results from 12 Accreditation Council for Graduate Medical Education accredited orthopaedic surgery programs
.
J Surg Educ
.
2018
;
75
(1)
:
122
131
.
81. 
Merlo
LJ,
Matveevskii
AS.
Personality testing may improve resident selection in anesthesiology programs
.
Med Teach
.
2009
;
31
(12)
:
e551
e554
.
82. 
Friedman
AM.
Using organizational science to improve the resident selection process: an outsider's perspective
.
Am J Med Qual
.
2016
;
31
(5)
:
486
488
.
83. 
Gardner
AK,
Dunkin
BJ.
Evaluation of validity evidence for personality, emotional intelligence, and situational judgement tests to identify successful residents
.
JAMA Surg
.
2018
;
153
(5)
:
409
416
.
84. 
Hughes
BD,
Perone
JA,
Cummins
CB,
et al
Personality testing may identify applicants who will become successful in general surgery residency
.
J Surg Res
.
2019
;
333
:
240
248
.
85. 
Gardner
AK,
Grantcharov
T,
Dunkin
BJ.
The science of selection: using best practices from industry to improve success in surgery training
.
J Surg Educ
.
2018
;
75
(2)
:
278
285
.
86. 
Lin
DT,
Kannappan
A,
Lau
JN.
The assessment of emotional intelligence among candidates interviewing for general surgery residency
.
J Surg Educ
.
2013
;
70
(4)
:
514
521
.
87. 
Kurian
EB,
Desai
VS,
Turner
NS,
et al
Is grit the new fit? Assessing non-cognitive variables in orthopedic surgery trainees
.
J Surg Educ
.
2019
;
76
(4)
:
924
930
.
88. 
Breitkopf
DM,
Green
IC,
Hopkins
MR,
Torbenson
VE,
Camp
CL,
Turner
NS
III.
Use of asynchronous video interviews for selecting obstetrics and gynecology residents
.
Obstet Gynecol
.
2019
;
134
(suppl 1)
:
9
15
.
89. 
Crawford
SB,
Monks
SM,
Wells
RN.
Virtual reality as an interview technique in evaluation of emergency medicine applicants
.
AEM Educ Train
.
2018
;
2
(4)
:
328
333
.
90. 
Love
JN,
Smith
J,
Weizberg
M,
et al
Council of emergency medicine residency directors' standardized letter of recommendation: the program director's perspective.
2014
;
21
(6)
:
680
687
.
91. 
Benzinger
R.
From the desk of the program director: show me the rankings!
J Grad Med Educ
.
2014
;
6
(3)
:
413
414
.
92. 
Boysen-Osborn
M,
Mattson
J,
Yanuck
J,
et al
Ranking practice variability in the medical student performance evaluation: so bad, it's “good”
.
Acad Med
.
2016
;
91
(11)
:
1540
1545
.
93. 
Eisenberg
RL.
Suggestions for improving the medical student performance evaluation
.
Acad Med
.
2016
;
91
(11)
:
1468
.
94. 
Green
MM,
Sanguino
SM,
Thomas
JX
Jr.
Standardizing and improving the content of the dean's letter
.
Virtual Mentor
.
2012
;
14
(12)
:
1021
1026
.
95. 
Boysen-Osborn
M,
Yanuck
J,
Mattson
J,
et al
Who to interview? Low adherence by U.S. medical schools to medical student performance evaluation format makes resident selection difficult
.
West J Emerg Med
.
2017
;
18
(1)
:
50
55
.
96. 
Messner
A,
Teng
M,
Shimahara
E,
et al
A case for the standardized letter of recommendation in otolaryngology residency selection
.
Laryngoscope
.
2014
;
124
(1)
:
2
3
.
97. 
Wang
RF,
Zhang
M,
Alloo
A,
Stasko
T,
Miller
JE,
Kaffenberger
JA.
Characterization of the 2016-2017 dermatology standardized letter of recommendation
.
J Clin Aesthet Dermatol
.
2018
;
11
(3)
:
26
29
.
98. 
Coates
WC.
Choosing the right resident: implications of the new electronic emergency medicine standardized letter of evaluation
.
J Grad Med Educ
.
2019
;
11
(2)
:
187
188
.
99. 
Field
NC,
Gullick
MM,
German
JW.
Selection of neurological surgery applicants and the value of standardized letters of evaluation: a survey of United States program directors
.
World Neurosurg
.
2020
;
136
:
e342
e346
.
100. 
Perkins
JN,
Liang
C,
McFann
K,
Abaza
MM,
Streubel
SO,
Prager
JD.
Standardized letter of recommendation for otolaryngology residency selection
.
Laryngoscope
.
2013
;
123
(1)
:
123
133
.
101. 
Friedman
R,
Fang
CH,
Hasbun
J,
et al
Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender
.
Laryngoscope
.
2017
;
127
(12)
:
2738
2745
.
102. 
Bajwa
JM,
Yudkowsky
R,
Belli
D,
Vu
NV,
Park
YS.
Validity evidence for a residency admissions standardized assessment letter for pediatrics
.
Teach Learn Med
.
2018
;
30
(2)
:
173
183
.
103. 
Love
JN,
Ronan-Bentle
SE,
Lane
DR,
Hegarty
CB.
The standardized letter of evaluation for postgraduate training: a concept whose time has come?
Acad Med
.
2016
;
91
(11)
:
1480
1482
.
104. 
Kominsky
AH,
Bryson
PC,
Benninger
MS,
Tierney
WS.
Variability ratings in the otolaryngology standardized letter of recommendation
.
Otolaryngol Head Neck Surg
.
2016
;
154
(2)
:
287
293
.
105. 
Rajesh
A,
Rivera
M,
Asaad
M,
et al
What are we really looking for in a letter of recommendation?
J Surg Educ
.
2019
;
76
(6)
:
e118
e124
.
106. 
Jackson
JS,
Bond
M,
Love
JN,
Hegarty
C.
Emergency medicine standardized letter of evaluation (SLOE): findings from the new electronic SLOE format
.
J Grad Med Educ
.
2019
;
11
(2)
:
182
186
.
107. 
Kang
HP,
Robertson
DM,
Levine
WN,
Lieberman
JR.
Evaluating the standardized letter of recommendation form in applicants to orthopaedic surgery residency
.
J Am Acad Orthop Surg
.
2020
;
28
(19)
:
814
422
.
108. 
Bayrak
SB,
Villwock
JA,
Villwock
MR,
Chiu
AG,
Sykes
KJ.
Using word clouds to re-envision letters of recommendation for residency applicants
.
Laryngoscope
.
2019
;
129
(9)
:
2026
2030
.
109. 
Bird
SB,
Hern
HG,
Blomkalns
A,
et al
Innovation in residency selection: the AAMC standardized video interview
.
Acad Med
.
2019
;
94
(10)
:
1489
1497
.
110. 
Gallahue
FE,
Hiller
KM,
Bird
SB,
et al
The AAMC standardized video interview: reactions and use by residency programs during the 2018 application cycle
.
Acad Med
.
2019
;
94
(10)
:
1506
1512
.
111. 
Melendez
MM,
Dobryansky
M,
Alizadeh
K.
Live online video interviews dramatically improve the plastic surgery residency application process
.
Plast Reconstr Surg
.
2021
;
130
(1)
:
240
241
.
112. 
McMakin
KK,
Caputo
FJ,
Hoell
NG,
Trani
J,
Carpenter
JP,
Lombardi
JV.
Trends in the 10-year history of the vascular integrated residency match: more work, higher cost, same result
.
J Vasc Surg
.
2020
;
72
(1)
:
298
303
.
doi:101016/j.jvs.2019.10.066
113. 
Frush
BW,
Byerley
J.
High-value interviewing: a call for quality improvement in the Match process
.
Acad Med
.
2019
;
94
(3)
:
324
327
.
114. 
Luftig
D.
The residency interview scheduling process: unintended consequences and a proposal for change
.
J Grad Med Educ
.
2015
;
7
(1)
:
134
.
115. 
Melcher
ML,
Wapnir
I,
Ashlagi
I.
May the interview be with you: signal your preferences
.
J Grad Med Educ
.
2019
;
11
(1)
:
39
40
.
116. 
Bhalla
V,
Sykes
K,
Kraft
SM,
Chiu
AG.
Commentary on Bowe et al, “The state of the otolaryngology Match: a review of applicant trends, ‘impossible' qualifications, and implications”
.
Otolaryngol Head Neck Surg
.
2018
;
158
(2)
:
217
218
.
117. 
Wood
JS,
David
LR.
Outcome analysis of factors impacting plastic surgery match
.
Ann Plast Surg
.
2010
;
64
(6)
:
770
774
.
118. 
Claiborne
JR,
Crantford
JC,
Swett
KR,
David
LR.
The plastic surgery match: predicting success and improving the process
.
Ann Plast Surg
.
2013
;
70
(6)
:
698
703
.
119. 
Gardner
AK,
D'Onofrio
BC,
Dunkin
BJ.
Can we get faculty interviewers on the same page? An examination of a structured interview course for surgeons
.
J Surg Educ
.
2018
;
75
(1)
:
72
77
.
120. 
Nallasamy
S,
Uhler
T,
Nallasamy
N,
Tapino
PJ,
Volpe
NJ.
Ophthalmology resident selection: current trends in selection criteria and improving the process
.
Ophthalmology
.
2010
;
117
(5)
:
1041
1047
.
121. 
Cannon
K,
Hartsell
Z,
Ivanov
I,
et al
Perceptions of internal medicine residency program candidates on the use of simulation in the selection process
.
J Grad Med Educ
.
2014
;
6
(2)
:
338
340
.
122. 
Nagarkar
PA,
Janis
JE.
Fixing the “Match”: how to play the game
.
J Grad Med Educ
.
2012
;
4
(2)
:
142
147
.
123. 
Wu
AJ,
Vapiwala
N,
Chmura
SJ,
et al
Taking “the game” out of the Match: a simple proposal
.
Int J Radiat Oncol Biol Phys
.
2015
;
93
(5)
:
945
958
.
124. 
Ramkumar
PN,
Navarro
SM,
Chughtai
M,
Haeberle
HS,
Taylor
SA,
Mont
MA.
The orthopaedic surgery residency application process: an analysis of the applicant experience
.
J Am Acad Orthop Surg
.
2018
;
26
(15)
:
537
544
.
125. 
Chervenak
FA,
McCullough
LB,
Cefalo
R.
An ethical justification and policy for making commitments during computerized residency application processes: the matching program as a laboratory for needed reform
.
Adv Health Sci Educ Theory Pract
.
2011
;
16
(3)
:
427
433
.
126. 
Hammoud
MM,
Andrews
J,
Skochelak
SE.
Improving the residency application and selection process: an optional early result acceptance program
.
JAMA
.
2020
;
323
(6)
:
503
504
.
127. 
Wong
BJ.
Reforming the Match process—early decision plans and the case for a consortia match
.
JAMA Otolaryngol Head Neck Surg
.
2016
;
142
(8)
:
727
728
.
128. 
Ray
C,
Bishop
SE,
Dow
AW.
Rethinking the Match: a proposal for modern matchmaking
.
Acad Med
.
2018
;
93
(1)
:
45
47
.
129. 
Alvin
MD,
Magid
D.
Free-market approach to the Match: AN unfair and unfeasible proposal
.
Acad Med
.
2018
;
93
(5)
:
671
672
.
130. 
American Academy of Emergency Medicine.
Joint letter to the AAMC on the standardized video interview (SVI)
.
2021
.
131. 
Kramer
S.
Is the program-specific paragraph responsible for declining application numbers? A commentary
.
Otolaryngol Head Neck Surg
.
2018
;
158
(2)
:
215
216
.
132. 
Bowe
SN,
Schmalbach
CE,
Laury
AM.
Regarding “is the program-specific paragraph responsible for declining application numbers? A commentary.”
Otolaryngol Head Neck Surg
.
2018
;
158
(6)
:
1150
1151
.
133. 
Chretien
KC,
Raj
JM,
Abraham
RA,
et al
AAIM recommendations for the 2020-2021 internal medicine residency application cycle in response to the COVID-19 pandemic
.
Am J Med
.
2020
;
133
(10)
:
1223
1226
.
134. 
Aiyer
AA,
Granger
CJ,
McCormick
KL,
et al
The impact of COVID-19 on the orthopaedic surgery residency application process
.
J Am Acad Orthop Surg
.
2020
;
28
(15)
:
e633
e641
.
135. 
Rekawek
P,
Henry
A,
Moe
J,
Schlieve
T,
Panchal
N.
The COVID-19 pandemic: implications for the oral and maxillofacial surgery residency application process
[published online ahead of print July 22,
2020]
.
J Dent Educ. doi:10.1002/jdd.12310
136. 
Xie
DX,
Hillel
AT,
Ward
BK.
Otolaryngology residency match during the COVID-19 pandemic: What happens next?
[published online ahead of print June 4,
2020]
.
JAMA Otolaryngol Head Neck Surg. doi:10.1001/jamaoto.2020.1078
137. 
Harary
M,
Bergsneider
M.
Letter: approaches to mitigate impact of COVID-19 pandemic on neurosurgical residency application cycle
.
Neurosurgery
.
2020
;
87
(2)
:
e212
e213
.
138. 
Pelletier-Bui
A,
Franzen
D,
Smith
L,
et al
COVID-19: a driver for disruptive innovation of the emergency medicine residency application process
.
West J Emerg Med
.
2020
;
21
(5)
:
1105
1113
.
139. 
Haas
MRC,
He
S,
Sternberg
K,
et al
Reimagining residency selection: part 1—a practical guide to recruitment in the post-COVID-19 era
.
J Grad Med Educ
.
2020
;
12
(5)
:
539
544
.
140. 
Sternberg
K,
Jordan
J,
Haas
MRC,
et al
Reimagining residency selection: part 2—a practical guide to interviewing in the post-COVID-19 era
.
J Grad Med Educ
.
2020
;
12
(5)
:
545
549
.
141. 
Association of American Medical Colleges.
Specialty response to COVID-19
.
2021
.
142. 
Gabrielson
AT,
Kohn
JR,
Sparks
HT,
Clifton
MM,
Kohn
TP.
Proposed changes to the 2021 residency application process in the wake of COVID-19
.
Acad Med
.
2020
;
95
(9)
:
1346
1349
.
143. 
Association of American Medical Colleges.
Apply smart: Data to consider when applying to residency
.
2021
.
144. 
Carmody
JB.
Applying smarter: a critique of the AMMC apply smart tools
.
J Grad Med Educ
.
2020
;
12
(1)
:
10
13
.
145. 
Sweet
ML,
Williams
CM,
Stewart
E,
et al
Internal medicine residency program responses to the increase of residency applications: differences by program type and characteristics
.
J Grad Med Educ
.
2019
;
11
(6)
:
698
793
.
146. 
National Resident Matching Program.
Charting outcomes in the Match: Senior students of U.S. medical schools, 2020
.
2021
.
147. 
Lee
AH,
Young
P,
Liao
R,
Yi
PH,
Reh
D,
Best
SR.
I dream of gini: quantifying inequality in otolaryngology residency interviews
.
Laryngoscope
.
2019
;
129
:
627
633
.
148. 
Morgan
HK.,
Winkel
AF,
Standiford
T,
et al
The case for capping residency interviews
[published online ahead of print September 14,
2020]
.
J Surg Educ.
149. 
Burk-Rafel
J,
Standiford
T.
A novel ticket system for capping residency interview numbers: reimagining interviews in the COVID-19 era
.
Acad Med
.
2021
;
96
(1)
:
50
55
.
150. 
Quillen
DA,
Siatkowski
RM,
Feldon
S.
COVID-19 and the ophthalmology match
.
Ophthalmology
.
2020
;
128
(2)
:
181
184
.
151. 
American Medical Association.
Murphy
B.
The Match process is packed with stress. OB-GYNs aim to fix it.
2021
.
152. 
Dermatology residency program director information regarding the interview process during the 2020-2021 application cycle.
2021
.
153. 
American Orthopaedic Association.
Universal interview offer day
.
2021
.
154. 
Chang
CWD,
Pletcher
SD,
Thorne
MC,
Malekzadeh
S.
Preference signaling for the otolaryngology interview market
.
Laryngoscope
.
2021
;
131
(3)
:
e744
e745
.
155. 
Coalition for Physician Accountability. Reviewing the transition from UME to GME.
2021
.

Author notes

Editor's Note: The online version of this article contains search strategies for each database used in the study, a visual representation of the number of references proposing systems-level reforms to the US resident selection process by authors' specialty, the distribution of reforms proposed by otolaryngology and orthopaedic surgery, and a visual representation of the number of references proposing systems-level reforms to the US resident selection process by year.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

Supplementary data