Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021

The transition to residency has become costly in time and money.¹  Virtual interviews have reduced some costs but increased the number of interviews. Several proposals suggest ways to increase the efficiency of applications and interviews.^2-10 

One proposal, for residency recruitment in obstetrics and gynecology (OB/GYN), is the Early Result Acceptance Program (ERAP).^11-13  ERAP would be an optional, binding “early match” in which applicants submit a limited number of applications to OB/GYN programs, which proponents suggest would promote more in-depth consideration of applications. Residency programs could offer up to half their positions early. The standard application, interview, and match processes would follow for applicants and programs unmatched after ERAP.

ERAP proposers hypothesize that an early match would ease the burden of applications and interviews, allowing for holistic review of applicants.^11,14  Critics have raised concerns about destabilizing the Match,¹⁵  increased applications and stress, and potentially stigmatizing applicants who do not match early.¹⁶ 

We report the first data-driven analysis of ERAP, using full National Resident Matching Program (NRMP) rank order list (ROL) and Match data from 2014 to 2021. Through computer simulation, we modeled how OB/GYN applicants and programs would have fared with ERAP vs the historical (actual) match for each year. We explain how blocking pairs (ie, mutually dissatisfied applicant-program pairs) are inevitable in an early match with a fraction of positions available and discuss how such destabilization historically led to unraveling of recruitment processes.

De-identified ROLs and Match results from 2014 to 2021 for all residency programs and applicants, across all specialties, were provided by the NRMP (including ROLs for single applicants and couples).

Table 1 describes applicant and program match populations from 2014 to 2021. For example, in the 2021 NRMP Match, 294 OB/GYN programs offered 1478 positions. A total of 2039 applicants ranked OB/GYN programs, and of those, 1880 were single applicants, 1773 of whom were OB/GYN-preferring, having listed an OB/GYN program as their first choice. A total of 414 applicants listed both OB/GYN and other specialties, with family and internal medicine being most common, and 294 of these applicants ranked both OB/GYN and non-OB/GYN programs in their top 5 choices. Of the 1773 applicants who listed OB/GYN as their first choice, 307 ranked multiple specialties. One hundred and ninety-four of these applicants ranked a non-OB/GYN program in their top 5 choices. Out of the 2039 applicants, 159 entered the match as members of a couple. Only 3 couples had both members of the couple ranking OB/GYN programs.

We programmed the Roth-Peranson algorithm used by the NRMP.¹⁷  To provide validity evidence for the algorithm, we reproduced the single-round match with ROLs provided by the NRMP and compared this to the actual outcome that occurred in each historical NRMP Match. In every year, the reproduction of the single-round match resulted in identical outcomes for at least 98.7% of individual applicants and at least 99.94% of couples compared to the historical NRMP outcomes. For OB/GYN applicants, fewer than 0.05% of applicants received different outcomes than their actual NRMP matches. This provided validity evidence for the reproduced algorithm. Note that the slight incongruity in outcomes is expected due to the presence of couples whose joint ROLs may impact the final match due to arbitrary ordering decisions needed to confirm that the Match algorithm converges.

We simulated ERAP through a computer model (programmed in C#) that included an early match followed by a main (regular) match, using each season's NRMP ROLs as input. We used the same model to reproduce the current single-round match for comparison (as above). At each round, we ran the Roth-Peranson matching algorithm. In the early match, only OB/GYN-preferring applicants were included. In the main match, applicants to all specialties were included. In the simulation, we excluded matches from the Supplemental Offer and Acceptance Program (SOAP). ERAP was simulated as follows:

Each OB/GYN-preferring applicant applies to the top 5 OB/GYN programs from their original NRMP ROL, or all of them if they ranked fewer than 5 OB/GYN programs. Each OB/GYN program offers 50% of its available positions (rounding down if needed), except for programs offering only one position, in which case the single position is included in the early match. Programs use their actual NRMP ROLs to rank applicants.

Applicants who match and program positions that fill in the early round are considered matched and excluded from the “main match” round. Sensitivity analyses evaluated alternative application limits, positions available, and alternative specifications of program ROLs.

Following the early match, all ERAP-unmatched OB/GYN applicants and all unfilled OB/GYN programs enter the main residency match, along with applicants and programs from all other residency specialties. OB/GYN programs offer positions that were unfilled in the early match round. Other specialties' programs include all positions. Applicants and programs submit their actual NRMP ROLs.

In the early match, members of couples interested in OB/GYN are treated as singles, using their original NRMP ROL to approximate their preferences. Note that almost all couples are mixed-specialty couples, where one member applies in OB/GYN and the other in another specialty. In the main match, partners of OB/GYN couple members (who matched in the early round) enter as singles. Reported ROLs approximate applicants' preferences.

Finally, we compared the simulated 2-round ERAP and single-round match outcomes for OB/GYN for each year (2014-2021) as if the early match had been introduced in that year (ie, without modeling any strategic adjustments that might result after the first year of the early match). Each year was analyzed separately to demonstrate the likely variability of future outcomes.

The study was reviewed by the Stanford Institutional Review Board (IRB 56742), which determined that the study of these de-identified data does not require consent from applicants or programs.

In each year (2014-2021), the percentage of OB/GYN positions filled in the early match (among those offered early) was between 98% and 99%. Both the simulated ERAP and single-round match filled more than 98.5% of all positions. We report here the analysis based on 2021 outcomes, but results for all years are included (Table 2 and Figure 1).

Of the 1773 OB/GYN-only preferring applicants, 138 (7.8%) received a more preferred position in ERAP overall than the single-round (traditional) match, while 256 (14.4%) of applicants received a less preferred position (more and less preferred refer to an applicant's placement on their historical ROL). Of the 625 OB/GYN-only preferring applicants who matched in the early ERAP round, only one applicant received a more preferred outcome, while 188 received a less preferred outcome. The remaining 436 (70%) who matched in the early ERAP round matched to the same program as in the single (traditional) match. Figure 1 shows the number of applicants who received more or less preferred outcomes by year, stratified by applicant type: US MD Senior, US DO Senior, IMG (international medical graduate, including US IMGs), and US Grad (US medical school graduates from prior years). Using US MD Seniors as a benchmark, the results from US DOs, IMGs, and US Grads were 68.1%, 121.8%, and 150.4% more matched to less preferred positions, respectively. The results are robust to the inclusion of non-OB/GYN-preferring applicants in ERAP, with only 4 in this category obtaining different match outcomes.

Of the 294 OB/GYN programs, 2 programs filled a different number of positions under the 2 scenarios. One hundred twenty-two (41.5%) programs received a more preferred outcome in ERAP overall than the single-round (traditional) match (ie, for each applicant in the program's single-round match, there exists a distinct applicant of equal or higher rank in their ERAP match). Seventy programs (23.8%) had less preferred outcomes with ERAP (ie, the program's match results were nonidentical; all applicants that matched were ranked lower by the program). Sixty programs (20.4%) filled with an identical set of applicants with ERAP, while 42 (14.3%) had mixed outcomes (ie, the program matched some combination of more and less preferred applicants based on their historical rank lists). Every year, except 2021, fewer programs received more preferred outcomes than either a less preferred or a mixed outcome, with ERAP.

Of the 156 of 159 couples with only one member preferring OB/GYN, 58 (37.2%) matched early with ERAP. Thirty (19.2%) couples received a less preferred match with ERAP. Thirty-five (22%) couples whose non-OB/GYN-preferring member matched in the main match would have achieved a joint outcome not listed on their original joint ROL (ie, at a pair of programs they had not indicated as an acceptable match). This may happen if only one partner matches early, then the 2 partners are effectively participating as 2 single applicants (OB/GYN-preferring in the early ERAP, the other in the main match).

A mutually dissatisfied pair consists of an applicant who didn't match to a more preferred program, and their preferred program, which also would have preferred this same applicant, compared to at least one of the other matched applicants. That is, both the applicant and the program would have preferred each other to their ERAP-determined match. In economics, this is referred to as a “blocking pair.” The current NRMP Match, governed by the Roth-Peranson algorithm, provides a “stable matching” outcome that generally contains no blocking pairs.

With the 2-round ERAP match, 205 (11.6%) of the 1773 single (traditional) match applicants who ranked an OB/GYN program first on their ROL became part of mutually dissatisfied pairs. One hundred eighty (70.4%) single applicants who received a less preferred outcome were part of a mutually dissatisfied pair, and 154 (52.4%) of all OB/GYN programs were in mutually dissatisfied pairs at the end of the 2-round ERAP match. Notably, 92 (75.4%) of 122 programs that received a more preferred outcome were assigned an applicant who was part of a mutually dissatisfied pair (with a different program).

ERAP results in the creation of previously non-existent blocking pairs. By instituting a 2-round match, 205 (11.6%) of the 1773 single applicants who ranked an OB/GYN program first on their ROL were part of mutually dissatisfied pairs. Two hundred one of the applicants in such pairs matched early, while 4 matched in the main match. Eighty-eight percent (225 of 256) of applicants who received a less preferred outcome were part of a mutually dissatisfied pair. Approximately 52% (154 of 294) of all OB/GYN programs are in mutually dissatisfied pairs at the end of the 2-round ERAP match. Notably, 75.4% of programs that received a more preferred outcome were assigned an applicant who was part of a mutually dissatisfied pair (with a different program).

We varied the application limit in the early round from 1 to 6 applications (k=1-6, online supplementary data). Some single applicants receive more preferred outcomes, but even more receive less preferred outcomes as k increases (eg, when k=3, 6.4% receive more preferred outcomes and 10.6% receive less preferred outcomes). As k increases, more applicants are in mutually dissatisfied pairs (eg, for k=3, 8.3%), and more mutually dissatisfied pairs are present among applicants who match early (eg, for k=3, 17.7%). Note the opposite trend is observed among applicants who match in the ERAP main match round. Couples' outcomes are similar for all values of k.

We also varied the percentage of positions offered in the early round from 20% to 80% (P=20% to 80%, online supplementary data). The smaller the value of P, the more mutually dissatisfied pairs are present among applicants who match early. As the value of P increases, more couples with one member preferring OB/GYN match early.

This simulation assumes applicants and programs will interview and rank programs, in the context of an early match and ERAP, similarly to how they acted in the historical single-round match. Applicants might not apply early to all programs, and programs might not consider every applicant who applies in the early round. To relax these assumptions, we simulated ERAP under 3 models. In model 1, OB/GYN-preferring applicants apply in the early round only to OB/GYN programs higher on their ROL than any non-OB/GYN program, and to no more than 5 programs. In model 2, programs rank only applicants in the top k% (k=50, 30, 10) of their original ROL. In model 3, programs consider only their preferred k% (k=50, 30, 10) of applicants who applied. The first model demonstrates robustness with respect to mutually dissatisfied pairs. In models 2 and 3, the percentage of applicants receiving less preferred outcomes, or part of mutually dissatisfied pairs, decreases substantially only when the percentage of early round unfilled positions increases (Table 3). That is, the harms measured in this analysis are those caused by early matches, and they decline only when early matches decline.

Our simulation study of ERAP, a novel proposal to address the growing challenges of residency recruitment, shows that a 2-round match would not affect the number of positions filled in the match. However, it would alter which applicants match at which program and generate mutually dissatisfied, applicant-program pairs. Additionally, more applicants would match to less preferred programs, with DOs and IMGs disproportionately affected, and strategic challenges for couples would be created.

The NRMP Match and algorithm allows multispecialty applications and couples matching, including mixed-specialty couples matching. The algorithm results in a stable final match (ie, there are no mutually dissatisfied applicant-program matches). This approach promotes participants stating their true preferences on ROLs. There is a considerable body of evidence from other markets and from the history of the NRMP that these properties are critically important for the orderly and successful operation of the Match.^18-20  Our simulation findings suggest that ERAP will not maintain these properties. In this ERAP simulation, because only 50% of a program's positions are available in the early round, applicants who would have matched with a program if more of its positions were available may match early with a program they prefer less. Similarly, programs may match applicants who are less preferred. Since matches are binding, this often creates mutually dissatisfied pairs (Figure 2).

The early match alone does not produce mutually dissatisfied pairs (ie, non-stable matches); this occurs with the combined early and late matches. Proponents of ERAP hoped to continue NRMP's stable matches by using the Roth-Peranson algorithm for both matches.¹⁴  However, because of the binding nature of the early match, this simulation shows that mutually dissatisfied matches are likely to occur, and most of these will occur in the early match cycle. A primary concern with mutually dissatisfied applicant-program pairs is that they provide incentives for outside-the-match bargaining in subsequent years. This issue led to the creation of the NRMP in 1952. Bargaining behaviors have been observed in medical matching in the United Kingdom and in other matching markets with artificial capacity constraints, such as sorority matching at US universities. In short, matching procedures that produce mutually dissatisfied pairs have a history of failure.^20-25 

Strategic behaviors caused by mutually dissatisfied pairs may not be limited to one specialty. For example, 17% of OB/GYN-preferring applicants rank other specialties, and 63% of these historically rank another specialty in their top 5 ROL choices. These applicants would not be able to express their full preferences in the limited choices in the early ERAP match. This situation and concerning behaviors have been observed in ophthalmology and urology, which have earlier matches through non-NRMP services. It is also difficult to speculate how this would impact combined specialties (eg, pediatrics/anesthesiology, internal medicine/psychiatry, etc). ERAP creates strategic challenges for couples, 98% of which are mixed-specialty couples. Such couples will need to consider the implications of the OB/GYN-preferring partner matching early, which creates a scenario whereby the non-OB/GYN-preferring partner enters the main match as a single applicant. Couples thereby may adapt their ROLs to avoid matching in an unacceptable pair of programs, or consider forgoing ERAP or matching into OB/GYN entirely.

If early matches were widely adopted, many of the benefits of the NRMP would be lost, renewing the incentives for unraveling, including by couples (but not limited to couples), and creating further challenges for DOs and IMGs to apply on a level playing field.²⁵ 

Limitations to this study include the assumption, used in the simulation, that historical ROLs would mirror future applicant and program preferences in ERAP, in either early or late ERAP rounds. This includes effects on ROLs pre- and post-interview. It is also challenging to determine the change in satisfaction an applicant or program would experience in matching at a position on their ROL compared to a program at a lower position. It is likely that applicants would consider many factors (including, but not limited to, chance of match and program preferences) when selecting a limited number of applications in the early match round. These unknown factors, including the burden on applicants and programs to participate in 2 matches, were not considered in the simulation (eg, 2 applicants with identical ROLs may differ in how much they prefer each program, leading to different behaviors under ERAP). Behaviors of couples, after the early match round, are similarly unknown but may include change in match strategy or arranging for a residency position directly (as couples needed to do before the introduction of the couples' match). It is also possible that not all OB/GYN applicants or programs would choose to participate in ERAP. This, together with the limit on the number of applications, could possibly allow programs to spend more time reviewing applications, which may impact their ROLs.

Future research steps may include simulating the effects of preference signaling that several specialties are piloting, as these strategies are more aligned with the stability of the match. Additionally, more in-depth study of the factors influencing applicant and program decisions to interview and create ROLs is needed.

This simulation of the proposed OB/GYN ERAP found that more applicants received less preferred match outcomes and created mutually dissatisfied applicant-program pairings, which disproportionately affected DOs and IMGs and created tough decisions for multispecialty couples. These results suggest that an ERAP match would destabilize the market, open the door to rule-breaking behavior, and destabilize the Match in subsequent years.

The authors would like to thank Laurie Curtin, Donna Lamb, Cathy Lazarus, Marc Melcher, Kenneth Simons, and Irene Wapnir. This study would not have been possible without the NRMP providing detailed data that allowed for a comprehensive understanding of ranking and matching patterns over a period of years.