Internal Medicine Program Directors' Perceptions of the “All In” Match Rule: A Cross-Sectional Survey

The National Resident Matching Program (NRMP) is a private, nonprofit organization that manages the system to pair residency training programs with medical school graduates in the United States. The NRMP seeks to “provide a uniform time for both applicants and programs to make their training selections without pressure.”¹  The program employs an algorithm to connect applicants and residency programs through the use of rank order lists, with the process favoring the rank order lists of applicants.¹ 

Since the 2013 Main Residency Match, the NRMP invoked a rule change requiring all residency programs within an institution to individually declare each recruitment year whether they will be “all in” (ie, fill all of their positions completely through the NRMP process) or “all out” (ie, fill all of their positions completely outside the NRMP process).¹  Program directors who recruit non–US citizen applicants expressed concern with this change, citing that the out-of-Match approach allowed more time to process visa applications, and prevent late residency start dates that disrupt training cycles.²  They also noted that “all in” strategies would require significant increases in program resources for recruiting and interviewing, as has been described in other residency and fellowship program matches.^2–7  Programs were further challenged in 2015, when malfunctions of the visa processing software at the US Department of State combined with a record number of applicants stressed residency resources.⁸  In light of these challenges and the rule change, this study sought to investigate whether concerns cited prior to the change proved to be valid or not by surveying program directors regarding the positive and negative effects of the “all in” Match rule on their programs.

Using a mixed model cross-sectional survey, we anonymously surveyed directors of Accreditation Council for Graduate Medical Education–accredited internal medicine (IM) residency programs via e-mail using a SurveyMonkey link (SurveyMonkey, Palo Alto, CA) directed to their Alliance for Academic Internal Medicine–registered e-mail address. The survey asked their impressions of the effects on the policy change with regard to their current status (“all in” or “all out”), whether they are university- or community-based programs, and their overall impressions of the impact of the policy change. Prior to distribution, the survey was alpha tested on a subgroup of 5 program directors, which resulted in refinement of the questions. The survey used skip logic to ensure that program directors were only asked the follow-up questions relevant to answers given earlier.

This project was approved by the Institutional Review Board of the Reading Health System.

Statistical analysis was performed with SPSS version 23.0 (IBM Corp, Armonk, NY) and included summary statistics (ie, percentage of responses). Chi-square testing was performed to compare associations among answers, and a P value of < .05 was determined to be significant.

There were several free-text questions in the survey. SurveyMonkey aggregated the answers anonymously into a list of responses. We used a constant comparative analysis approach associated with grounded theory emergent qualitative data analysis⁹  to examine free-text data within the predetermined research objectives (ie, focusing on the impact of the changes on recruitment practices, unintended consequences, and overall impressions of the rules change). The unit of analysis was the meaning of an individual utterance: 1 responder could therefore contribute more than 1 response to the data. Rather than using predetermined themes, emergent analysis allowed for review of the textual material multiple times to identify the themes that “emerged” from the data. Three researchers (R.A., M.S.K., A.D.) independently read the entirety of the free-text commentaries, with quotes corrected for grammar to improve readability where necessary. Quotes were analyzed using the constant comparative method,¹⁰  and a detailed codebook was developed to identify the preliminary recurring themes. After a second review of all survey responses by the same 3 researchers, preliminary themes were refined to the final recurrent themes, and then were collectively reviewed by the entire author group as a member check.

Of 396 directors of accredited IM residency programs, 127 (32%) responded to the survey. Twenty-one respondents (17%) defined their program as university based, 72 (57%) as community-based university affiliate, 23 (18%) as community based, and 1 (> 1%) as military based.

A total of 122 of 127 respondents (96%) identified their program as “all in,” with the remaining 5 as “all out.” Of the 5 “all out” programs, 2 had started as “all in” and switched to “all out,” and 1 of these 2 indicated it was considering switching back. Twenty programs had switched to “all in” from “all out,” with 1 contemplating switching back to “all out.” A total of 51 of 127 programs (40%) reported that over the last 3 years, less than 10% of their residents were non–US citizens holding a visa, and 60 of 127 (47%) programs reported that more than 10% of their residents over the last 3 years were non–US citizens.

University-based programs predominantly had a neutral or positive view of the rule change (14 of 21, 67%), while the perspective of community-based programs was mixed (37 of 70, 53%). The difference was not statistically significant. Programs with significant numbers of visa holders were split on the impact the rule change had on their programs (positive in 22 of 60, 37%), while programs with few visa-holding residents felt generally favorably toward the rule (positive in 31 of 51; 61%; table 1).

Of the 112 respondents, 108 (96%) noted either no effect or an increase in the number of applicants to their residency program since the rule change, and were evenly split between these 2 results. The remaining 15 programs did not respond to this question. A substantial minority of programs (41 of 112, 37%) reported an increase in the quality of their residents, while the remainder indicated no effect on resident quality. A significant minority of programs (27 of 112, 24%) reported an increase in the number of residents who had delayed start dates due to visa issues, while the remainder indicated no impact (table 2).

Two questions generated enough free-text responses to lend themselves to qualitative analysis. In the first of these, program directors were asked to explain why their program had selected the “all in” approach. Four themes were identified and grouped: (1) “perceptions of increased competitiveness”; (2) “the program had no choice”; (3) “a desire to simplify logistics”; and (4) “increased fairness to residents and applicants.”

The second question asked residency program directors about unintended consequences (positive or negative) on their residency programs as result of the NRMP policy change. Initial themes were broken into “positive,” “negative,” and “neutral/no change.” The themes of unintended positive effects and 4 themes of negative effects were identified. Positive themes included “perceptions of increased fairness to programs and applicants,” “perception of improved quality of residents,” and “decreased stress on programs through the lack of making pre-Match decisions.”

Representative positive quotes include:

• •

  “I'm delighted to be out of the ‘twisting arm' mode of out of Match spots. I always felt that I was placing the program's interests over the candidates.” (Theme: perceptions of increased fairness to applicants)

• •

  “Previously, some IMG and DO candidates would interview and request contracts, which made us believe that if we did not take them outside of the Match they would move on to the next program on their list and we would not have a chance to Match with them.” (Theme: decreased stress on programs)

Negative themes that emerged were (1) “increased program resource use”; (2) “lower-quality residents enrolling in the program”; (3) “increased applicant resource use”; and (4) “perceptions of bias against certain types of applicants.”

Representative negative quotes include:

• •

  “We have had to increase the number of applicants that we interviewed since we could no longer cherry pick a few pre-Matches. This has led to increased costs (hotels, food, decreased productivity of faculty) and increased organizational headaches. We then had to purchase Interview Broker to deal with the organizational headaches of more interviews, which, again, led to increased costs.” (Theme: increased program resource use)

• •

  “Lost candidates to other programs that are ‘all out' that would have preferred to enter our program but thought a commitment to be a priority.” (Theme: lower-quality residents enrolling in the programs)

This study found that a slight majority of residency programs reported a neutral or negative impression of the NRMP rule change. Programs with higher percentages of visa-holding residents noted increased numbers of residents having delayed starts, and were evenly split between positive and negative impressions. At the same time, the quality of residents was felt to be unchanged or improved in the majority of residency programs after the rule change. A substantial minority of residency programs reported an increased number of applicants interviewed and associated increased resource use by programs and applicants.

A previous survey from the Association of Program Directors in Internal Medicine indicated that residency program directors predicted an increase in application volume and recruitment costs by adopting an “all in” approach, and expressed concerns of decreased applicant quality and increased numbers of visa-holding residents having delayed starts.²  With the notable (positive) exception of applicant quality, our study confirmed these predictions.

Although not directly studied, several qualitative comments confirmed NRMP assertions that the “all in” policy may have lessened the high-stakes, high-stress, time-limited, decision-making pressures on both programs and applicants. Our findings were strikingly similar to the experiences reported from family medicine,³  radiation oncology,¹¹  nephrology,¹²  gastroenterology,¹³  and obstetrics and gynecology,^14,15  in which declines were reported in these high-stress, time-limited decisions by programs and applicants as well as changes in applicant volume.

Limitations of this study include the low survey response rate, which may have led to under- or overreporting of programmatic issues. We collected our information during the recruiting season to ensure information was fresh in program directors' minds, but this may have been an inopportune time for programs to provide a thoughtful response. Our sample did overrepresent community programs (75% of respondents [95 of 127], while community programs make up only two-thirds of all residency programs¹), and our findings may not generalize to the entire community. Additionally, our survey instrument does not have established validity evidence, and participants may have interpreted the meaning of our questions differently from what we intended.

Further study is needed to determine methods to reduce these burdens for both graduate medical education programs and applicants. Whether moving the date of the Match to earlier in the calendar year or whether national advocacy from governing bodies with US visa-granting authorities could decrease the rate of delayed start times are matters for further discussion. Finally, while most residency positions have adopted the “all in” approach, the opposite is true of fellowship positions. As subspecialty societies decide whether to adopt an “all in” approach, their members will need to consider the increased resource and time costs.

A slight majority of programs reported neutral or negative impressions of the “all in” rule change. Since the adoption of this policy, some programs reported increased application volume and growing resource utilization, including interview-associated costs. Programs with higher levels of visa-holding residents noted increases in delayed start dates for residents.