The Relative Influence of Program Signaling, Geographic Preferences, and In-State Status in Determining Odds of Interview Invitation in Residency Selection

To enhance the residency application and selection process, the Association of American Medical Colleges (AAMC) first piloted program signaling and geographic preferences in a supplemental application (SuppApp) of the Electronic Residency Application Service (ERAS) with 3 specialties during the 2022 recruitment cycle and expanded to 16 specialties in the 2023 cycle.¹  The use of program signals and geographic preferences was intended to provide applicants agency and transparency in self-selecting individual program and geographic preferences in their application, while also allowing programs to identify applicants who selected their program or region as one of interest. With the addition of these variables, data from numerous specialties show an increased likelihood of an applicant being selected to interview when a program signal is received.^2-8 

Data from Match cycles prior to the implementation of program signals and geographic preferences have also shown that a high rate of applicants match in locations with geographic proximity to their medical school.^9,10  Geographic bias has been previously reported in residency recruitment as applicants having a higher likelihood of matching at a program within close geographic proximity. A report of National Resident Matching Program (NRMP) data for allopathic seniors from 2011 to 2015 revealed 51% of applicants matched in the same divisions as their medical school.⁹  Similarly, an analysis from 2018 to 2020 revealed that applicants who came from allopathic state schools with a higher percentage of in-state matriculants were more likely to match in a geographically closer program.¹⁰  As the application process has adopted the use of self-selected geographic preferences, it is unknown if the same kind of geographic bias remains for in-state applicants. Furthermore, it is unclear how programs use geographic preferences and in-state status in their selection process when most applicants send signals to programs in a preferred geographic division or the same geographic division as their permanent address or their medical school.¹¹  In seeking to understand how geographic variables and program signals are used together in interview selection, a gap remains in investigating the relative influence of program signals and geographic preferences when applicants also have geographic ties to the same state as a program to which they applied.

Understanding the relative influence of these 3 variables would allow those involved in the residency recruitment and application processes to consider how self-selected and fixed geographic factors may be used when considering what is important for residency selection. The aim of this study is to identify the relative influence of program signaling, geographic preferences, and an applicant’s in-state status in determining the odds of receiving an interview invitation during residency recruitment.

Applicant and program data were accessed by AAMC staff through the ERAS and Program Director’s WorkStation (PDWS) databases.

Sixteen specialties required applicants to provide program signals and geographic preferences for the 2023 ERAS application cycle. This study focused on those specialties that met the following inclusion rules: (1) fewer than 10 signals were provided per applicant; (2) no tiered signals were offered (ie, gold and silver); (3) signals were inclusive of only one specialty (ie, diagnostic and interventional radiology shared signals and were therefore excluded); and (4) at least 10 programs participated per specialty. This resulted in the inclusion of 9 specialties: adult neurology, anesthesiology, dermatology, general surgery, internal medicine, neurological surgery, pediatrics, physical medicine and rehabilitation, and psychiatry.

Programs using the PDWS were eligible participants and provided consent for data to be used for research upon registration. To be included in the final analytic sample for each specialty represented, individual programs met the following inclusion criteria: (1) programs provided number of approved year 1 positions in the 2021 GME Track Resident Survey; (2) programs provided interview offer information in the PDWS by March 15, 2023; (3) programs met the required interview selection ratio reported in the PDWS per available residency positions (each specialty defined the required interview selection ratio that was used as the upper limit of applicants selected to interview per available residency position; this ratio was identified to mitigate risk of including programs with incomplete interview selection data in the PDWS); (4) programs received at least 90 applications for enough power to run the model (ie, 30 cases for each of the 3 independent variables) to meet minimum sample size requirements¹² ; and (5) programs sent at least one interview invitation within each predictor variable (ie, sent an interview invitation to at least one applicant who signaled their program and at least one applicant who did not send a signal, sent at least one interview invitation to each geographic preference variable including aligned, not aligned, and no preference, and sent at least one interview invitation to one in-state and one out-of-state applicant).

Applicants from US medical schools who submitted an application to one of the participating programs in the 2023 cycle were included. Participants provided consent for their data to be used in research when they submitted their applications using ERAS.

We collected data for program signals and geographic preferences from all applicants who submitted the SuppApp as part of their residency application process from August 1 to September 16, 2022, when SuppApp closed. All questions in the SuppApp were optional for applicants. An applicant’s in-state status was identified and pulled from geographic information listed on their ERAS application.

Program signals were binary indicators of an applicant’s interest in a program at the time of application. Each specialty determined the maximum number of signals applicants could send to programs in their specialty.

Geographic preferences indicated an applicant’s interest in a geographic region at the time of application. Applicants could select up to 3 (of 9 possible) US Census Divisions. They could alternatively indicate “no preference” or decline to participate in the geographic preference selection. In this analysis, applicants were defined as being in 1 of 3 categories: “aligned” if one of their preferred regions was the same as an applied program; “no preference” if the “no preference” option was selected; or “not aligned” if any of their preferred regions were not the same as an applied program. This last category was used as the main comparison group for analyses with the other 2 categories.

This was a generated variable indicating whether applicants’ permanent address was located in the same (or different) state as a signaled program’s state. All applicants from international medical schools were excluded.

This variable is a binary indicator from each participating program indicating whether an applicant was selected for an interview in the PDWS. To help improve data quality, all programs were asked during interview season to document their interview selections in the PDWS for research purposes only. Interview status was collected from the PDWS on March 15, 2023.

All analyses were conducted using R version 4.2.2 (R Project). Each model explored the predicted probability of interview invitation based on either program signaling, geographic preference signaling, or in-state status predicting interview invitation. Results were first analyzed separately at the program level because each program likely uses slightly different selection criteria, and it is important the data reflect those differences. Area Under the Curve Receiver Operating Characteristic (AUC-ROC) results were calculated to estimate how well each predictor variable discriminates between applicants who received an interview invitation and those who did not.^13,14  Programs were included in results based on meeting a minimum threshold for model fit (ie, the lower-bound 95% confidence interval of the AUC-ROC was 0.50 or greater),¹⁵  as this value indicates the model is no better than predicting outcomes at random and provides a less biased reporting of the data distribution.^16,17  Results were then aggregated across programs within each specialty by computing the median odds ratios (ORs). The proportion of programs included with significant (P<.05) ORs is also reported.

This study was reviewed by the AAMC Human Subjects Research Protection Program, and data were approved for publication by the institutional review board of the American Institutes for Research.

Analytic samples were generally representative of each specialty’s total ERAS population with respect to program size and region, but some programs within several specialties in our sample had lower than average numbers of applications (ie, adult neurology, internal medicine, pediatrics, psychiatry, and surgery; see online supplementary data). More than half (51% to 81%) of the programs that participated in SuppApp from each specialty met inclusion criteria, and their data were included in the final sample. Table 1 shows the breakdown of programs meeting inclusion criteria by specialty.

Table 2 shows the distribution of ORs for all programs within the specialty, using the median as well as upper and lower OR ranges for each predictor variable (excluding outliers). Across specialties, applicants who signaled programs were 2.71 to 9.07 times more likely to receive interview invitations at those programs than applicants who did not send signals. The proportion of programs with significant ORs ranged widely—from 58% to 94%—within and across specialties. In specialties having a higher proportion of programs with significant ORs, the odds of being invited to interview between the 2 comparison groups (eg, those who signaled and those who did not) are significantly different. Therefore, the higher the proportion of programs with significant ORs, the more confident one can be in the reported values being more generalizable to programs within that specialty.

Applicants who reported a geographic preference that aligned with a program to which they applied were slightly more likely to receive an interview invitation than those applicants who reported a geographic preference that did not align with a program’s geographic region (median ORs between 1.83 and 2.75). These results were significant for 14% to 78% of programs. Applicants who reported no geographic preference were slightly more likely to receive an interview invitation than those applicants who reported a geographic preference that did not align with a program’s geographic region (median ORs between 1.19 and 2.16). These results were significant for 10% to 39% of programs.

Applicants with in-state status (permanent addresses in the same state as a program to which they applied) were 2.45 to 5.14 times more likely to receive an interview invitation from those programs than applicants who were out of state. These results were significant in 26% to 85% of programs.

For 7 specialties, the odds of receiving an interview invitation were similar for applicants who sent a signal or had in-state status (adult neurology, dermatology, internal medicine, neurological surgery, pediatrics, psychiatry, and surgery). For anesthesiology, sending a signal was almost twice as likely to increase likelihood of receiving an interview invitation than being in-state. For physical medicine and rehabilitation, sending a signal was over twice as likely to increase the chances of receiving an interview invitation than in-state status. For all 9 specialties, the median odds of receiving an interview related to geographic preference (for both alignment and no preference) was lower than for applicants who sent a signal or had in-state status.

Figure 1 shows the large positive skew across specialties and the wide range of values for the impact of program signaling and in-state status on receiving an interview invitation. Figure 2 shows a similar positive skew across specialties and the range of values for the impact of different geographic preferences on receiving an interview invitation, but to a much lesser degree than program signals or in-state status.

The results of this study suggest that the use of program signals, geographic preferences, and having in-state status generally increased the likelihood of an applicant receiving an interview invitation both within and across specialties for the 2023 application cycle. Despite these findings, it is important to note that all specialties showed wide ranges for their ORs. This likely reflects the highly variable way programs within specialties used the different pieces of application information.

Program signals had the highest odds of interview selection compared to the other predictor variables. When an applicant indicated they had no geographic preference, or when the geographic preference aligned with a program’s location, the likelihood of receiving an interview offer also increased when compared to when geographic preferences did not align. However, there was wide variability in the number of programs where this occurred. For example, this applied to only 14% of programs in neurological surgery, which may be due to wider variability in selection processes at the program level or due to advising patterns for applicants to apply more broadly, regardless of geography. Another key finding was that applicants’ in-state status had an important positive impact on their likelihood to receive an interview invitation across specialties, supporting previous reports of enhanced Match rates for in-state applicants across several specialties.¹⁸  For out-of-state applicants, these findings may encourage applicants to use program signals and geographic preference alignment to increase their odds of receiving an interview invitation. For programs, the increased odds of being invited to interview for applicants with an in-state status may indicate a geographic bias in the selection process that may hinder efforts to recruit applicants with the skillset and potential to be successful in their programs. Alternatively, some programs may prioritize in-state applicants based on their program’s specific mission and values, in which case these efforts should be transparent to potential applicants.

This study has several limitations that should be considered. First, to ensure reasonable data quality and confidence in results, some specialties and programs were excluded from analyses, which likely depressed the size of the ORs. Second, despite the wide ranges of ORs, internal specialty variability was still depressed such that programs that did not meet the minimum requirement for number of applicants per comparison condition in the models were excluded. Thus, the programs included in our analyses may have considered more variables when determining which applicants were selected for an interview. The high proportion of programs within each specialty that showed variation in the number of applicants selected to interview suggests that there are additional factors that influenced the likelihood of receiving an interview invitation (eg, application content, examination scores), which is in line with the adoption of holistic review with recruitment. Third, our findings are limited to only US-granting allopathic and osteopathic applicants, and so these findings may not apply to international medical graduate applicants. Finally, the system used to collect data on interview invitations was voluntarily provided by programs and may vary in accuracy. Therefore, the results presented in this study should be interpreted cautiously; intentional, personalized advising around the use of signals, geographic preferences, and in-state status continues to be important.

Future studies should investigate whether there are group differences based on applicant demographics in how program signals, geographic preferences, and in-state status influence residency recruitment. The variability within specialties also suggests future work should look at program-specific characteristics (eg, mission, values) that might influence the relative weight of these variables on interview invitations, and ultimately, Match outcomes, in order to better align applicants and programs.

Despite the variation both within and across specialties, an applicant’s use of a program signal, an aligned geographic preference, no geographic preference, and in-state status all individually increase the likelihood of an applicant receiving an interview invitation.

The authors would like to thank the specialty societies for participating in the supplemental Electronic Residency Application Service (ERAS) application pilot. The authors also wish to acknowledge these societies and their respective task forces including the Association of Program Directors in Surgery Application Cycle Task Force for their contributions in original planning of the supplemental ERAS application.

The online supplementary data contains further data from the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: Laura Fletcher, PhD, is an employee of the Association of American Medical Colleges.