Demographic Differences Between Applicants Who Submitted Rank Order Lists and Applicants Who Matched in 2022 and 2023 Open Access

The residency selection process is an important opportunity to foster diversity within the physician workforce. Although it represents a late phase in the physician training pathway, it remains a critical checkpoint in training where biases may influence the inclusion and exclusion of various groups. However, little is known about how each step in the process affects the selection of diverse attributes among resident physicians.

In the United States, the Match is the national clearinghouse for residency placement for most specialties and is administered by the National Resident Matching Program (NRMP). Herein, we conducted a novel analysis comparing the demographic characteristics of applicants who submitted ROLs and those of matched applicants in the Main Residency Match. This comparison excludes the impact of other stages of the selection process and investigates a single decision point in the training selection pathway: applicant ranking. We discuss the potential implications of such differences on workforce diversity, which we define as the distribution of select demographic characteristics among surveyed individuals. As the inclusion of health care professionals of different backgrounds and perspectives within medical teams has been shown to foster innovation, enhance cultural competence, and promote effective patient care,¹  ensuring diversity in the future medical workforce is critical to optimize the quality of our health care system.

We analyzed publicly available NRMP data for the 2022 and 2023 Matches to compare self-reported demographics for applicants who submitted rank order lists (ROLs) and applicants who matched.²  This report was derived from survey data reported by the 86% of active applicants in the Match who consented to provide demographic data for research in the years 2022 and 2023.²  The data for both years were aggregated. Applicants who were offered training positions through the Supplemental Offer and Acceptance Program (SOAP) were not included. Data for all 23 specialty groups available, excluding transitional year, were separately extracted from the tables provided by the NRMP that describe the demographics of active applicants. Multispecialty programs, such as those combining internal medicine with another specialty, were categorized by the NRMP under a single parent specialty group, with priority given to internal medicine, pediatrics, emergency medicine, or family medicine, as applicable, in that order of precedence, except for medicine-pediatrics, which was included as a separate specialty. “Preferred specialty” refers to the specialty of the program that an applicant ranked first in the Main Residency Match. “Matched specialty” refers to the specialty of the program that an applicant was ultimately matched into by the Match algorithm.

The proportions of all applicants who submitted ROLs and matched applicants were compared for 6 demographic variables: sex assigned at birth, sexual orientation, race, ethnicity, disability status, and US citizenship status. Answers missing or reported as “don’t know/prefer not to answer,” which comprised 0.0% to 5.8% of the data for each characteristic, were excluded from analyses. Applicants who reported more than one race were classified into a single category according to the following order chosen by the NRMP that prioritizes smaller groups in the general US population: Native American or Alaska Native, Pacific Islander, Asian, Black or African American, White, and Other. For more details on survey questions, please refer to the NRMP website for Charting Outcomes: Demographic Characteristics of Applicants in the Main Residency Match and SOAP.²  Due to some small group sizes, the number of participants in each specialty belonging to each demographic group are not reported herein to preserve applicant anonymity.

We used descriptive statistics to show the proportions of the applicant and matched applicant cohorts identifying with each demographic group. Differences in the percentage representation were calculated with confidence intervals based on the normal approximation of the binomial (Wald interval). The chi-square test with Yates’ continuity correction was employed to compare the observed distribution of groups within each demographic variable among matched applicants against the expected frequencies proportional to the representation of groups among applicants. To control the false discovery rate from multiple comparisons, the Benjamini-Hochberg procedure was used, and adjusted P values (or, strictly speaking, q-values) were reported. The Benjamini-Hochberg procedure was performed separately for the statistics pertaining to all specialties (6 tests) and the specialty-specific statistics for each demographic variable (6 sets of 23 tests). Statistical analyses were performed in Microsoft Excel (Version 2310). To measure effect size for significant comparisons, Cohen’s h was calculated for each set of proportions.

As the data are public and not identifiable, the institutional review board at Johns Hopkins University deemed this research as exempt from review.

The demographic characteristics of 73 426 applicants who submitted ROLs and 60 655 matched applicants were included in the analysis (Table 1). Sex, sexual orientation, race, ethnicity, and US citizenship status showed significant differences in proportions among all applicants and matched applicants (Table 1). There was no difference in proportions of applicants with or without disabilities (0.1% difference, 95% CI, -0.2 to 0.4, adjusted P=.36). A higher percentage of matched applicants were female as compared to all applicants who submitted ROLs (percentage matched applicants minus all applicants: +2.0%, 95% CI, 1.4 to 2.5). A higher proportion of matched applicants identified as bisexual (+0.2%, 95% CI, 0.04 to 0.4), whereas a lower proportion of matched applicants identified as heterosexual (-0.4%, 95% CI, -0.7 to -0.1), as compared to the respective proportions for all applicants. Those who self-reported as White represented a higher percentage of matched applicants as compared to all applicants (+2.9%, 95% CI, 2.3 to 3.4). Those who self-reported as Asian and Black or African American represented a lower percentage of matched applicants as compared to the percentages for all applicants (Asian: -1.5%, 95% CI, -2.0 to -1.0; Black or African American: -0.6%, 95% CI, -1.0 to -0.3). Hispanic/Latino individuals as well as non-US citizens were represented in lower proportions among matched applicants compared with all applicants (Hispanic/Latino: -0.5%, 95% CI, -0.8 to -0.1; non-US citizens: -4.9%, 95% CI, -5.3 to -4.5). All significant differences were of small effect size (Cohen’s h <0.2).

Specialties participating in the NRMP Match had differing results for the applicant demographics studied (Table 2 and Figures 1-3). Comparing the percentages of all applicants and matched applicants with each demographic characteristic, many specialties were more likely to match female applicants and applicants who were US citizens (Figure 1A and Figure 2C), although a minority of these comparisons were statistically significant for individual specialties when adjusting for multiple hypothesis testing (Table 2). Some specialties appeared to match a higher percentage of White applicants (Figure 2A; Table 2). A few specialties appeared to match more applicants with a specific sexual orientation or applicants who were not Hispanic/Latino (Figure 1B and Figure 2B), but none of these comparisons were statistically significant after adjusting for multiple hypothesis testing (Table 2). A few specialties differed in selection of applicants reporting a disability (Figure 3): a lower proportion of applicants who matched into orthopedic surgery reported a disability as compared to those who applied (-1.9%, 95% CI, -3.3 to -0.5, adjusted P=.03), while a higher proportion of applicants who matched into internal medicine reported a disability as compared to those who applied (+0.5%, 95% CI, 0.1 to 0.8, adjusted P=.008).

This study corroborates prior research comparing the demographics of applicants to residents, suggesting that selection processes alter the representation of individuals with certain demographic characteristics, such as sex and race.^3,4  Furthermore, we show that individual specialties have a range of propensities to select for applicants with different demographic characteristics, including sex, sexual orientation, race, ethnicity, and citizenship. However, the magnitude of these differences is small overall. In comparison to existing studies,^3,4  the present study excludes the impact of other stages of the selection process, offering insight into a unique step in the training selection pathway: applicant ranking. The data includes active applicants, which were those who submitted certified ROLs and thus approximate those individuals who were invited to interview. Concurrently, matched applicants represent those who were ranked highly enough at their desired programs to achieve a successful match. In a recent research letter looking at a subset of the data presented in this study, Nguyen et al found a significantly lower match rate of applicants reporting disability than applicants not reporting disability.⁵  In contrast, we took a different statistical approach and examined the representation of self-reported disability among all applicants and matched applicants, and we found no significant difference. For example, although some surgical specialties appeared to match more applicants without disabilities, internal medicine matched a significantly higher proportion of those with disabilities. The results of these 2 different comparisons may be reconciled under the conclusion that such a small difference in match rate for a small portion of the applicant pool results in a negligible difference in the overall representation of that demographic. However, these conflicting findings invite more nuanced investigation into how disability is factored into the applicant ranking process and how different types of disabilities are considered by learners and educators in different types of specialties. For example, Takakuwa et al surveyed program directors in emergency medicine to explore how they select residents and accommodate for those with disabilities.⁶ 

Detecting differences in the selection process for demographic characteristics does not necessarily indicate evidence of discrimination. However, it is well established that biases exist in hiring and selection processes both in and outside of medicine,^7-11  and holistic review practices must be carefully implemented.^12,13  Further research is necessary to determine why such differences exist for each stage of resident selection, whether certain differences are beneficial toward promoting a diverse health care workforce more representative of the patient communities serviced, or whether others might be discriminatory in nature.¹⁴  For example, we note that a number of specialties that appear to select for female applicants also have low percentages of active female residents, such as orthopedic surgery (20.3% female residents as of 2022-2023).¹⁵  Notably, a number of specialties with some of the lowest proportions of female residents such as interventional radiology (22.8%) and neurosurgery (23.7%) did not select for more female applicants,¹⁵  although samples sizes were small for these specialties in this study. Conversely, those specialties that already have the largest proportions of female residents, such as obstetrics and gynecology (87.1%) and pediatrics (73.3%),¹⁵  do not show a propensity to rank female applicants. Similarly, both practical and philosophical arguments could be made for the preference for US citizens over non-US citizens, which was the demographic with the largest difference in proportions of applicants compared to matched applicants. Practically, for example, programs may not be willing to bear the costs and/or risks of hiring a non-US citizen who may need a visa.¹⁶  If the applicant is also an international medical graduate, programs may be concerned about the equivalence of their undergraduate medical training. Philosophically, some would argue that US programs have an obligation to prioritize the training of US citizens, or else that training non-US citizens results in greater brain drain from their countries of origin.¹⁷ 

For other demographic characteristics such as race or ethnicity, further research beyond the scope of this study must be conducted to examine why the proportion of matched applicants who are White or not Hispanic/Latino were higher than those who applied. These findings are in line with existing studies in the literature: Poon et al analyzed 10 years of data from the Electronic Residency Application Service (ERAS) to find that race was associated with admissions in orthopedic residency programs, where minority applicants had lower odds of admission compared to White applicants controlling for various academic performance metrics.¹⁸  Similarly, Bowe et al compared applicant data from ERAS with resident data from the Accreditation Council for Graduate Medical Education, finding that 6 of 11 specialties studied had significantly higher proportions of White residents compared to applicants, while all 11 specialties had lower proportions of those underrepresented in medicine.³  Additional research is needed to determine the underpinnings of these differences and the steps to address the barriers to selecting individuals underrepresented in medicine.^19-21 

Lastly, sexual orientation as a demographic characteristic is difficult to examine critically due to the lack of studies on this topic in the residency selection literature. This study is the first to examine a large dataset on applicant sexual orientation. Unlike race, sex, or citizenship, which applicants are asked to report on ERAS, sexual orientation is not reported to programs on applications. Moreover, the NRMP Communication Code of Conduct instructs programs to avoid asking illegal questions, including those about sexual orientation.²²  While some programs may still ask illegal questions, surveys of applicants found that programs are least likely to inquire about sexual orientation compared to other restricted topics like plans for child rearing or childbearing.²³  As such, it may be surprising that sexual orientation was significant in this study’s analyses. One possibility may be that some applicants identifying with sexual orientations other than heterosexual might have voluntarily disclosed this information to programs (eg, in personal statements or interviews), possibly as a way to highlight diversity.²⁴  Further qualitative and quantitative investigations on this issue are needed.

Study limitations include, first, a lack of data before 2022, precluding an examination of trends over time, and the possibility that a small number of unmatched applicants who reapplied were double counted across the 2 years included in the study. The small number of years of data available for analysis may also have resulted in limited sample sizes for smaller specialties, reducing the detection of differences (particularly for demographics that were broken down into multiple categories such as sexual orientation, with very small samples sizes in some categories). Second, respondents were also required to select from fixed categories that may not fully capture their demographic characteristics. For example, the survey question about disability grouped physical and cognitive disabilities together. Third, there could be variables outside of those available for study that are responsible for the differences detected. Fourth, this study did not include the applicants who placed into a residency through the SOAP for clarity of results interpretation and to capture only the results of single step in the resident selection process. Lastly, there is a possibility that the differences captured in this study may represent not only applicant ranking selection decisions, but also applicant ranking behaviors. For example, Liang et al have shown that a small number of international medical graduates exhibit disadvantageous ranking behaviors, perhaps due to poor understanding of ranking/matching mechanisms.²⁵ 

Despite the limitations, this study invites future research into related questions about the relationship between applicant demographics and selection decisions along physician training pathways. For example, what mechanisms drive the differences found in this present study? Do regional restrictions or local attitudes on diversity initiatives play a role in the demographics of matched individuals to each region? Leaders and researchers in graduate medical education may find that the answers to these questions may result in ways to uncover or even address evidence of bias in selection processes.

This analysis of 2 recent years of available NRMP Match data finds that there are differences in the demographic characteristics of applicants who submitted ROLs and those of matched applicants.

The authors would like to thank Gayane Yenokyan, MD, MPH, PhD, from the Johns Hopkins Biostatistics Center, whose support was made possible by the Johns Hopkins Institute for Clinical and Translational Research (ICTR), which is funded in part by Grant Number UL1 TR003098 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. The contents of the article are solely the responsibility of the authors and do not necessarily represent the official view of the Johns Hopkins ICTR, NCATS, or NIH. The authors would also like to thank Sarah N. Bowe, MD, EdM, for her broad intellectual contributions to this paper.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.