Comprehensive Reform and Greater Equity in Applying to Residency—Trainees' Mixed Responses to a Pass/Fail USMLE Step 1

As the number of residency applications has grown, programs have increasingly relied on United States Medical Licensing Examination (USMLE) scores to screen applicants.^1–3  While standardized test performance may reflect academic potential, it is also laden with inequities that disproportionately hinder residency prospects for minority groups underrepresented in medicine (UiM).^4–6  To decrease reliance on Step 1 scores, USMLE co-sponsors announced a transition from numerical to binary (pass/fail) score reporting as early as 2022.⁷  In a 2020 national survey of program directors (PDs) from 30 specialties with a 45% response rate, the majority (61%) disagreed with the change compared to 15% who thought the USMLE Step 1 score reporting change was beneficial.⁸  As to the consequences of the score reporting change on socioeconomic disparities, 14% of PDs agreed the change would be beneficial and 43% disagreed.⁸ 

Currently, published perspectives from trainees are limited to small groups of medical students, primarily from US allopathic backgrounds. Through a national survey, we sought to understand medical student and resident/fellow perceptions regarding the change to pass/fail USMLE Step 1 score reporting and whether demographic or training factors were associated with these perceptions.

In July and August 2020, 197 deans of accredited medical schools (142 US allopathic, 30 US osteopathic, and 25 international allopathic) and 822 designated institutional officials (DIOs) from Accreditation Council for Graduate Medical Education (ACGME)–accredited institutions were invited to distribute anonymous electronic surveys among their trainees. US medical school deans and DIOs were identified through publicly available directories published by the Association of American Medical Colleges (AAMC), the American Association of Colleges of Osteopathic Medicine (AACOM), and the ACGME.^9–11  Email addresses were obtained from publicly available sources. Because we corresponded only with medical school deans and DIOs, and survey distribution was handled by separate officials, it was not possible to know which institutions successfully distributed the survey and to what proportion of trainees. Therefore, a response rate was estimated using the total number of trainees.

Separate analogous survey instruments were designed for medical students and residents/fellows.¹²  Survey items were developed by the authors (one medical student, one resident, one fellow, and one attending) based on prior experience sampling PDs regarding their perceptions of pass/fail Step 1 score reporting.⁸  Survey questions utilized 3-item Likert scales. Instruments were pilot tested with a diverse group from each target population (medical students: 4 White, 1 African American, 1 American Indian or Alaska Native, and 2 Asian American; residents/fellows: 3 White, 2 African American, 2 Asian American, and 1 respondent who self-identified as “Other”). Internal consistency reliability of each instrument was assessed using Cronbach's alpha. The final student and resident/fellow instruments can be found in supplemental data.

UiM trainees were defined as those who identified as African American, Mexican American, and Native American (ie, American Indians, Alaska Natives, and Native Hawaiians).¹³  To contextualize the proportion of UiM trainees in our cohort relative to the national proportion of UiM trainees, these numbers are also reported using national estimates.

The primary outcome of interest was to assess student and resident/fellow sentiment toward the change by asking if “Changing the USMLE Step 1 to pass/fail is a good idea.” Additionally, we assessed for potential improvements in socioeconomic disparities by asking if “Changing the USMLE Step 1 to pass/fail will decrease socioeconomic disparities in the residency application process.” Free-text responses were solicited to identify reforms that trainees believed would improve the residency application process. Lastly, trainee opinions regarding Step 2 Clinical Knowledge (CK) score reporting were examined.

Comparative analyses were performed using 2-tailed Fisher's exact test, and differences were considered statistically significant with a P value less than .05. Additionally, 95% CIs are reported for differences in subgroup responses.¹⁴  As this was exploratory research without prior hypotheses, 8 comparisons were made without adjusting for multiple comparisons, and the P level remained set at .05 for each comparison. Free text survey responses were reviewed by the authors to identify salient suggestions regarding residency application reforms.

Both surveys received exemption from the Vanderbilt University Institutional Review Board.

Both student and resident/fellow survey instruments demonstrated internal consistency (Cronbach's alpha 0.76 and 0.91, respectively). After invitation, 37 deans (19%) and 266 (32%) DIOs agreed to survey distribution. In total, 11 633 trainees (4379 medical students and 7254 residents/fellows) responded, of which 9303 were US allopathic or osteopathic trainees. With an estimated 232 000 allopathic and 53 000 osteopathic medical students and residents in the United States, our response rate approximates 3.3% of all US trainees.^15,16  Nationally, nearly 8.5% of DO medical students (∼2300) and 21% of MD medical students (∼20 000) identify as UiM.^17,18  In our national sample, 14% (∼600) of medical students identified as UiM.

Overall, 36% of respondents (4146 of 11 609) agreed that the score reporting change was a good idea and 27% (3133 of 11 589) agreed the change would decrease socioeconomic disparities in the residency application process (Table and online supplementary data). More medical students agreed with the score reporting change than residents/fellows (1891 of 4374 [43%] vs 2255 of 7235 [31%]; P < .001; 95% CI 0–24).

UiM trainees were more likely to agree with pass/fail score reporting compared to non-UiM trainees (493 of 986 [50%] vs 3653 of 10 623 [34%]; P < .001; 95% CI 0–32). Additionally, UiM trainees were more likely to agree that pass/fail score reporting would decrease socioeconomic disparities associated with Step 1 compared to non-UiM trainees (433 of 983 [44%] vs 2700 of 10 606 [25%]; P < .001; 95% CI 0–38). A greater proportion of UiM trainees agreed with pass/fail Step 2 CK than non-UiM trainees (366 of 980 [37%] vs 2468 of 10 562 [23%]; P < .001; 95% CI 0–28).

Fewer US osteopathic (DO) and international medical graduate (IMG) trainees agreed with the change compared to MD trainees (1190 of 4112 [29%] vs 2806 of 7120 [39%]; P < .001; 95% CI 0–20). DO and IMG trainees were less likely to agree that the score reporting change would decrease socioeconomic disparities in the application process (2097 of 4002 [52%] vs 2935 of 7111 [41%]; P < .001; 95% CI 0–22). Nearly twice as many DO and IMG students agreed they would be disadvantaged by their training backgrounds relative to US MD students (620 of 1024 [61%] vs 1012 of 3282 [31%]; P < .001; 95% CI 0–60; Table). DO/IMG students were also more likely to agree with being unable to distinguish themselves on the basis of merit without a numeric Step 1 score (678 of 1023 [66%] vs 1665 of 3282 [51%]; P < .001; 95% CI 0–30).

A common free-text suggestion to improve equity in the residency application process was capping the number of interview invitations an applicant may accept. Many respondents also suggested increasing transparency (eg, disclosure of score and medical school rank cutoffs) and standardization of application components (eg, Medical Student Performance Evaluation and letters of recommendation). See Table and online supplementary data for trainee opinions on USMLE Step 2 CK reporting.

This survey of allopathic and osteopathic students' as well as graduate medical trainees' perceptions of the upcoming change of USMLE Step 1 score reporting to pass/fail found that about one-third of students and trainees agree with the change. UiM trainees were more likely to support this change, whereas DO and IMG trainees were less likely to support the change in comparison with US allopathic trainees.

The effect of pass/fail Step 1 score reporting on UiM trainees remains an open question. While some have argued that pass/fail reporting will help address racial disparities in examination performance,⁵  others have suggested that the adoption of substitute heuristics will increase bias in the residency selection process.¹⁹  Our findings support the notion that pass/fail score reporting will have a positive impact on trainees from underrepresented backgrounds. UiM trainees were significantly more likely to support the change compared to non-UiM trainees. Moreover, UiM trainees were more likely to favor additional reforms aimed at reducing emphasis on standardized testing, such as pass/fail score reporting of Step 2 CK.

The score reporting change has prompted concerns in the IMG community. Because Step 1 numeric scores served as a standard metric of medical knowledge that could be compared among applicants regardless of medical school background, many IMGs are concerned they will be unable to distinguish themselves from US graduates.²⁰  As it stands, IMGs believe they will have a more difficult time securing residency positions.²¹  With Step 2 CK as the last remaining standardized test with numeric score reporting, many IMGs have expressed undue stress.²¹  This potential consequence contradicts one of the initial reasons for the score reporting change, which was to decrease the negative impact of Step 1 numeric scores on student well-being.^7,22  These perspectives may explain why the majority (59%) of IMGs in our study disagreed with the score reporting change.

Osteopathic students face similar challenges to IMG students. Lin et al²²  describe how, for students of osteopathic and international backgrounds, the loss of Step 1 numeric scores represents the loss of a key metric to ensure applications from these non-allopathic students are reviewed. Similarly, in a single institution cross-sectional survey of 215 medical students, osteopathic students were more likely to report that the pass/fail transition will negatively impact their residency match compared to allopathic students (adjust OR 1.454, 95% CI 0.515–4.106).²³  This sentiment is supported by our findings, as only a third of osteopathic trainees agreed with the change, and more osteopathic students felt they would be disadvantaged compared to their allopathic counterparts.

The primary limitation of this study is a low response rate, which is estimated using the largest possible denominator. The actual number of trainees that received the surveys is unknown. Thus, the respondents may not represent all medical students and GME trainees. In addition, other than pilot testing for clarity and consistency, the survey has limited validity evidence, and respondents may have interpreted questions differently than intended. In particular, demographic categories may not have included all possible responses. As this was an exploratory project, the P level of .05 was not corrected for multiple comparisons, and some differences may not be true. Next research steps may include qualitative approaches to examine concerns expressed by medical and osteopathic students regarding the upcoming reporting change, as well as mitigating strategies to ensure some students, such as osteopathic or international medical students, are not disadvantaged by the change.

Trainee perspectives regarding pass/fail USMLE Step 1 score reporting are mixed. Overall, a third of trainees agreed with the score reporting change, with more students than residents/fellows supporting the change. UiM trainees were more likely to support the change than non-UiM trainees. In contrast, DO and IMG trainees were less in favor of the change relative to MD trainees.