Racial Bias on the Emergency Medicine Standardized Letter of Evaluation

Racial discrimination and implicit racial bias are widespread throughout medical education, resulting in disparities in assessment measures and the residency Match.^1-6  Studies demonstrate evidence of racial/ethnic bias in grading with an association between lower clerkship grades and non-White race/ethnicity,²  and significant systematic differences exist in the language used to describe White vs Black applicants on the Medical Student Performance Evaluation (MSPE).³  Further, social determinants of learning can disproportionately and negatively influence the standardized test scores of underrepresented in medicine (UIM) students.^7,8 

Importantly, the impact of racial/ethnic disparities in assessment have created inequities in the residency Match. Studies have shown that a higher proportion of Black⁵  and UIM students⁶  are denied residency interviews compared to White or non-UIM students when using a minimum cutoff score for United States Medical Licensing Examination (USMLE) Step 1. As Step 1 transitions to pass/fail grading and more specialties develop a Standardized Letter of Evaluation (SLOE) to assess residency applicants, the effect of racial/ethnic bias on the SLOE and the potential for exacerbating racial/ethnic inequities in the Match must be considered.

In 1995, the emergency medicine (EM) SLOE was created to provide a more standardized and less biased assessment of medical students' clerkship performance.⁹  It consists of the following (see online supplementary data for an example SLOE):

1. “Rank Against Peers”: students are ranked against all other students applying to EM residency who were assessed by the SLOE author

2. Predicted placement on the institution's Match list

3. Grade on the EM rotation on which the SLOE is based

4. Qualities necessary for success in EM, ranked against peers

5. Narrative portion

The EM SLOE has become the component of an application that program directors value most when selecting students to interview and rank.^10-12  Other specialties (including otolaryngology, dermatology, orthopedics, and obstetrics and gynecology) have adopted a SLOE for their residency selection process, and the Coalition for Physician Accountability (COPA) recommends that all specialties cease using a narrative letter of recommendation in favor of a standardized evaluation letter.¹³  Understanding the influence of racial/ethnic bias on the EM SLOE could have broad implications for applicants across multiple specialties.

A subgroup analysis of a recent study comparing the EM SLOE to the Standardized Video Interview found that rankings on the EM SLOE “slightly favored White applicants”¹⁴ ; however, no study has specifically examined racial/ethnic bias in the EM SLOE. The primary aim of this study is to determine whether there is a difference in rankings on the EM SLOE between UIM and non-UIM applicants and to examine whether differences persist after controlling for other factors in the application. The secondary aim is to determine whether there are SLOE ranking differences between non-Hispanic White and non-White applicants.

This is a retrospective quantitative document review study. The sample was drawn from the students who applied to the study institution's EM residency program (a Midwest, urban, university-based, 3-year program with 18 residents per class) in 2019, including all US MD, US DO, and international medical graduate (IMG) applicants. This represents 39% of the total EM applicant pool¹⁵  and 55% of the US MD EM applicant pool in 2019.¹⁵ 

We obtained data from the Electronic Residency Application Service (ERAS) file of each student. SLOE rankings, student race/ethnicity, type of school (US MD/US DO/IMG), gender, USMLE Step 1 score, affiliated program vs visiting clerkship SLOE, Alpha Omega Alpha (AOA) status, and MSPE class percentile were paired and de-identified. Students self-identify their race/ethnicity on ERAS, with the following options: “American Indian or Alaska Native,” “Asian,” “Black or African American” (Black), “Hispanic, Latino, or of Spanish Origin” (Hispanic), “Native Hawaiian or Other Pacific Islander,” “White,” “Other,” or “Unknown.” Students may select as many as needed, or leave it blank.

The 3 main variables on the SLOE are:

1. Rank Against Peers (RAP), ranging from top 10%, top 1/3, middle 1/3, lower 1/3

2. Rank List Prediction (RLP), ranging from top 10%, top 1/3, middle 1/3, lower 1/3, unlikely to rank

3. Grade, which generally ranges from Honors, High Pass, Pass, Fail

The RAP and Grade are 4-point ordinal scales; the RLP is a 5-point ordinal scale. A ranking of one correlates with top 10% on RAP and RLP, and Honors on Grade.

The primary outcome is a comparison of the RAP, RLP, and Grade between UIM (defined as students who identified as Black, Hispanic, American Indian/Alaska Native, and/or Native Hawaiian/Pacific Islander on their application)⁸  and non-UIM students, using the Wilcoxon rank sum test. Using ordinal regression, we controlled for USMLE Step scores, AOA status, type of school (US MD, US DO, IMG), MSPE class percentile, affiliated program vs visiting clerkship SLOE, gender and the interaction of race/ethnicity and gender, and adjusted for students submitting multiple SLOEs (students generally obtain at least 2 SLOEs, one from their affiliated program clerkship and one after a visiting clerkship, with some students completing multiple visiting clerkships). Because MSPE class percentiles are variable between schools (quartile, tertile, etc), percentiles were converted to an ordinal ranking with one being the top ranking.

The secondary outcome is a comparison of the RAP, RLP, and Grade between non-Hispanic White (White) students and Black, Asian, and Hispanic students, using the Wilcoxon rank sum test. Ordinal regression was used to control for the above performance variables. Statistics were performed using R Studio (RStudio, Boston, MA).

This study received Insitutional Review Board exemption from the University of Chicago and the University of Illinois at Chicago.

In 2019, 1555 students applied to the study institution's EM residency program. Of these, 1493 applicants submitted at least one SLOE and, of those, 1418 students self-identified their race/ethnicity. In all, 3515 SLOEs were available for analysis from applicants who self-identified their race/ethnicity.

For each outcome variable, we assessed only the SLOEs in which that variable was assigned. There were 3507 SLOEs assigning a RAP, 3389 assigning an RLP, and 3041 that assigned a Grade and were not graded as pass/fail (SLOEs graded as pass/fail were not included in the grade analysis, as only 13% of SLOEs were graded pass/fail and, of those, 100% of students received a pass).

Table 1 represents the demographics of the applicants in our sample who had at least one SLOE and self-identified their race/ethnicity and the demographics of the national EM applicant population for 2019.

The median rank for RAP for the cohort was 2 (interquartile range [IQR] 2-3). Table 2 represents the percent of SLOEs with each rank by UIM status and race/ethnicity.

SLOEs from UIM students had significantly lower rankings on the RAP section of the SLOE compared with SLOEs from non-UIM students (P<.05). After controlling for gender, the interaction of race/ethnicity and gender, MSPE class percentile, AOA status, type of school, affiliated program vs visiting clerkship SLOE, and Step 1 score, non-UIM students were significantly more likely to be ranked higher than UIM students (OR 1.46, 95% CI 1.03-2.07).

SLOEs from Asian, Black, and Hispanic students had significantly lower rankings on the RAP section compared with SLOEs from White students (P<.05). After controlling for the predictors above, White students were significantly more likely to be ranked higher than Asian students (OR 1.46, 95% CI 1.00-2.13) and Black students (OR 1.81, 95% CI 1.15-2.83), respectively. There was no difference between White and Hispanic student rankings after controlling for the predictors.

The median rank for RLP for the cohort was 2 (IQR 2-3). Table 3 represents the percentage of SLOEs with each rank by UIM status and race/ethnicity.

SLOEs from UIM students had significantly lower rankings on the Predicted Match section of the SLOE compared with SLOEs from non-UIM students (P<.05). After controlling for gender, MSPE class percentile, AOA status, type of school, and affiliated program vs visiting clerkship SLOE, non-UIM students were significantly more likely to be ranked higher than UIM students (OR 1.30, 95% CI 1.04-1.62). After controlling for Step 1 score and the interaction of race/ethnicity and gender, there was no difference between non-UIM and UIM rankings.

SLOEs from Asian, Black, and Hispanic students had significantly lower rankings on the Predicted Match section compared with SLOEs from White students (P<.05). After controlling for the above predictors, White students were significantly more likely to be ranked higher than Asian students (OR 1.49, 95% CI 1.04-2.15). After controlling for gender, MSPE class percentile, AOA status, type of school, affiliated program vs visiting clerkship SLOE, and Step 1 score, White students were significantly more likely to be ranked higher than Black students (OR 1.41, 95% CI 1.12-1.78). After controlling for the interaction of race/ethnicity and gender, there was no difference between White and Black student rankings. There was no difference in rankings between Hispanic and White students after controlling for predictors.

The median ranking for Grade for the cohort was 2 (IQR 1-2). Table 4 represents the percentage of SLOEs with each Grade by UIM status and race/ethnicity.

SLOEs from UIM students had significantly lower rankings on the Grade section of the SLOE compared with SLOEs from non-UIM students (P<.05). After controlling for gender, the interaction of race/ethnicity and gender, MSPE class percentile, AOA status, type of school, affiliated program vs visiting clerkship SLOE, and Step 1 score, non-UIM students were significantly more likely to rank higher than UIM students (OR 1.46, 95% CI 1.05-2.04).

SLOEs from Asian, Black, and Hispanic students had significantly lower rankings on the Grade section compared with SLOEs from White students (P<.05). After controlling for the above predictors, White students were significantly more likely to rank higher than Black students (OR 1.65, 95% CI 1.08-2.52). There is no difference in grades between Asian or Hispanic students and White students after controlling for these predictors.

Rankings on the EM SLOE were lower for UIM students compared to non-UIM students, Black students compared to White students, Asian students compared to White students, and Hispanic students compared to White students across all studied measures on the SLOE. Lower rankings for UIM students compared to non-UIM students on RAP and Grade persisted even after controlling for other factors, including the MSPE class percentile and Step 1 score. Further, when controlling for other factors, Asian and Black students were ranked lower than White students on RAP, Asian students were ranked lower than White students on RLP, and Black students received lower grades than White students. These results demonstrate racial/ethnic bias on the EM SLOE, for both UIM students compared to non-UIM students, and Asian, Black, and Hispanic students compared to White students.

Our results are consistent with the racial/ethnic bias found in other forms of medical student assessment and add to the literature in 2 ways. First, we found that disparities in rankings by race/ethnicity persist on the EM SLOE after controlling for multiple measures of competency, suggesting that these disparities cannot be attributed to differences in clinical performance. Second, previous studies demonstrating racial/ethnic bias examined assessments in which there is a wide variability between schools.^2,4  Our study shows that similar racial/ethnic bias is present on clinical assessments that are standardized throughout the country. The recent statement from COPA lists “the presence of individual and systemic bias” as one reason for all specialties to adopt a SLOE.¹³  This study demonstrates that simply making an assessment standardized does not eliminate racial/ethnic bias and further action via a systemic, anti-racist strategy is necessary.

One way to approach this problem is through a recently published framework to address systemic change through an anti-racist lens: See, Name, Understand, Act.¹⁶  The results from our study allow us to “See” the problem. Program and clerkship directors, in all specialties using a SLOE, need to acknowledge the limitations of a SLOE in assessing students of different racial/ethnic backgrounds and the potential to exacerbate racial/ethnic inequity in the Match by placing too much emphasis on a specific metric rather than conducting holistic review.^17,18 

There are several limitations to this study. First, this study was conducted by convenience sample, not randomization, therefore there may be selection bias. The SLOEs studied were submitted to a single institution, therefore results may not be generalizable to the entire applicant pool. Specifically, our sample weighs heavily toward US MD applicants and underrepresents US DO and IMG applicants. Second, we only analyzed SLOEs that were submitted in applications, thus SLOEs that students purposely did not submit, due to perceived bias or other reasons, were not included. Third, due to the low number of students identifying as American Indian/Alaska Native or Native Hawaiian/Pacific Islander, we did not have the power to include them in the White vs non-White analysis. They were, however, included within the UIM group for the UIM vs non-UIM group analysis. Finally, the race/ethnicity of the SLOE author may also contribute to the presence or absence of bias. Although 68% of the US EM workforce identifies as White,¹⁹  to our knowledge no data exist describing the race/ethnicity of SLOE authors, and authors do not self-identify their race/ethnicity when creating a SLOE.

Future work can be directed by the rest of the anti-racist framework—“Name,” “Understand,” and “Act.” Leaders in education need to “Name” the problem. The consistency of findings across many different assessment measures in medical education demonstrate that the identified racial/ethnic disparities are not isolated to specific situations, institutions, or assessment tools. Education leaders must be willing to name the more insidious factors leading to assessment disparities by race/ethnicity across the spectrum of assessment, including systemic racial/ethnic inequalities and social determinants of education.^7,20  Next, we need to further “Understand” the problem to propose effective solutions. The structure of the SLOE may contribute to racial bias and represents an opportunity to further understand how changes to an assessment tool can affect equity. The current norm-referenced “Rank Against Peers” likely introduces bias into the assessment that could be mitigated by changing to a criterion-based assessment. Recent work suggests that using a “deficit-based” approach to assessment (such as putting students into a lower third ranking) compared to utilizing a competency-based approach to assessment may “disproportionately disadvantage UIM learners.”²¹  Additionally, literature suggests that any assessment utilizing a comparison to peers is inherently inequitable.²²  Changing to a criterion-based system would require writers to anchor their assessments of students to specific competency descriptors and reduce subjectivity. Finally, these findings should be a call for further immediate “Action,” including using group SLOEs with diverse faculty representation²¹  and committing to exploring solutions to the pervasive racial bias present in medical student assessment uncovered by us and others.^1-8  As multiple specialties currently use and more continue to adopt the SLOE, this will not be a problem limited to EM.

Rankings on EM SLOEs submitted to the study institution demonstrate disparities by race/ethnicity after controlling for other measures of competency and achievement.