eSLOE 2.0: Examining Data From the First 2 Application Cycles of the Updated Emergency Medicine Electronic Standardized Letter of Evaluation

Recent changes in undergraduate medical education (UME) assessment, such as the increase of pass/fail grading schemes for core clerkship experiences and the transition from scored to pass/fail United States Medical Licensing Examination/Comprehensive Osteopathic Medical Licensing Examination of the United States Step 1 testing,^1,2  has led to fewer differentiating factors between residency applicants on residency applications. The Standardized Letter of Evaluation (SLOE), which was first developed in emergency medicine (EM) in 1995, was designed to create standardized and concise letters of recommendation to aid in distinguishing between residency applicants.³  With the Coalition for Physician Accountability recently recommending that structured letters of recommendation replace narrative letters of recommendation, we anticipate even more specialties adopting their own version of the SLOE.⁴  Therefore, the SLOE’s role in filling this gap in the residency application process for the broader UME and graduate medical education (GME) community is even more valuable now.

The SLOE has been analyzed over the years and has been shown to offer valuable insights into EM applicants.^5,6  In the last decade, additional specialties have started using standardized letters with similar goals in mind. Despite the SLOE’s success, challenges remain in the ability of a standardized letter to differentiate between applicants, with score inflation being cited in both EM and other specialties utilizing standardized letters.^7-9  Still, most specialties highlight the EM letter as the hallmark they looked to for guidance and inspiration.^10,11 

In recent years, the Association of American Medical Colleges’ (AAMC) mission has been to increase the use of entrustable professional activity (EPA)-based assessments to offer a practical approach to assessing competence in real-world settings.¹²  At the same time, the Accreditation Council for Graduate Medical Education (ACGME) has transitioned to competency-based Milestones to evaluate residents on 6 Core Competencies within medicine.¹³  In an effort to align the EM SLOE with this transition to competency-based assessment, which has been shown to be more objective and less biased,¹⁴  a section of the EM SLOE transitioned to a competency-based assessment. Prior normative-based sections were either revised or removed. Hereinafter, the EM SLOE was referred to as the eSLOE 2.0, in the 2022-2023 and 2023-2024 residency application cycles.¹⁵ 

In this study, we examine the data from the first 2 application cycles utilizing the eSLOE 2.0 format (2022-2023, 2023-2024) with a focus on the distribution of ratings in Parts A, B, and C. The intent is to provide preliminary validity evidence for the new EM eSLOE 2.0, which now includes both normative and competency-based assessments.

With the permission of the Council of Residency Directors in EM (CORD) Board of Directors, data were accessed from the eSLOE 2.0 letters that were submitted for the 2022-2023 and 2023-2024 residency application cycles. Raw data from the national eSLOE database (where all standardized letters are stored after completion) were transferred into a Microsoft Excel spreadsheet. The data were then de-identified and analyzed using basic statistical analyses (eg, sums, percentages). Of note, only eSLOE 2.0 letters, which are written by EM faculty based on a clinical EM rotation at an EM-based residency institution, were included in this data analysis. Non-residency-based, EM subspecialty, and off-service SLOEs were excluded from this analysis.

Data from the demographic section, as well as sections A, B, and C, were analyzed.

Part A of the eSLOE 2.0 asks letter writers to place applicants into the following categories: fully entrustable, mostly entrustable, or pre-entrustable in the areas of: ability to perform a focused history and physical examination, ability to generate a differential diagnosis, ability to formulate a plan, ability to perform common emergency department (ED) procedures, and ability to recognize and manage basic emergent situations.

Part B of the eSLOE 2.0 asks letter writers to place students on a 5-point Likert scale (1=minimally acceptable for an EM resident, 5=exceptional EM candidate) in the following areas: (1) compassion, sensitivity, and respect toward patients and team members; (2) receptivity to feedback and ability to incorporate feedback; (3) dependability, responsibility, initiative, and work ethic; (4) punctuality, attendance, and preparation for duty; (5) timeliness and responsiveness in completing administrative tasks; (6) interpersonal and communication skills with patients and family members; and (7) interpersonal and communication skills with faculty, residents, and health care professionals.

Part C of the eSLOE 2.0 asks 2 questions of the letter’s authors: (1) “How much guidance do you anticipate this candidate to require to become clinically proficient and meet graduation requirements?” (answer options: minimal, standard, moderate, or most) and (2) “How highly would you estimate the candidate will reside on your rank list?” (answer options: top 10%, top third, middle third, or lower third).

A complete eSLOE 2.0 can be found in the online supplementary data.

In total, 11 789 letters, representing 6543 unique applicants, underwent data extraction throughout 2 application cycles. The results have been aggregated from both years. The characteristics of the letters can be found in the Table.

In Part A, students were ranked as fully entrustable, mostly entrustable, or pre-entrustable for core emergency medicine skills (Figure 1). Results of Part A data can be found in Figure 1. Of note, 6.3% of rotations were unable to assess students in the category of ability to perform common ED procedures.

In Part B (Figure 2), students are ranked in each category on a 5-point Likert scale, with 5 being designated as the exceptional EM candidate and 1 as minimally acceptable for an EM resident. eSLOE writers were instructed that their average yearly ratings should typically look like a bell-shaped curve distribution. Results for section B data can be found in Figure 2. In Part B most students (81.7%-85.7%) were cumulatively ranked either 4 or 5 in each category. Given that so few students were designated not acceptable for EM resident (<0.0007% in each category), this designation was not included in the bar graph in Figure 2.

Part C (Figures 3 and 4) asks how much guidance is anticipated that an applicant will require during residency and how highly the letter writer would estimate the candidate would reside on their rank list. Data from Part C can be found in Figures 3 (guidance) and 4 (rank list). Of note, more than half of the letters indicated the applicant would be anticipated to need standard guidance. Approximately 57.7% (6789 of 11 789) of letters indicated an applicant was estimated to reside in the top 10% or top third of their rank list.

With the changes to the eSLOE 2.0, the letter now contains a mix of both competency-based items in Part A and normative-based items in Parts B and C. While the responses to normative-based questions in the eSLOE 2.0 continues the trend toward positive evaluations, as seen in prior versions of the SLOE, the entire scale is utilized for each question, allowing for some differentiation between applicants. At the same time, the incorporation of competency-based questions to the eSLOE 2.0 offers additional insights into an applicant’s anticipated needs and success for transition into residency.

In addition to asking where a student might fall on the rank list, previous versions of the EM SLOE asked letter writers to arbitrarily divide applicants into similar categories compared to their peers (lower, middle, top, and top 10%). While giving the appearance of being more discerning, this did not always paint an accurate picture or assessment of an applicant’s capabilities and had the potential to place applicants at risk in the Match.^16,17  These rankings were also prone to bias against candidates underrepresented in medicine.^18,19  We can also consider Section B of the eSLOE 2.0 format where the majority of students were evaluated at a 4 or 5 rating on the softer skills, such as communication and work ethic. While the eSLOE 2.0 format continues the trend of skewing evaluations more positively, this is not unique to EM. This trend has been noted not only in prior versions of the EM SLOE but also in UME letters of recommendation in general and also on other specialties’ standardized letters.⁵  Orthopedics and dermatology have found similar shifts to the “left” (more positive ends of the evaluation spectrum).^7-9  This begs the age old question of whether there is value in utilizing the full spectrum of a rating scale. Perhaps it is more likely that when raters use these higher ratings they are actually presenting a more accurate picture of applicants who are potentially already skilled in many of these domains.

Section A of the eSLOE 2.0, focusing on competency-based assessment, was designed to be more objective and less biased. While most students were found to be either fully or mostly entrustable in these categories, this is what we would expect for the majority of applicants given that most SLOEs are completed based on a fourth-year EM medical student rotation. This is a time when EM-bound students are likely at their peak medical school clinical performance and are likely completing multiple EM rotations, which enhances clinical competency-based skills. This section is currently closely aligned both with EPA-based assessments that are being utilized with increased frequency in both UME and GME settings, as well as the National Clinical Assessment Tool for Emergency Medicine, an EM-specific shift evaluation tool.²⁰  These observable clinical skills feed forward to competency-based medical education, a framework in which learners are assessed based on ability to attain competency with a task.²¹  They can also provide residency leadership with information regarding resident readiness,²²  so that residency programs can be primed to offer structured support to learners in previously identified areas of concern. Incorporating a competency-based section into letters for different specialties may hold significant value, as providing this “starting point” of clinical skills for incoming interns is necessary to ensure a smoother handoff between UME and GME. This is one of the large goals of the Coalition for Physician Accountability recommendations for comprehensive improvement of the UME-to-GME transition for all specialties.⁴ 

Regarding the question of anticipated guidance during residency, a larger percentage of students are marked as needing moderate/most guidance on the eSLOE 2.0 when compared to “more guidance than peers,” which was present on prior SLOE versions. It is possible that there has been a decrease in quality of EM applicants over the past 2 application cycles with the decreasing competitiveness of the specialty.^23,24  Another possible explanation for the increased use of “moderate” or “most” guidance compared to “more guidance than peers” may involve the explanation provided on the eSLOE 2.0 for “moderate” (may need slightly more than the standard support from time to time, no major issues anticipated) and “most” (has the potential to succeed, but will likely require extra support throughout residency) guidance. Both of these explanations have caveats suggesting the applicant has potential compared to the prior SLOE version, which could be interpreted in an entirely negative light; thus, a letter writer might be more willing to choose one of these newer options. This question may hold value for additional specialties developing standardized letters, as it also allows communication of anticipated needs, smoothing the transition from medical school to residency.

Limitations for this article include difficulties in defining what it means to be a “standardized letter.” Due to significant variability among students’ rotation experiences, advisor experience, and institutional evaluation tools, it can be difficult to know if you are comparing apples to apples when you look at different SLOEs. This article also explores the data from the first 2 eSLOE 2.0 cycles, but it is unable to make sweeping comparisons between this letter format and prior versions.

Looking forward, it may be worthy to focus efforts on how to increase letter writers’ use of the full Likert scale. As this is a recognized challenge experienced by many specialties, there may be value in attempting to adjust rater behaviors by prompting further consideration for both the highest and lowest ratings in certain normative categories. These efforts may lead not only to improved discernment between applicants for the EM SLOE and other specialties’ standardized letters, but also to a clearer and more helpful picture for residency programs.

This report describes data from the first 2 application cycles utilizing the eSLOE 2.0 format and provides preliminary validity data for this new letter format.

The online supplementary data contains a complete eSLOE 2.0.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.