A Call to Action for Standardizing Letters of Recommendation

In 1997, the Council of Residency Directors in Emergency Medicine (CORD) reported on the development and implementation of a standardized letter of recommendation (SLOR) by emergency medicine (EM) residency programs.⁷  Subsequently, other specialties including orthopedics and otolaryngology developed and implemented their own SLORs.^8,9  In 2014, the EM SLOR was updated and renamed the CORD Standardized Letter of Evaluation (SLOE) to better reflect its purpose in providing a summative evaluation of a candidate. Now considered the gold standard by CORD for evaluating potential candidates, the EM SLOE has 4 sections: Background Information, Qualifications for EM, Global Assessment, and Written Comments. Background Information includes the length of the evaluator-student relationship, the nature of their contact, and the student's end-of-rotation grade on a 5-point scale: Honors, High Pass, Pass, Low Pass, and Fail. The Qualifications for EM section asks the evaluator to compare students to their peers on 7 criteria: commitment to EM, work ethic, differential development and treatment plan, teamwork, communication skills, guidance needed during residency, and likelihood of success. The Global Assessment section asks evaluators to rank students in comparison to others they recommended in the prior academic year: top 10%, top third, middle third, and lower third. Finally, the Written Comments section allows evaluators to expand on students' noncognitive attributes.¹⁰  Students must obtain a SLOE, completed by faculty trained in writing SLOEs, for each EM rotation they complete and submit it with their residency applications. Like narrative LORs, students may opt to waive their right to view the SLOE. In 2020, the Association of Professors of Gynecology and Obstetrics piloted the obstetrics and gynecology (OB/GYN) SLOE followed by full implementation in 2022; the sections of the OB/GYN SLOE include Competency Assessment Rubric, Most Outstanding Feature, Areas of Focus, and Narrative.¹¹ 

While SLOEs have been adopted by several specialties, most letters submitted for GME applicants use narrative approaches. Program directors identify narrative LORs as critical to the selection process; however, narrative LORs have not been found to reliably provide substantial information on applicants' competencies required by the Accreditation Council for Graduate Medical Education, discriminate between candidates, or predict future performance.^12-20  While there are guidelines on how to write narrative LORs, a paucity of literature exists to address the optimal content for narrative LORs to assist faculty who often have little or no training in writing narrative LORs.^21,22 

The number of applicants to GME programs across the United States has been steadily rising.^23-34  Performing a thoughtful, holistic review of each candidate's application and selecting who to invite for interviews is a challenging task. Studies performed in various specialties have identified significant linguistic differences in narrative LORs, including infrequent use of language describing strong leadership traits for women and UIM candidates, suggesting that implicit biases related to both candidates and letter writers exist.^25-34  Several studies have demonstrated that SLOEs may be better than narrative LORs at mitigating implicit biases that adversely affect women and UIM candidates.^33,35-41  Additionally, the SLOE represents a potential tool to better differentiate applicants; Jackson et al demonstrated that the electronic EM SLOE provided more discrimination, even above the original SLOR, with improved spread of rank categorization.^42-44  While SLOE implementation may standardize applications and enable candidate comparison in a more objective and unbiased manner, the potential remains for inflation of applicants' qualifications by authors; medical schools and training programs have an interest in helping their graduates secure preferred positions.^38-40  It will be imperative to invest time and resources to provide faculty development related to the writing and interpretation of SLOEs for selection committees.

To develop and adopt SLOEs across more specialties, it is important to learn from the prior experience of GME programs where implementation has been successful.^41,45-47  The Table suggests an action plan with concrete steps that program leaders can take. The total number of applications that selection committees must review within a short time frame has risen sharply. For example, in 2022, 7 of the 17 pediatric subspecialties received, on average, over 100 applications per program, with pediatric emergency medicine (PEM) being one of the most competitive subspecialties. PEM program directors from across the country meet semiannually. During the most recent meeting, we polled attendees, and the overwhelming majority were interested in the development of a subspecialty-specific SLOE. Other discussion points included ensuring the SLOE encourages holistic review of candidates, is designed to meet the needs of institutions where PEM specialists are not present, and is prefaced by faculty development and training. The authors plan to continue this process and hope the effort will be expanded to engage members of the broader GME community.