As the only peer-reviewed quarterly publication dedicated to topics and issues critical to medical regulation in the United States and around the world, the Journal of Medical Regulation addresses all aspects of this vital element of public protection. JMR's Assessment Briefs offer periodic updates on important developments in the examinations relied upon for medical licensure in the United States and globally.
United States Medical Licensing Examination (USMLE)
The United States Medical Licensing Examination® (USMLE®), established in 1991 as a joint program of the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), is a three-step examination for medical licensure in the United States.
Step 1 Transition to Pass/Fail: In early 2020, the USMLE program announced that effective January 26, 2022, all Step 1 examinations taken on or after that date would no longer report a numeric score but only a pass/fail outcome. This change in score reporting stemmed from the program's concerns in multiple areas, such as the overemphasis on Step 1 numeric scores as a screening tool in determining residency interviews and selection, Step 1's outsized impact on the medical education curriculum, and the effect on student wellness.
This score reporting change garnered considerable attention and continued interest among all stakeholders, including its primary users, the state medical and osteopathic licensing boards. A common question involves changes to examinee performance. Aggregate performance data is presented below before (2021) and after (2022) score reporting change.
| Step 1 Exam | 2021 | 2022 |
|---|---|---|
| First time takers | ||
| US/Canadian MD degree | 96% | 93% |
| US/Canadian DO degree | 94% | 89% |
| International | 82% | 74% |
| All Takers | 88% | 82% |
| Step 1 Exam | 2021 | 2022 |
|---|---|---|
| First time takers | ||
| US/Canadian MD degree | 96% | 93% |
| US/Canadian DO degree | 94% | 89% |
| International | 82% | 74% |
| All Takers | 88% | 82% |
(Note: Data through the first 9 months of 2023 is comparable to that from 2022) Complete aggregate performance data on all Steps can be found at https://www.usmle.org/performance-data.
Several elements should be mentioned in considering this data, starting with what did not change for Step 1 in 2022: the exam blueprint and the “difficulty” of each test form. At the same time, caution is warranted when comparing 2022 data to previous cohorts. There were substantial changes in examinee demographics and testing behavior. A larger number of examinees tested immediately before and after the transition to pass/fail reporting in January 2022—more than the typical volumes in those timeframes. Moreover, nearly 6000 more international medical graduates completed Step 1 in 2022 compared to 2021. These shifts in examinee behavior and demographics make the Step 1 performance in 2022 less directly comparable to previous years.
It should also be noted that the regularly scheduled standard setting for Step 1 coincided with this change. The USMLE Management Committee's decision to change the minimum passing standard for 2022 (increased by 2 points to 196) likely contributed to the Step 1 performance differential. Finally, end-of-exam survey data detected changes in examinee pre-test behavior for Step 1, including a decrease in the self-reported start of pre-exam preparation activities and the daily number of hours spent preparing for the exam.
All of these factors, along with disruptive elements of COVID-19 to the medical education curriculum and experience in 2020-2021, further complicate data comparisons and interpretation.
The USMLE program continues advocating for additional systemic change that improves the undergraduate-to-graduate medical education transition process. There are no plans to revert the Step 1 pass/fail only decision or cease reporting numeric scores on Step 2 CK or Step 3.
USMLE Performance Associations with Patient Outcomes: A recent article published in Academic Medicine showed higher performance on the United States Medical Licensing Examination® (USMLE®) was associated with improved patient outcomes1 . The study analyzed nearly 200000 hospitalizations of 5 common inpatient diagnoses in Pennsylvania between 2017 and 2019. These patients saw more than 1750 family physicians and general internists. Results showed that higher physician USMLE performance across the series of exams was associated with lower mortality and shorter length of stay. This research further supports the validity of USMLE to assess competencies essential to safe and effective care and the importance of the exam's role in medical licensure.
Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA)
The Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) is a three-level, national standardized licensure examination designed for the practice of osteopathic medicine. COMLEX-USA assesses osteopathic medical knowledge, knowledge fluency, clinical skills, and other competencies essential for practice as an osteopathic generalist physician. It is a graduation requirement for attaining a DO (Doctor of Osteopathic Medicine) degree from colleges of osteopathic medicine (COMs) in the United States.
More than a year after the implementation of pass/fail score reporting only for the COMLEX-USA Level 1 examination, feedback from students suggests that the intended outcome of reduced test preparation stress for Level 1 candidates has been achieved but may have been accompanied by additional stress leading up to the Level 2-CE. Pass rates overall for Level 1 dipped slightly in the 2022-2023 testing cycle but have recovered in the current cycle. Research evaluating the impact of the change is ongoing.
The NBOME continues to pilot the Core Competency Capstone for DOs (C3DO), with expansion to 8 COMs in 2024. C3DO is a standardized assessment of patient-physician communication and interpersonal skills, history building and hands-on physical exam, and osteopathic manipulative treatment (OMT) skills in a multi-station objective structured clinical encounter (OSCE) model using standardized patients that is developed and scored centrally but delivered at the colleges. Feasibility studies in 2023 were promising for the future of this model. Phase 2 studies in 2024 will focus on the reliability and validity of the assessment.
In the absence of a current standardized osteopathic clinical skills assessment in the COMLEX-USA licensure series, COM deans continue to attest that their graduates have the fundamental osteopathic clinical skills necessary for graduation as part of the COMLEX-USA Level 3 eligibility. Program director attestation for Level 3 eligibility remains in effect.
The NBOME continues to adapt COMLEX-USA to the changing practice of osteopathic medicine and the landscape of patient care. The NBOME Reproductive Health Task Force released recommendations in October 2023 following the Dobbs v. Jackson ruling to support currency, fairness, and validity of content surrounding reproductive health. The NBOME developed and published a public Mindful Language Guide to demonstrate continued commitment to bias-free language in all assessments. The NBOME is actively developing applications for artificial intelligence in its assessment activities. Examples include the use of natural language processing in scoring, content development, and assessment design. Research to expand the reliable and safe use of these technologies is an organizational priority.
The NBOME National Faculty includes practicing osteopathic physicians, residents, educators, researchers, and assessment experts who are critical to our mission of protecting the public through the creation and review of test items, cases, and rationales, standard setting activities, and committee participation to ensure quality and fairness of our assessments. This includes state licensing board representatives, who provide thoughtful and unique perspectives for assessment development and validity activities. For more information or to apply, e-mail [email protected].