To the Editor: Limitations and Alternative Solutions to a USMLE COMLEX-USA Concordance

To minimize the perceived need for osteopathic medical students to take both the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) and the United States Medical Licensing Examination (USMLE), Barnum et al, representing the National Board of Osteopathic Medical Examiners (NBOME), published a study that concords score ranges between the analogous USMLE and COMLEX-USA assessments.¹  The intent of the proposed concordance to reduce student burden is laudable. However, the technical limitations of aligning COMLEX-USA and USMLE scores do not support this approach. Data constraints and test design variations make creating an adequate concordance untenable. Given USMLE's overweighted secondary use in graduate medical education due to the absence of alternative standardized data, residency program directors and other potential users should be aware of the serious limitations when interpreting NBOME's independently developed concordance.

Ahmed and Carmody covered 2 meaningful data constraints.²  First, the self-selected high ability sample may create an unrepresentative concordance. The distinctiveness of examinees who take both examinations presents an issue, as the concordance will be applied only to examinees outside this group—those choosing not to take the USMLE. Second, the lower than recommended correlations for a concordance lead to imprecise score conversions.³  Barnum et al acknowledge that the results do not imply perfect predictions.¹  However, the high stakes for students, programs, and the public associated with the use of this information demand precise score comparisons.

Furthermore, both the COMLEX-USA and USMLE are designed to support state medical boards in making licensure decisions. Medical boards in 43 states accept either the USMLE or COMLEX-USA as a measure of competency for osteopathic licensure. A concordance demonstrating that the passing standard on one examination corresponds to below passing on another examination, as observed in the published concordance, may create meaningful dilemmas for many stakeholders.

Another significant technical challenge is that COMLEX-USA scores are not directly comparable over time. The COMLEX-USA score scale maintains a static minimum passing score (eg, Level 2-CE is set at 400). As the same score reflects a different performance after a standard change, which is reviewed every 3 to 5 years,⁴  each modification results in scores that are not comparable to past COMLEX-USA scores or to USMLE scores.

A final challenge is that the COMLEX-USA and USMLE have independent governance structures that make design and content improvements asynchronously. To the extent these changes reflect distinct and substantive alterations, they would likely amplify variations in score comparison over time.

Concordance does not represent the most direct solution to improve the systemic pressure placed on students to take multiple examinations. Greater transparency about filter and selection practices at individual residency programs would empower students to take a more data-driven approach to residency applications. Application caps also could augment this to simplify the challenging selection task faced by program directors. Both the NBOME and USMLE have supported the growing calls for program directors to integrate examination scores with other available data to make informed, holistic residency selection decisions.^5,6  We note that perhaps the most comprehensive solution to eliminating the pressure to take multiple examinations, creating structural equity, and reducing the costs to all examinees involves reviving prior discussions toward a single pathway to medical licensure that addresses the needs of all aspiring physicians regardless of medical degree awarded.⁷  After all, this was the original basis for creating the USMLE program.