Enhancing COMLEX-USA: Evidence-Based Redesign of the Osteopathic Medical Licensure Examination Program

As a licensing examination, the primary purpose of the Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) is to assure the public that candidates who have passed the examination series have met an established national standard of osteopathic medical knowledge and clinical skills, making them eligible to apply for the privilege of practicing osteopathic medicine in all 50 states. This primary purpose directly supports the end goal of the Federation of State Medical Boards (FSMB), to “keep patients safe.”1 

Initiated in 1995, the COMLEX-USA examinations have been developed around a patient presentation-based, two-dimensional blueprint, 2 a unique concept at its introduction and one that has since been employed by other medical licensing organizations.3,4 Following a comprehensive review of COMLEX-USA and the United States Medical Licensing Examination, the Special Committee on Licensing Examinations of the FSMB endorsed both examinations as valid and reliable for their intended and respective purposes, concluding the support for the validity of COMLEX-USA is exemplary.5 Recent studies have continued to provide validity evidence for COMLEX-USA. One such study reported that the COMLEX-USA has been shown to assist family medicine residency program directors in predicting later resident performance on both the American Board of Family Medicine's In-Training Examination and Maintenance of Certification for Family Physicians Examination.6 A summary of published evidence for the validity of the current COMLEX-USA program for licensure and other secondary uses can be found on the NBOME website.7 

The purpose of an examination blueprint is to outline the content and format specifications for the examination, including the percentage of total test questions and other content associated with each topic.8 In the years since its implementation, the COMLEX-USA blueprint has been reviewed and revised regularly to reflect the practice of osteopathic medicine, consistent with the recommendations of the Standards for Educational and Psychological Testing (Standards) established by the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME) in 2014.9 The addition of a performance assessment of fundamental clinical skills, COMLEX-USA Level 2 — Performance Evaluation, was added to the series in 2004. Additional test item formats have been developed and incorporated into the examinations. In 2018–2019, the NBOME plans to implement the enhanced test blueprint and specifications described herein.

In order to ensure that COMLEX-USA was most effectively positioned to anticipate new trends and changes in osteopathic medical practice and education, the NBOME commissioned the Blue Ribbon Panel in 2010. The panel included diverse representatives from various organizations, among them the American Association of Colleges of Osteopathic Medicine (AACOM), the FSMB, American Association of Osteopathic Examiners (AAOE) and the Accreditation Council for Graduate Medical Education (ACGME). The panel was charged with outlining a plan to implement a two–decision-point, competency-based COMLEX-USA that is consistent with NBOME's mission to protect the public by providing the means to assess competencies for osteopathic medicine and related health-care professions. Decision Point One was defined as the grouping of pre-doctoral assessments leading to successful promotion along the licensure pathway for entry into supervised practice (i.e., graduate medical education), while Decision Point Two was defined as a series of postdoctoral assessments leading to unsupervised practice (i.e., eligibility for licensure).

An important accomplishment of the panel was its recommendation of a revised COMLEX-USA blueprint consisting of two dimensions — the Competency Domains and the Clinical Presentations — updated from the current blueprint dimensions of Patient Presentations and Physician Tasks. As a high-stakes summative examination, COMLEX-USA follows the Standards, which emphasize the importance of validity as the fundamental consideration of a test and the importance of examination blueprints being job-related. The collection of evidence to support this requirement, a practice analysis, may include a variety of procedures such as a comprehensive review of relevant research, evaluation of expert opinion, and task inventory data.10 As a result, a major focus of the panel's work in developing the new blueprint was gathering and integrating multiple sources of evidence based on established standards and psychometric principles.

In initiating the practice analysis, the panel first focused on the Competency Domains. The importance of competency domains as a framework for good physician performance and medical practice has been emphasized by the Coalition for Physician Accountability.11 The panel reviewed previous NBOME research12,13 and conducted a literature survey that included a review of national and international medical competency models.14,15,16 An essential outcome of this research was the publication of FOMCD 2016, 17 revised from FOMCD 201118 to provide a clarified presentation of Required Elements and Outcomes that comprise the Competency Domains for osteopathic medical practice.

In the FOMCD 2016, Competency Domains were defined as “related sets of foundational abilities and represent the required elements and outcomes that define the knowledge, skills, experience, attitudes, values and behaviors of established professional standards” (FOMCD, 2016),17 providing an overall framework for the practice of osteopathic medicine. The Competency Domains Dimension includes the Measured Outcomes, statements of well-defined abilities, including knowledge, skills, experiences, attitudes, values, and/or behaviors that are observable and measurable and can be directly assessed in a reliable manner in the assessments that make up the COMLEX-USA examination program. In addition to competency domains such as patient care and application of knowledge, the new blueprint introduces emphasis on knowledge and skills related to patient safety, such as practice-based learning and improvement and systems-based practice. FOMCD 2016 further delineates Measured Outcomes, Attested Outcomes and Not-Measured Outcomes, some of which typically would require workplace-based assessment. While not currently assessed in the new COMLEX-USA, attested and not-measured outcomes could provide opportunities for further collaboration and research and eventually be included in an expanded portfolio for licensure.

In order to obtain current data describing the clinical practices of osteopathic physicians, the panel evaluated national survey data, including data from the 2010 National Ambulatory Medical Care Survey19 that reports medical care statistics related to osteopathic physicians' practice, such as patient demographics and visit characteristics. Comparing data from DO and MD physicians confirmed several important differences in practice patterns. For example, osteopathic physicians generally encounter more patients with musculoskeletal and respiratory presentations than MDs. The panel also evaluated data from the 2010 National Hospital Ambulatory Medical Care Survey20 and the 2010 National Hospital Discharge Survey,21 as well as the 2004 National Nursing Home Survey.22 

In addition to reviewing current research and national survey data, the panel sponsored additional practice and job task analyses research. A practice analysis survey23 asked respondents to rate the importance of the specific competency domain required elements at Decision Point One (entry into supervised practice, residency training) and Decision Point Two (entry into unsupervised practice, unsupervised practice/licensure). This survey was distributed to members of the Association of Osteopathic Directors and Medical Educators (AODME), AAOE, recent graduating classes from the nation's colleges of osteopathic medicine (COMs), as well as NBOME's National Faculty and Deans of the COMs. Survey results indicated that the highest rated elements at Decision Point One were a commitment to continuous learning and the ability to work effectively with other members of the interprofessional collaborative team. At Decision Point Two, the highest rated elements were the abilities to formulate a prioritized differential diagnosis and to gather data from all sources, including patients' secondary sources, records, and physical examination. These data were considered in test development including weighting of the blueprint topics.

Concurrent with the research on Competency Domains, the panel established the Blueprint Workgroup, a task force that included representative specialties from the osteopathic medical profession. The Blueprint Workgroup was charged with developing recommendations to the panel based on empirical research and their expert opinion for the development of the Clinical Presentations dimension, including its nomenclature and weighting. A Clinical Presentation was defined as the manner in which a particular patient, group of patients, or community present to osteopathic physicians.24 Clinical Presentations are high-frequency, high-impact categories based on evidence from osteopathic medical practice and are further classified as categories and topics.

Using their consensus expert opinion, the Blueprint Workgroup carefully considered the categories of the current blueprint's Patient Presentations, reorganized the categories, and modernized its language to propose the new Clinical Presentations. They evaluated national survey data19–22 and reviewed the program curricula of U.S. osteopathic and MD-granting medical schools as well as of international medical schools, graduate medical education objectives, and competencies and milestones. Using their clinical expertise and a consensus-driven process focusing on how patients present to osteopathic physicians, the Blueprint Workgroup further defined the topics and subtopics that composed the categories of the Clinical Presentation dimension. These recommendations were presented back to the panel and ultimately NBOME's Board of Directors for its consideration.

The outcome of the research and analysis conducted by the panel and its Blueprint Workgroup was the development of an enhanced, competency-based, two-dimensional blueprint that is aligned with the evolving practice of osteopathic medicine. It includes as its introductory preamble the tenets of osteopathic medicine:25 

• The body is a unit; the person is a unit of body, mind, and spirit.

• The body is capable of self-regulation, self-healing, and health maintenance.

• Structure and function are reciprocally interrelated.

• Rational treatment is based upon an understanding of the basic principles of body unity, self-regulation, and the interrelationship of structure and function.

Figure 1 illustrates the two-dimensional blueprint consisting of Competency Domains and Clinical Presentations.26 

Dimension One in the new blueprint24 includes the Competency Domains as delineated in the FOMCD 2016, previously described. However, the blueprint includes only the Competency Domains' Measured Outcomes, the well-defined abilities and behaviors that are observable and reliably measurable through a COMLEX-USA assessment. Neither Attested Outcomes nor Not Measured Outcomes are currently included in the new blueprint, although research is underway for the possibilities of future assessment. Dimension Two, the Clinical Presentations, contains 10 categories grouped by organ-body systems organizing the manner in which patients and communities present to osteopathic physicians. Ultimately, the weighting for Competency Domains and Clinical Presentations, based on the comprehensive analyses of osteopathic medical practice, was determined by the panel and is available on the NBOME website. Table 1 depicts the minimum weighting of Competency Domains and Clinical Presentations.27 

While the previous COMLEX-USA blueprint was also two-dimensional, there are significant differences between it and the new blueprint. As can be seen from Table 2, the previous blueprint's Dimension 1, Patient Presentations, has similarity to the new blueprint's Dimension 2, Clinical Presentations. However, in keeping with the current emphasis on competency-based assessment in medical education and practice as well as the recognized importance of competency domains as a framework for good physician performance, the new blueprint's Dimension 1 includes NBOME's Competency Domains. In both the Competency Domains and Clinical Presentations, the descriptors and the language have been modernized to reflect current medical language. Importantly, in an effort to provide additional detail, the NBOME developed a Guide to Clinical Presentations, which lists examples of presentations for each category. While the Guide does not present an exhaustive list, it provides relevant examples to provide greater transparency. In addition, the new blueprint will employ a minimum percentage per topic in assigning items to an examination.

In developing the Master Blueprint 2018–2019, the panel sponsored new research and evaluated existing research, considered expert opinion, and relied on psychometric best practices to create an evidence-based blueprint design for the COMLEX-USA examinations. Because the new Master Blueprint 2018–2019 calls for revised content with different weighting, its implementation may result in changes to the content of other assessments currently being administered in osteopathic medical education. With this in mind, the NBOME began communicating the Master Blueprint at various professional meeting venues beginning in 2017. Test specifications for each level and component of COMLEX-USA are currently available on the NBOME website to provide more detailed direction to stakeholders.

As shown in Table 3, the new blueprint began to be implemented beginning with Decision Point Two (COMLEX-USA Level 3) in September 2018.27 Several changes to the Level 3 examination have been implemented to ensure that candidates are ready for the privilege of medical licensure.

Beginning in September 2018, the COMLEX-USA Level 3 examination has been extended from one day to two days. The purpose of extending the examination duration from one day to two days is to permit a broader assessment of competency subsets that includes both traditional multiple-choice as well as novel item formats. Elements important to patient safety, such as practice-based learning and improvement, as well as professionalism and system-based practice are enhanced in this expanded assessment. One such novel item format is Clinical Decision Making (CDM),28,29 which focuses on assessing the critical steps or “key features” in decision making. CDM test items include clinical cases that use constructed response, short-answer response items as well as extended multiple choice formats. This item format is a major component of other medical licensure examinations, notably the Medical Council of Canada.30 As well, the NBOME is investigating the potential of point-of-care item formats, called Point-of-Care Knowledge, Education, and Testing (POCKET), that would assess candidates' use of electronic resources, such as clinical decision-making tools and resource applications typically available to physicians in actual clinical practice. The POCKET format is intended to assess measured outcomes included in the practice-based learning and improvement competency domain. Despite the addition of a second day of testing, the 2018 registration fee has remained the same for test-takers.

In addition, in order to register for the COMLEX-USA Level 3 exam, candidates will be required to obtain an attestation from their Residency Program Directors that they are meeting the academic and professional requirements of the residency program and are eligible to continue as residents in the program. This attestation is significant in that it underlines the importance of meeting both the professional as well as the academic expectations of a Residency Program Director. It also enables competency-based score reports to be issued to Residency Program Directors, fostering continuous professional development and aiding in documentation of residency program and specialty milestones.

The attestation requirement has been implemented with the new blueprint and the two-day Level 3 examination and is consistent with the professional expectations from numerous medical regulatory as well as teaching authorities, and concerns generated by highly publicized unprofessional or criminal conduct on the parts of physicians. This requirement appears to align with Policy Statement on Duty to Report by the FSMB's Ethics and Professionalism Committee.31,32 The importance of “Duty to Report” was emphasized during a recent plenary address by FSMB President and CEO Humayun Chaudhry at the AACOM's 2018 Educating Leaders conference.33 

The enhanced blueprint will be implemented for Decision Point One examinations, COMLEX-USA Level 1, Level 2-CE, and Level 2-PE, beginning in 2019. These examinations will remain similar in format, but will follow the new blueprint's content outline. In order to register for the COMLEX-USA Levels 1 and 2, a candidate will be required to obtain the attestation from his/her dean that the student is enrolled in good academic and professional standing at the college.

Now under the governance of NBOME's COMLEX-USA Composite Examination Committee (CCEC) and its Board of Directors, the NBOME is continuing its ongoing efforts to ensure that COMLEX-USA remains current in meeting the needs of the state medical licensing boards and other constituents. With the science and practice of osteopathic medicine changing so rapidly, NBOME policy charges the CCEC to review the new blueprint for the potential need for revisions on a biannual basis. Current data and practice trends will be reviewed by a group of content experts. A communication plan is in place to announce changes to relevant stakeholders in a timely manner. In keeping with its mission to protect the public, and in support of the state licensing boards and other stakeholders who rely on the COMLEX-USA examination for other important secondary purposes, the NBOME remains committed to the development, administration, and continuing quality improvement of the COMLEX-USA examination program.