Development of a new examination for the Canadian Chiropractic Examining Board

The Canadian Chiropractic Examining Board (CCEB) is incorporated under the Canada Not-for-profit Corporations Act, with 10 members corresponding to the provincial chiropractic regulatory bodies of Canada. The CCEB conducts entry-to-practice examination on behalf of its members and ensures fair and defensible evaluation of candidates using psychometrically valid and reliable processes. (In this context “valid” refers to the need for candidate assessment criteria to directly reflect the needs of practice, and “reliable” refers to the need for assessment methodology to yield consistent information when used repeatedly over time and candidate cohorts¹ ).

The first administration of the Canadian National Board Examination (as it was then known) took place in July 1963.²  This followed groundwork by the Education Committee of the Canadian Chiropractic Association (CCA), including a practice analysis. The examination involved 10 separate long-answer, essay-format components, each with a different content area.

In 1966 the name of the examining entity was changed to the Canadian Chiropractic Examining Board, and a long-term goal was set to make the CCEB arms length from the CCA, in both finances and governance.

Annual candidate numbers grew from 9 in 1963 to 182 in 1982, while the examination format remained substantially unchanged. By 1985 a total of 2167 candidates had attempted the examination. In a 1985 report, CCEB resolved that in the future further collaborative effort should be made to ensure that the content of the examination was jointly determined by the profession and the scientific community.

On February 14, 1991 CCEB formally secured the autonomy that had long been desired and became federally incorporated. A period of organizational modernization and examination evolution followed, including the incorporation of a modified Delphi process³  into examination structuring decisions. By 1997 the examination remained based upon job analysis data; however, the 10-topic essay format had been replaced by 3 overarching subject areas: basic science, applied science, and chiropractic science. In 1998 a clinical component was incorporated into the examination, first on behalf of a single province, later to evolve into Objective, Structured Clinical Examination (OSCE)⁴  testing Canada-wide.

Examination content was substantially refocused through a comprehensive Examination Blueprint Validation Study in 2008–2009. The study included a core competency survey, ratings of clinical conditions for frequency of occurrence, and a study of the curricula of chiropractic education institutes from around the world. Expert committees worked under psychometric guidance to produce specifications for the examination’s 3 components:

• Component A: (day-long) incorporating approximately 200 multiple-choice questions focusing on knowledge of the health sciences.

• Component B: (day-long) incorporating approximately 200 multiple-choice questions focusing on clinical decision-making.

• Component C: consisting of 10 12-minute OSCE stations involving up to 24 model patient presentations.

Over the period 2012–2019 each component of the examination was administered 3 times annually. By way of example, Table 1 lists the candidate numbers in the 2018–2019 cycle. Early in 2020 the normal examination delivery cycle was interrupted by the COVID-19 pandemic.

In November 2018 the Federation of Canadian Chiropractic (FCC) published “Entry-to-Practice Competency Profile for Chiropractors in Canada.”⁵  The FCC is a national group made up of regulators, education institutions, and chiropractic specialty colleges whose mission is promoting communication, cooperation, and education in the regulatory and accreditation processes of chiropractic in Canada. The profile was developed over the period 2013–2018 and is intended as a guide to ensure that education program curricula and learning outcomes meet the needs of entry-level practice.

The drive to develop an entry-to-practice competency profile for chiropractic began with the creation in 2013 of new competency-based accreditation standards for doctor of chiropractic education programs. That work was undertaken by the Accreditation Standards and Policies Committee of FCC’s Council on Chiropractic Education (Canada), in response to a review of the increasing popularity of competency-based educational frameworks among health professions around the world.⁶ 

The Accreditation Standards and Policies Committee examined competency-based models of medical education in Canada, the United States, and the United Kingdom, and concluded that the best fit for chiropractic in Canada would be to adopt the CanMEDS competency framework developed by the Royal College of Physicians and Surgeons of Canada.^7,8  CanMEDS was well established in medicine, and in some other health professions in Canada, by that time.

In reviewing the 7 standard roles around which the CanMEDS framework is based, the Accreditation Standards and Policies Committee modified “Expert” to “Neuromusculoskeletal Expert” (which it saw as being more appropriate for the chiropractic profession) and aligned with the term “Leader,” which at the time was in transition within CanMEDS from “Manager” (Table 2).

FCC then developed “key competencies” for each of the roles in the chiropractic competency framework, and identified “enabling competencies” as components, or steps, to demonstrating the key competencies. The publication by FCC of the entry-to-practice competency profile created a renewed impetus for CCEB to update its examination and to firmly establish the examination’s roots in the new chiropractic competencies. This paper describes the process used by the CCEB to restructure the examination to ensure it was current and to align it with the 2018 Federation of Canadian Chiropractic’s Canadian Chiropractic Entry-to-Practice Competency Profile.

We define a competency as an ability required of a chiropractor to enable safe, effective, and ethical practice. For the purposes of restructuring the examination, we aligned this definition with FCC’s enabling competencies. The enabling competencies (70 in all) were selected because collectively they provide a sufficient level of detail to enable meaningful evaluation of candidate abilities in a preregistration context.

Prior to initiating the examination updating project, CCEB had stated its intention to continue with examination methodology comprising multiple-choice questions (MCQ) and in-person OSCE stations. This decision had been made for several reasons:

• To continue the established practice in Canadian chiropractic whereby testing administered by CCEB supplements clinical testing within the internship component of accredited education programs and jurisdiction-specific testing undertaken by provincial regulators.

• To enable continuing use of already established CCEB test items that remain relevant going forward.

• To maintain alignment with high-stakes registration examinations commonly used in other North American healthcare professions.

Given that direct demonstration within the examination of many enabling competencies would not be possible, we define an indicator as an observable behavior of an examination candidate in a test situation, which suggests that the candidate possesses an enabling competency.

Indicators play a critical part in guiding item development since they identify aspects of competencies that are observable within the examination methodologies utilized. Indicators aid the efficiency of the item writing process by focusing item writers; they also facilitate transparency of examination content in that they enable candidates a clear view of what may be expected of them.

To develop indicators, CCEB assembled a 6-person team of subject-matter experts, made up of experienced chiropractors and chiropractic educators active with CCEB, and representative of Canada’s various regions and contexts of practice. The team worked under the leadership of a consultant in outcome-based learning.

To enhance the validity and reliability of the examination, the team set a goal to craft indicator statements that were:

• Outcome-focused (that is, externally observable within either MCQ- or OSCE-style testing).

• Explicit, but broadly stated (thereby enabling construction of numerous test items from each indicator).

• Appropriate at entry to practice (that is, representing situations that are commonly encountered in entry-level practice).

Key resource documents used by the team in drafting indicator statements were: the FCC competency profile; CCEB’s listing of Examination Content published in 2016;⁹  the Optometry Examining Board of Canada’s 2015 competency-indicator listing¹⁰  (this being a recently developed examination document relying on a similar conceptual framework to our own and providing examples of indicators for MCQ and OSCE testing relevant to generic professional healthcare competencies).

Miller’s Pyramid of Clinical Competence¹¹  was utilized to help guide the selection of action verbs within indicator statements. Miller’s Pyramid is a commonly used framework relating assessment methodologies to workplace-based performance expectations. It classifies MCQ testing as involving assessment of “knows” and/or “knows how” performance, whereas OCSE testing involves assessment of “shows how” performance.

Indicator development took place primarily through four 2-day, in-person meetings over the period May–October 2019, supported by background work of individual team members coordinated through email communications. Following high-level scrutiny by the team, to ensure comprehensiveness, balance, and consistency of language, the initial listing of draft indicators was 111 in number. Subsequent review of the draft indicators for reliability by independent subject-matter-expert panels focusing on blueprinting and item development reduced this number to 109. Table 2 shows the distribution of the indicators across FCC’s competency profile.

Each indicator was identified as being applicable to MCQ testing, to OSCE-style testing, or to both. Table 3 provides, by way of example, indicators for the Collaborator role.

To assist with the selection and prioritization of indicators for inclusion in the examination blueprint, a survey was undertaken to obtain relevant, current, Canada-wide practice-based information. Prior to the work we report here, CCEB possessed information about Canadian chiropractic practice that had been derived from job analysis surveys. While valuable for the purposes of examination item weighting, the job analysis data were not grounded in entry-to-practice competencies; in fact, FCC’s publication of its competency profile in 2018 was the first attempt to express the Canadian entry-to-practice standard in competency-based terms. Furthermore, FCC’s enabling competencies provide the organizational framework for our indicators. The primary data sought in our survey were practitioner ratings of the importance of each enabling competency relative to the provision of safe, effective, and ethical entry-level practice. Independently, we sought practitioner ratings of the frequency of use of each enabling competency.

Additional survey questions included each respondent’s years of practice experience and their patient age profile, responses being used to supplement practice characteristics already in CCEB’s possession.

In an attempt to maximize the survey response rate nationally, and thereby reduce the possibility of sampling bias, steps were taken as follows:

• Requests to respond, and reminders, were issued by every provincial regulator, each to their entire registrant body.

• The context of the survey and its importance to the profession were made clear in advance.

• The survey was kept as brief as possible and limited to simple rating questions (no open questions were included).

• Incentives, in the form of prize draws, were offered, with eligibility limited to those who completed the entire survey.

The survey was administered online using SurveyMonkey (Momentive Inc) and made available in both English and French languages, over a period of 6 weeks in late 2019.

A total of 1513 valid survey responses were received from an approximate population of 9520 Canadian chiropractors. The overall response rate (15.9%) is typical for surveys of this nature. The approximate margin of error on survey response data is better than ± 3% with 90% confidence. An extract from the survey instrument is provided in the supplementary file that accompanies this paper.

After cleanup of raw response data, analysis was undertaken using SPSS (IBM Corp). Some examples of analytical data are provided in Table 4, focusing on respondent language of practice, province of practice, years in practice, and importance and frequency-of-use ratings for selected enabling competencies.

Survey data for responses by province of practice, and language, were in line with expectations for a Canada-wide survey of chiropractors, suggesting that the survey sample is representative. Years-in-practice data is noteworthy with 25% of respondents reflecting 1 to 5 years; given our intention that the examination should mirror the needs of entry-level practice, it is appropriate that the survey sample included strong representation from this cohort.

Survey response data for importance and frequency-of-use of each enabling competency was analyzed based on respondent ratings (see examples in Table 4). The resulting information was used to determine indicator weightings within the new examination blueprint (see section below “Creation of New Examination Blueprint”).

Among high-stakes licensure examinations in the health professions, the most common format is a single written component, with a multiple-station OSCE administered separately (usually on a second day).

After considerable discussion, CCEB made the decision that the new examination is to involve a single written component, delivered in two 3-hour segments over 1 day. The morning session consists of single-situation MCQ items focused on patient-centered content, with the afternoon session comprising both single-situation and case-based scenarios with multiple questions. The case-based format includes a “case data” section providing information about a patient’s presenting problem, history, relevant test results, and other appropriate information. Three multiple-choice questions are based on the case data, to provide candidates an opportunity to demonstrate clinical decision-making skills in an integrated fashion.

Candidates are required to be successful on the written examination component before moving on to challenge the OSCE component. The number of scored OSCE stations is increased from 10 (previously) to 12 to better accommodate the broader landscape of assessment associated with the FCC roles.

In updating the CCEB examination to reflect a competency-based approach, it was determined that examination content at the item level must relate directly to FCC’s key competencies and enabling competencies. While items will continue to rely in part on factual knowledge, greater emphasis will be placed on a candidate’s ability to apply such knowledge in the context of everyday professional practice. Achieving this goal and determining the weighting that each FCC role should reflect within each examination component was tasked to an examination blueprint committee comprised of 10 actively practicing chiropractors representing all regions of Canada. Committee member years in practice ranged from 3 to 30, and graduates of both Canadian and international education institutions were represented; all members had prior experience with development or administration of the CCEB examination. The committee worked under the leadership of an experienced psychometrician.

The approach of Kane et al¹²  was used to combine the frequency and importance ratings provided by survey respondents for each enabling competency. Kane’s model is particularly relevant in the context of licensure examinations when the aim of the assessment is to support public safety. Simple additive models that treat frequency and importance equally do not suffice. Kane’s model uses a multiplicative approach, which acknowledges that activities that are important, even if they occur with low frequency, still may have a significant impact on public protection and should be so recognized.

For each enabling competency, the frequency and importance ratings were combined using Kane’s algorithm to create an index reflecting the contribution the competency makes to safe, effective, and ethical practice. A review group of practicing chiropractors was assembled to examine the frequency-importance indices for each enabling competency, together with the corresponding indicators. The group considered primarily:

• The “testability” of the competency (the ability to compose valid and reliable test items from the relevant indicators).

• The appropriateness of the competency for MCQ- or OSCE-based testing rather than workplace-based testing in internship.

As a result, 8 enabling competencies were eliminated from inclusion in the blueprint.

Survey response data and frequency-importance indices for the remaining enabling competencies were then referred back to the blueprint committee. The committee worked through a process of consensus to determine final weightings (expressed as percent of content) to be assigned to each practice role in the composition of the examination components (Tables 5 and 6).

To determine the relevance of pre-existing examination items to the 2018 FCC competency profile, items were mapped to the profile’s assessable enabling competencies as determined by a remapping committee comprising 14 chiropractors. Committee members had prior experience with various aspects of the item development process and therefore were familiar with examination content. Each MCQ item was reviewed and mapped as appropriate. The majority of items were successfully mapped to the FCC enabling competencies; others were deleted since they did not align with the new framework. This exercise revealed some item gaps, primarily in the Health Advocate and Leader roles. Gaps were filled by item-writing teams. A similar process was used with OSCE items, with some additional items being developed.

CCEB undertook a member survey in 2020 to identify regulatory barriers (if any) to modifications of examination policy, content, or format. The results were compiled and shared with stakeholders to confirm a comprehensive understanding and a path ahead. More recently CCEB implemented several additional initiatives to inform stakeholders about the new examination. CCEB’s website is a central repository for such information. Work includes:

• Communication with all training programs (including those outside of Canada) from which CCEB has received examination candidates in the past

• Direct presentations (upon invitation) to potential candidate audiences

• Publication of a new Examination Candidate Handbook

• Development and publication of sample test items

CCEB administered the MCQ component of the new examination for the first time in February 2024, followed shortly thereafter by the OSCE component. The new examination reflects an important transition to an assessment process that better focuses on the competencies needed to practice entry-level chiropractic in Canada.

The authors and CCEB acknowledge the vital contributions to the reported work that were made by chiropractors and chiropractic educators from across Canada who participated on committees, in working groups, and in responding to surveys.