Introduction, development, and evaluation of the miniclinical evaluation exercise in postgraduate education of chiropractors

Although many chiropractic colleges and universities have postgraduate residency programs for specific subject areas such as diagnostic imaging, sports, clinical sciences, and other specialties, only a small proportion of graduating chiropractors choose to continue their education in these full-time programs. In order to practice as an independent chiropractor in Switzerland, all graduating chiropractors must complete a 2-year, full-time, postgraduate program (residency) that has full accreditation with the Swiss government under the same conditions as medical postgraduate residency programs.¹ 

Documenting that a graduating resident has mastered, at some predetermined level, the knowledge, skills, and attitudes associated with each of the core competencies of their specific program, while informative, does not ensure that the individual is a competent physician.²  Something more is needed: Graduating residents must be able to translate and integrate their knowledge, skills, and attitudes so they can perform the complex tasks required to deliver high-quality medical or chiropractic care.³  Determining that residents have taken this last crucial step is the responsibility of the faculty of the residency program, which must find better ways to critically observe the resident's care for patients in a variety of clinical settings and circumstances.⁴  Furthermore it has been shown that people in general are better at reproducing and applying knowledge and skills if the context in which they have to do so resembles the context in which the knowledge and skills were first learned. In this sense, the concept of authentic assessment is inseparably linked to constructivist learning theory.⁵ 

In order to improve the quality control of the clinical training during the postgraduate education (PGE) of chiropractors in Switzerland, one step was to introduce a formative method for evaluating the clinical skills of trainees throughout their residency program. The format was similar to the miniclinical evaluation exercise (mini-CEX), which is a workplace-based formative assessment method developed, piloted, and evaluated in the United States and is now widely used to assess doctors in US and European residency programs.^6–15  The mini-CEX entails direct observation by an educational supervisor of a trainee's performance in real clinical situations (15–20 minutes) and is designed to assess skills such as history taking, clinical examination, communication skills, diagnosis, and clinical management. The assessment is repeated on multiple occasions and can occur in various clinical settings.¹⁴  The mini-CEX differs in several ways from the traditional long case used previously and usually in high-stakes examinations. First, each mini-CEX session only assesses a single component of the clinical encounter, such as history taking or physical examination, and requires only 15–20 minutes. Thus, it does not require formal planning in most cases and can occur almost spontaneously. Second, the mini-CEX is repeated on numerous occasions throughout the postgraduate program. Third, each session provides immediate formative feedback to the postgraduate student in order to facilitate learning. Fourth, the mini-CEX has been shown to be reliable whereas the longer clinical encounters have suffered from a lack of reliability.¹⁰ 

Effective formative assessment is typically low stakes, often informal and opportunistic in nature, and is intended to stimulate learning. By definition, the criterion that stands out to characterize it is “catalytic effect.” It works best when it (1) is embedded in the routine instructional process and/or work flow, (2) provides specific and actionable feedback, (3) is ongoing, and (4) is timely. The method has been shown to be reliable, to have construct validity, and to be a good method of education as well as an assessment tool.⁷  Based on these characteristics and because the need for regularly implementing a formative method of real-life clinical assessment became inevitable, the mini-CEX was selected as the method of choice to study in order to determine its usefulness in the Swiss chiropractic postgraduate educational program.

Although there is quite a lot of literature on the use of the mini-CEX for PGE in medicine in the United States, very little literature is found for postgraduate medical education in Europe, and none so far for PGE in chiropractic, either in the United States or in Europe.^6–13  Whether or not the mini-CEX method used in medical education can be directly adopted and applied to chiropractic education is also unknown. The setting for trainees differs between medicine and chiropractic such that the medical trainee spends much more time in a hospital setting, whereas the chiropractic trainee spends most of his/her time in a private practice office. Thus the important question is whether or not the mini-CEX is a feasible assessment method for either setting. Because there are no studies evaluating the feasibility of using the mini-CEX in the postgraduate chiropractic setting, the purposes of this study are as follows: (1) to determine if the mini-CEX format is applicable and useful for evaluating clinical competency in postgraduate chiropractic residency programs as formative feedback, (2) to determine the reliability of competency ratings between different assessors for the same tasks, (3) to compare which of two different rating scales used to assess competency is more reliable in the postgraduate chiropractic educational setting, (4) to assess the relationships between the various clinical competencies.

This is a descriptive study exploring the introduction and adaptation of the mini-CEX into a chiropractic postgraduate residency program. Two different chiropractic practices in the German-speaking part of Switzerland participated.

All of the postgraduate residents (n = 23) enrolled in the program during the years 2009 and 2010 participated. Evaluators (official assessors and observers) were chiropractors approved by the Swiss Academy for Chiropractic as postgraduate supervisors and lecturers in the postgraduate program in Switzerland.¹  All evaluators had participated in a formal training session on the mini-CEX conducted by the first author of this study.

Each postgraduate resident participated in 3 different mini-CEX assessment sessions over a period of 12 months. Each of these 3 sessions contained 4 separate mini-CEX clinical encounters for a total of 12 clinical encounters over the 3 sessions. Actual patients were evaluated in 3 clinical encounters per session, and 1 of the clinical encounters per session was an oral diagnostic-imaging interpretation station. For each of the 3 mini-CEX sessions, there were the following stations: a station requiring the taking of a focused case history followed by clinical reasoning questions, a station requiring the performance of a physical examination followed by clinical reasoning questions, a station observing patient case management after history taking, and a radiology station consisting of 5 different imaging cases. For each station in which a patient was involved, various competencies were evaluated. Depending upon the clinical tasks required at a particular station, these competencies included medical interviewing (history taking), physical examination, organization/efficiency, clinical judgment, professionalism/communication, counseling, and overall clinical performance (Fig. 1).

For each mini-CEX encounter involving a patient, 2 evaluators were present. One was the official assessor who asked questions, and the other was an observer. Both the assessor and observer completed the written evaluation forms for the performance of the resident, however. Thus each of the 3 mini-CEX sessions involving 4 different Mini-CEX encounters would mean that each postgraduate resident was evaluated by a total of 7 different chiropractors (2 in each of the 3 patient encounter stations and 1 in the diagnostic imaging station) for that particular session. Thirty minutes were allowed per mini-CEX encounter, and the assessors and observers used the standard mini-CEX form from the American Board of Internal Medicine (ABIM) to rate 7 domains/competencies (medical interviewing, physical examination, professional qualities/professionalism, organization/efficiency, and overall clinical performance) for each patient encounter.^8–13  The first 2 of the 3 mini-CEX sessions used a 9-point rating scale where scores of 1–3 = unsatisfactory performance, 4–6 = marginal/satisfactory performance, and 7–9 = superior performance. The 3rd mini-CEX session used only a 5-point scale based on feedback from the assessors on their use of the 9-point scale. The assessors reported that they felt there was too much ambiguity in the 9-point scale. The 5-point rating scale was as follows: 1 = unsatisfactory performance, 2 = barely sufficient, 3 = satisfactory, 4 = good, and 5 = superior performance (Fig. 1). Thus, each verbal description of competency corresponded to only a single number. The assessors also rated the difficulty of each clinical encounter as low, medium, or high complexity.

In the country where this study was performed, formal ethics approval was not required for this study as it was a routine part of the formative assessment within the postgraduate program. All patients and assessors signed informed consent, however, for use of the data.

Reliability of the assessor and observer in the evaluation of the resident was assessed in 2 ways. The categorical data (i.e., unsatisfactory, satisfactory, superior, etc.) were analyzed using the Cohen κ coefficient. The reliability of the actual numerical values was compared using the intraclass correlation coefficient (ICC). The internal consistency of the 4 stations (history, physical exam, management, and imaging interpretation) per mini-CEX session was assessed using Cronbach α. The Spearman correlation coefficient was used to assess the level of correlation between the various competencies for each of the 3 mini-CEX sessions. It was also used to assess the correlation between the various combinations of the 4 clinical encounters per mini-CEX session. The Mann-Whitney U test was used to compare the median scores for each of the competencies between the 2 assessors at each patient station.

All of the postgraduate residents enrolled in the program (n = 23; 15 were male) participated. Seven different evaluators assessed each resident in each of the 3 sessions. The same 7 evaluators participated in all 3 sessions. Of the clinical encounters using patients, 5 were chronic patients, and the other 4 were acute or subacute patients. The complexity of patient encounters as rated by the evaluators was high in 43.7%, moderate in 14.6%, and low in 41.7% of cases. When looking at all 12 of the mini-CEX encounters over the 3 different sessions, organization/efficiency, professionalism/communication, and overall clinical performance were assessed in 100% of the encounters. Medical interviewing was evaluated in 48.8% of the clinical encounters, physical examination in 25.6%, clinical judgment in 24.4%, and counseling in 23.2%.

Cohen κ interrater reliability for all categories using the 9-point evaluation scale was κ = .31 (fair) and for the 5-point scale it was 0.42 (moderate). The overall percent agreement for the 9-point scale was 43% for the categorical data and 55% for the 5-point scale. The results for the interrater reliability of the various competencies comparing numerical values with the ICC are shown in Table 1.

Considering the fact that each mini-CEX station used a different patient, internal consistency of the 4 domains (history taking, physical examination, history taking plus case management, and X-ray) for the 3 different mini-CEX events were combined, and Cronbach α was calculated. The results are presented in Table 2.

There were significant positive correlations comparing the competencies for all 3 mini-CEX sessions with high correlation coefficient values as shown in Table 3. When assessing the correlation between the 4 stations, rather than the specific competencies, statistically significant correlations were found for all domains, with the exception of history taking/management with X-ray diagnosis and history taking alone with history taking/management (Table 4).

There were no statistically significant differences in the scores between the 2 assessors at a station for any of the competencies evaluated, with p values ranging from a low of .35 to a high of .98.

To our knowledge, this is the first study to evaluate the potential benefits and usefulness of the mini-CEX in a postgraduate chiropractic program. Certainly, the verbal feedback from the postgraduate residents involved in this study was unanimously positive, stating that these sessions were the “best educational experiences” of their 2-year program and that they would like to have more opportunities to be evaluated in this way. Additionally, the assessors reported that they found the experiences informative and positive. The use of 12 different stations in 3 different mini-CEX sessions spread over a 12-month time period and assessment of all of the 23 different postgraduate residents enrolled in the program should provide valid data to evaluate the mini-CEX in the postgraduate training of chiropractors. The variety of clinical competencies assessed, the acceptable reliability results, and the ability to give instant formative feedback to the postgraduate students further supports the use of the mini-CEX in chiropractic postgraduate programs.

Two assessors were used at all patient encounter sessions in order to look for significant differences between assessors in their scores for the residents. Not only was the reliability for the 5-point marking scale moderate (κ = .42), but also there were no significant differences between the assessors in the scores for any of the competencies assessed. This is important information as it supports using only 1 assessor per station in the future, which has a significant positive impact on resource and financial costs for the mini-CEX. If enough trained assessors are available, it has been suggested to increase the number of stations rather than have more than 1 assessor per station.^16,17  However, as no previous studies, to our knowledge, had investigated the usefulness of the mini CEX in chiropractic postgraduate training, our preference was to use 2 assessors per station initially.

The issue of training assessors is important. The stringency of examiners is improved with training, which leads to less variability between their ratings of postgraduate students. Additionally, examiners in general feel more comfortable with the evaluation of clinical skills after training, which, in turn, should facilitate giving more useful feedback.^18–20  Prior to the implementation of the first mini-CEX in the postgraduate program in Switzerland, a formal training session occurred for the assessors. Additional training sessions were included as workshops in a recent European Chiropractors' Union convention held in Switzerland, which had mandatory attendance for all Swiss chiropractors. New assessors are also trained by having them attend a mini-CEX session strictly as an observer. Other postgraduate programs could consider similar training methods.

A critical component of each mini-CEX encounter is providing instant (formative) feedback to the residents so that they have a focus for improvement. Ideally, this feedback should result in the formulation of an action plan. However, the literature indicates that this formative, timely feedback rarely results in creating an action plan and only sometimes involves self-assessment by the trainee.¹⁸  Further training of assessors should help them develop the confidence to create a useful action plan with the resident in a timely manner. Certainly, this was often reported to be the most difficult part of the mini-CEX encounter by the assessors. After completion of this study, workshops were held with all of the approved principals in the postgraduate program to train them to do mini-CEXs in their own practice environment with their own resident on a regular basis. Formal follow-up to determine whether or not this is routinely happening has not yet been done, but is planned for the future. Ideally the mini-CEX is imbedded in the routine clinical work of postgraduate students and should not have to be scheduled as a separate day of assessment with multiple clinical encounters occurring on the same day as was done in this study.

One of the criticisms of this study is the change from a 9-point marking system to a 5-point marking system during the study. The interrater reliability of the actual numerical values on the 9-point marking system used in the first 2 of the 3 mini-CEX sessions tended to show higher ICC values than the-5-point system, consistent with the study by Cook and Beckman.¹⁵  However, the 5-point scale had higher κ values of .42 (moderate) compared to .31 (fair) for the 9-point scale when comparing the category of assessment. The change from the 9-point to the 5-point scale occurred after feedback from the assessors. They felt there was too much ambiguity in the 9-point scale, with 3 different numerical values linked to a single written descriptor of competency. The specific descriptors for each of the numbers on the 5-point scale are very clear and distinct, and the assessors reported feeling more confident with this grading system.

The results for the internal consistency of the 4 different stations (domains) for the mini-CEX are somewhat disappointing, with a total score of .58 using the Cronbach α test. Cronbach α should be >.7 to be acceptable, but >.8 is preferable. As each domain was assessed for a different patient, and there were fewer than 10 items for the scale (4 or 5 only), Cronbach α turned out to be inappropriate for this data. With short scales (<10 items), it is common to find quite low Cronbach α values. In this case, it may be more appropriate to report the mean interitem correlations for the items.¹⁵ 

One limitation of this study is that it involves a very small group of students, and hence, the generalizability of the findings may be questionable given the small sample of candidates. However, all postgraduate students enrolled at the time were included, with a variety of competency levels noted in each of the clinical domains assessed. Additionally, we modified the implementation of the mini-CEX encounters so that a participant completed 4 mini-CEXs during 1 day. This method of sampling may have resulted in less variability in participant performance as they did not have time to respond to and improve from feedback given in an earlier encounter. In order to assess student progression, future studies should evaluate single mini-CEX encounters, rather than 4 on a single day, with the encounters occurring frequently over the course of the program.

Previous work has also indicated that the mini-CEX is subject to assessor error, for example the halo effect, which describes the tendency to erroneously apply conclusions regarding one attribute to another. Assessors specifically need to be made aware of this potential problem during training sessions prior to participating in a mini-CEX session. The first author of this study addressed this concept in her training session with examiners prior to the first mini-CEX session. However, reminders should be done prior to subsequent sessions.

Finally, this study was conducted at one institution only, thus limiting the generalizability to postgraduate students at other institutions and in other countries. It would be useful to compare the results in other postgraduate chiropractic programs with those obtained in this study. Additionally, it could also be tried during the final clinical year in international chiropractic programs that do not have a mandatory postgraduate program.

The mini-CEX as used in a postgraduate chiropractic education program includes a wide range of skills evaluated in a brief but focused format, allows for immediate formative feedback, is low cost, and provides the opportunity for faculty to identify students with deficiencies early in the process of PGE in order to facilitate remediation. The reliability of marking the stations is acceptable, and there is a strong positive correlation between the various competencies at a station. Additionally, the mini-CEX finds approval from both postgraduate students and their assessors.

There were no external sources of funding for this study, and the authors declare that they have no conflicts of interest.