The objective structured clinical examination (OSCE) is a commonly used assessment of clinical skill. Ensuring the quality and reliability of OSCEs is a complex and ongoing process. This paper discusses scoring schemas and reviews checklists and global rating scales (GRS) for marking. Also detailed are post-examination quality assurance metrics tailored to smaller cohorts, with an illustrative dataset.
A deidentified OSCE dataset, from stations with a checklist and GRS, of 24 examinees from a 2021 cohort was assessed using the following metrics: Cut scores or pass rates, number of failures, R2, intergrade discrimination, and between-group variation. The results were used to inform a set of implementable recommendations to improve future OSCEs.
For most stations, the cut score calculated was higher than the traditional pass of 50% (58.9.8%–68.4%). The number of failures was low for traditional pass rates and cut scores (0.00–16.7%), except lab analysis where number of failures was 50.0%. R2 values ranged from 0.67–0.97, but the proportion of total variance was high (67.3–95.9). These data suggest there were potential missed teaching concepts, that station marking was open to examiner interpretation, and there were inconsistencies in examiner marking. Recommendations included increasing examiner training, using GRSs specific to each station, and reviewing all future OSCEs with the metrics described to guide refinements.
The analysis used revealed several potential issues with the OSCE assessment. These findings informed recommendations to improve the quality of our future examinations.
Alice Cade (corresponding author) is a senior lecturer and research fellow in the Department of Basic Science at the New Zealand College of Chiropractic (6 Harrison Road, Mt Wellington, Auckland, New Zealand; firstname.lastname@example.org).
Nimrod Mueller is the co-head of the Clinic Unit at Barcelona College of Chiropractic (Carrer dels Caponata, 13, 08034 Barcelona, Spain; email@example.com).
This paper is the first prize winner of the Chiropractic Educators Research Forum/World Federation of Chiropractic Alan Adams Education Research Award award-winning paper presented at the World Federation of Chiropractic/Association of Chiropractic Colleges Global Education Conference, November 2-5, 2022. The award is funded in part by sponsorships from NCMIC, ChiroHealth USA, Activator Methods, Clinical Compass, World Federation of Chiropractic, and Brighthall. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement by, these sponsors.