Online vs in-person delivery of preclinical coursework: A retrospective cohort study evaluating National Board of Chiropractic Examiners performance

The COVID-19 pandemic caused disruptions in nearly every aspect of society, with academia being no exception. Government mandates limiting in-person gatherings resulted in the rapid transition to online learning, forcing faculty and students to adapt to new teaching and learning styles.¹  Students reported distractions at home along with creating a separation of study space from living space as common challenges during this transition.²  External factors brought on by the pandemic (e.g., campus closures and isolation) created motivational challenges and negatively impacted the psychological health of students.^2,3  According to Williams et al, “… emergency remote learning and attitudes about the emergency learning experience created a source of cognitive overload and stress to manage, among other distractions associated with the COVID-19 pandemic.”²  Faculty also faced unique challenges, including “identity dissonance,” due to the necessary development of new pedagogical strategies.⁴  Inadequate time to carefully design courses may have prevented faculty from taking full advantage of the online format.⁵  Some faculty lacked prior online teaching experience and encountered a steeper learning curve in adapting to new methods.⁵  Students identified spending additional quality time with family and friends along with a chance to focus on physical and mental health as positive outcomes during this challenging time.³  Faculty indicated that teaching online was an overall positive experience, as they were able to develop new skills and interact with students in a different academic setting, which some believed was more effective than the classroom.⁴ 

Mixed evidence exists supporting the effectiveness of online vs classroom instruction in higher education settings.⁶  Relevant to our study population, a 2019 systematic review⁶  evaluated post-test scores, pre- and post-test score gains and retention test scores in medical student populations utilizing online vs classroom learning methods. Of the 16 included articles, 7 reported no difference and 9 reported significant improvement in the online learning groups; none of the included studies concluded that online learning was inferior to classroom learning.

The National Board of Chiropractic Examiners (NCBE) part I is a standardized computer-based exam consisting of 50 multiple-choice questions in each of 6 domains (40 of which are used in the test analysis): General Anatomy; Spinal Anatomy; Physiology; Chemistry; Pathology; and Microbiology. Chiropractic students are eligible to sit for this examination after satisfactorily completing the courses being tested.⁷  Beginning in 2014, the NBCE began transitioning its exams to item response theory (IRT) scoring, which relates a test-taker’s ability to the probability of a correct response.^8,9  The range of scores for examinees are on a continuous scale (125–800), with 375 being required to pass; students failing 3 or more domains must retake the entire exam.^8,10,11  Himelfarb et al⁸  provide a detailed discussion of the psychometric properties of part I.

The primary objective of this study was to evaluate predictors of NBCE part I pass rates among student cohorts exposed to a forced online basic science (COVID online) curriculum in 2020 secondary to COVID-19 regulations vs those exposed to a traditional classroom (classroom) basic science curriculum 2 years prior. We hypothesized that the COVID online cohorts would have lower odds of passing part I vs the classroom cohort.

The study was reviewed and approved by the Palmer College Institutional Review Board (2022-004). Academic and demographic variables were obtained from the college registrar for students who matriculated in 2018 (classroom) or 2020 (COVID online) and had a reported part I exam score. The dataset was cleaned, de-identified, and analyzed independently by different study team members. The final dataset contained the following variables for each student (Table 1): Part I domain scores and pass/fail marker; matriculation date and campus; age; ethnicity; gender; alternative admission track plan (AATP); incoming bachelor’s degree; and incoming and in-program grade point averages (GPA). As incoming and in-program GPA exhibited slight non-normality, these variables were dichotomized into high and low categories. The Strengthening the Reporting of Observational studies in Epidemiology (STROBE) recommendations were used to report this study.

Data analyses were conducted using SAS version 9.4 (SAS Institute Inc). Descriptive statistics were calculated for each cohort, reported as counts and percentages for categorical variables, and means and standard deviations for continuous variables. Chi-square and independent samples t tests evaluated significant differences among student cohorts for each predictor. A general linear model (GLM) estimated odds ratios (OR) with 95% confidence intervals (95% CI) for the outcome of passing part I, adjusting for all variables in Table 1 (except age). To compare adjusted mean pass rates among online cohorts, a least square means (SAS proc lsmeans) procedure was included in the GLM.

There were 968 students who matriculated in 2018 or 2020, 55% of which constituted the classroom cohort (Table 2). The Spring 2020 cohort had the lowest pass rate (60%), the fewest incoming students with a bachelor’s of science degree (59%) and more students classified as having a high in-program GPA (61%) vs other cohorts (p < .05 for all comparisons). The Fall 2020 cohort had the highest pass rate (86%), the lowest percentage of students attending the main campus (40%), and the lowest mean age (25.1) vs other cohorts (p < .05 for all comparisons). The classroom cohort had fewer students classified as having high in-program GPA group (46%) and the highest mean age (28.1) vs other cohorts (p < .05 for both).

Output from the GLM model is presented in Table 3. The Spring 2020 cohort had significantly lower odds of passing part I [0.80, (95% CI: 0.73–0.87)] vs the classroom cohort, while the Fall 2020 cohort had higher odds [1.06 (95% CI: 1.00–1.13)]. Both branch campuses had nearly identical OR estimates, indicating a 13%–14% decreased odds of passing vs the main campus. Other ethnicity (vs White) and alternative admissions (vs none) were borderline significant (7% and 9% decreased odds, respectively), as were incoming bachelor’s degrees and high (vs low) incoming high GPA (8% and 7% increased odds, respectively) for passing part I. High (vs low) in-program GPA was the greatest predictor of pass rate, with an estimated OR of 1.36 (95% CI: 1.29–1.43). Although age was significantly different among cohorts in the univariate analysis (Table 2), the estimated OR in the GLM was 0.99 so it was excluded. Model fit statistics indicate the GLM (R² = 0.25, F-statistic = 29, p < .001) was a significant improvement over a bivariate model including only student cohort (not shown: R² = 0.02, F-statistic = 8.1, p < .001) in predicting the outcome.

The adjusted mean differences in pass rates among the online cohorts are presented in Figure 1. The Spring 2020 cohort had a significantly lower mean pass rate compared to summer 2020 (−24%) and fall 2020 (−29%) cohorts. The Summer 2020 cohort had a (nonsignificant) 4% lower mean pass rate vs the Fall 2020 cohort.

The purpose of the current study was to understand the implications of forced online learning during the COVID-19 pandemic on a key programmatic outcome: NBCE part I pass rates. To our knowledge, this is the first study to compare board exam pass rates among cohorts of students participating in 2 different delivery methods of a basic science curriculum.

Our hypothesis that the COVID online (vs classroom) cohorts would have lower odds of passing part I was confirmed only for the spring 2020 cohort. The summer 2020 cohort had no significant association with passing part I, while the fall 2020 cohort had slightly higher odds of passing. While there is a lack of directly comparable literature, we can speculate on factors that may have contributed to these findings. The rapid and forced transition to online learning experienced by the spring 2020 cohort, along with significant social and academic disruption likely impacted performance on part I board exams. The summer and fall 2020 cohorts potentially benefited from lessons learned and adaptions made after the initial disruption occurring in the spring.

Zhang et al¹²  evaluated objective structured clinical exam (OSCE) performance in a physical exam course at a U.S. chiropractic institution among students choosing between classroom and online lab sessions. Correlation coefficients indicated that OSCE scores were positively correlated with in-person class attendance (r = .62, p < .001), and the majority of students (84%) preferred in-person lab. Another study¹³  evaluated OSCE performance in a clinical neurology course at an Australian chiropractic institution in 2019 vs 2020. The comparison group (2019) experienced traditional classroom instruction, while the 2020 cohort had online instruction for session 1 and classroom instruction for session 2. The mean scores for the online cohort were significantly higher than the classroom cohort for session 1, while this trend reversed in session 2. The authors hypothesized that remote learning prevented individual guidance and feedback, resulting in poor performance when returning to a classroom format in session 2 of 2020.

Our results indicated that alternative admissions status (AATP) was associated with a 7% decreased odds of passing part I. Two of the authors found part I fail rates were significantly higher among chiropractic students with AATP status (37%) vs without (24%); however, the correlation among these 2 variables was weak (Φ = 0.11).¹⁴  Several studies confirm our finding that in-program GPA was the strongest predictor of part I pass rate. Himelfarb et al¹⁵  found that increasing in-program GPA by 1 unit (e.g., C to B) had a corresponding improvement in part I domain scores, including anatomy (+ 28 points), physiology (+ 30 points), chemistry (+ 25 points), pathology (+ 16 points), and microbiology (+ 18 points). Cunningham et al¹⁰  evaluated correlates to performance on part I, and found that adding in-program GPA to a step-wise regression model resulted in the largest improvement in model fit measures.

Noteworthy is the declining pass rate trend for part 1 NBCE from 2019 (72%) to 2022 (68%) during the COVID-19 pandemic period.¹⁶  Other healthcare professions experienced nearly identical trends during this time period (Figure 2): Nursing (NCLEX or National Council Licensure Examination for Registered Nurses) pass rate dropped from 88% to 82%;¹⁷  Medicine/Osteopathy (USMLE or United States Medical Licensing Exam) step 1 pass rate dropped from 96% to 91%;¹⁸  Pharmacy (NAPLEX or North American Pharmacist Licensure Exam) pass rate dropped from 82% to 78% (2020–2022);¹⁹  and Physical Therapy pass rate dropped from 91% to 85%.²⁰  This trend is undoubtedly related to the abrupt changes in curricular delivery method that occurred secondary to COVID-19 mandates.

There are several limitations that deserve mentioning. Comparing individual online cohorts to a combination of classroom cohorts could be considered a limitation. Modeling different comparison groups resulted in nearly identical estimates and model fit statistics as those provided here. Second, academic calendars do not align perfectly across campuses, which we do not believe influenced the reported outcomes. Additionally, the transition to online learning was uniform across campuses. Third, our results represent students attending a single institution and may not generalize to other chiropractic institutions. Lastly, there may be unmeasured variables associated with part I performance which we did not include or have access to.

Due to COVID-19 regulations, a rapid transition to online learning was necessary for all basic science courses which were traditionally taught in a classroom setting. Our study evaluated the impact of this transition on licensing board exam performance. The results indicate that the first COVID online cohort performed poorly, while the final online cohort outperformed the classroom reference cohort. Further research is necessary to understand the underlying factors contributing to our findings. As academic administrators look to increase the number of online course offerings across chiropractic curriculums, the results of this study can inform decision-making about the effectiveness of this model of learning based on a key programmatic outcome.

The authors thank Mindy Leahy, Senior Director, Registrar, for assisting with data collection.