Effects of a sudden change in curriculum delivery mode in postgraduate clinical studies, following the COVID-19 pandemic Open Access

The COVID-19 pandemic and quarantine provisions presented significant challenges to educational institutions. Education at all levels was affected due to the mass closure of universities. Clinically based education, particularly at the postgraduate level, was no exception. Indeed, in March 2020, Macquarie University halted face-to-face learning for most of its students due to social-distancing measures set by the government of New South Wales. There was a cessation of traditional classroom-based teaching. The Department of Chiropractic conveners had 2 weeks to adjust all teaching to an online form of delivery. This sudden adjustment presented significant challenges to the delivery mode of clinical neurology, a 1st-year postgraduate master of chiropractic course. In effect, it had to be modified as a distance-learning course.

Distance learning in medical education may be a suitable alternative to traditional learning in the delivery of high-quality education. Distance learning is defined as using computer technology to deliver training, including technology-supported learning online, offline, or both.¹  It aims to provide the effective construction of knowledge based on the individual experience, practice, and knowledge of the learner.²  Distance learning has been shown to be an efficient modality for the acquisition of theoretical knowledge in various studies on medical student performance.^3–6 

However, remote learning can also present challenges to practical-based learning and the acquisition of clinical skills.^7,8  Clinical neurology requires the acquisition of skills in neuroexamination and the development of professional competence. Before COVID-19, clinical neurology was taught only face-to-face at Macquarie University. The subject provided supportive, on-campus learning environments with clinical expert supervision and direct feedback on clinical skills and student-based interactive work. The subject has relied heavily on the acquisition of clinical skills and is underpinned by social constructivism. This educational theory aims to integrate behavior modeling with cognitive learning, which results in a strengthened understanding of the performance or task.⁹  Therefore, educators working within this model are responsible for providing supportive learning environments in which supervision and direct feedback are used to modify student performance in the desired direction.^9–11 

The transition to totally remote learning presented challenges in maintaining this model of teaching, especially in the delivery mode of practical classes, and may have led to significant consequences in student performance. Therefore, the primary objective of this study was to examine the students' assessment marks and compare them to that of their 2019 counterparts who had had traditional face-to-face teaching. The objective structured clinical examination (OSCE) assessment was specifically analyzed, as it best represents an invigilated appraisal of the acquisition of neuroexamination skills. The implications for the curriculum (modification of learning assessments, additional learning support/resources) and teaching methodology (modification of delivery, teaching quality) were also considered. Using testable hypotheses, it was predicted that the change to an online-based mode of delivery and assessment in session 1 influenced the 2020 group performance, particularly when on-campus assessment resumed in session 2.

This retrospective study with direct observation was conducted using raw assessment mark data of students enrolled in clinical neurology, a postgraduate subject within the master of chiropractic course at Macquarie University.¹²  Reporting bias was reduced using the STROBE (STrenthening the Reporting of OBservational studies in Epidemiology) cross-sectional checklist.¹³  Approval was obtained from the Faculty Ethics Subcommittee of Human Sciences at Macquarie University in Sydney, Australia (reference No. 52021948724742) on February 26, 2021. The study sample included 2019 and 2020 1st-year master of chiropractic students within the Department of Chiropractic at Macquarie University. To be eligible for participation, students had to have been enrolled in the clinical neurology course (CHIR873/874 in 2019 and CHIR8501/8502 in 2020) and had to have completed the subject in their respective academic year. No other eligibility criteria were made. Participants did not receive a reward, monetarily or otherwise, for their participation in this study. No personal data, such as names or contact details were obtained, thus ensuring the anonymity of participants.

Clinical neurology is a subject that introduces postgraduate students to common neurological conditions and to the skill of neuroexamination. A variety of teaching methods are used, including evidence-based lecture material that is also made available online and weekly tutorials. Diagnostic skills are developed using case studies.^14,15  Students develop competency in neurological examination and differential diagnosis based on the patient's signs and symptoms at clinical presentation. The knowledge and skills acquired during this subject are fundamental for diagnostic competence in chiropractic practice.

Students are evaluated for their ability to assess the history of the patient, their capacity for deciding on relevant neurological testing, and on their ability to construct a differential diagnosis. This is done using written case study examinations and the OSCE, an invigilated assessment designed to test students' clinical performance and competence in areas such as communication, clinical examination, clinical procedure prescription, and the interpretation of results. Ongoing noninvigilated assessment of skills in the tutorials provides feedback that builds students' decision making based on evidence-based practice for real-world clinical situations.^9,16 

Thus, it was that the closure of Macquarie University in session 1 2020 due to COVID-19 meant that this subject had to be restructured in such a way as to provide the same level of teaching required to build a competent clinician but within a virtual environment with virtual assessment of students (Table 1). Technology-based learning of the subject was constructively aligned to the established learning outcomes, and assessments were based on the learning outcomes. All lectures were recorded online (ECHO 360 Technology, Reston, VA, USA) and uploaded to a learning management system (iLearn Inc, Marietta, GA, USA). Weekly, real-time, drop-in clinics (Zoom Video Communications, Inc, San Jose, CA, USA) were established for interactive inquiry and discussion of subject content. The weekly tutorials were converted to an online format. These virtual classrooms were delivered using custom-made videos and real-time demonstrations by tutors, and they included guided, case-based inquiry learning in small breakout groups, followed by discussion. To further facilitate remote learning, procedures of tests from the neuroexamination were demonstrated and recorded by the tutors and made available on iLearn as downloadable video files. Tutorial manual-based, self-directed weekly readings were also prescribed to students. The usual face-to-face OSCE was substituted with students making a recording of the neurological examination tests in their homes and uploading these recordings to Turnitin (iParadigms LLC, Oakland, CA, USA) on the iLearn system. Students were given a 3-day window to upload their video files. These noninvigilated recordings tested the students' ability to perform the neurological tests and to communicate their understanding of why these tests are done and what a positive result may indicate. In 2020 session 2, with the cessation of COVID-19 restrictions, students and staff returned to usual campus tutorials. Consequently, these OSCEs were delivered in the same format as in the 2019 group.

The 2019 OSCE assessments, and the session 2 2020 assessments, were completely invigilated examinations. In the face-to-face scenario, there were 4 stations in the examination room, each with an examiner who was a clinically trained tutor. Students were allocated a time slot, and only 4 students were taken into the room at any 1 time. Students visited 2 stations, spending 4 minutes at each station. Each station had a final-year chiropractic student acting as the patient. Students were given immediate feedback at the completion of the OSCE. The OSCE again required students either to perform a neurological examination or to work through a case study, assess the history, decide on the appropriate neuroexamination that they then performed, and provide a differential diagnosis. The face-to-face format allowed for a more interactive assessment of the student knowledge and performance compared with a noninvigilated prerecording that was the limiting option of session 1 2020.

The data set comprised all assessment marks of each student in the 2019 and 2020 clinical neurology cohorts. The data were collected at the end of the 2019 and 2020 academic year and stored in an Excel spreadsheet (Microsoft Corp, Redmond, WA, USA) as per Macquarie University guidelines regarding grade collection.

Data analyses commenced in December 2020 and concluded in January 2021. Descriptive analyses were carried out using standard approaches, with study sample variables summarized using frequencies (percentages) for count variables. Continuous variables were summarized to the mean and standard deviation. Between-group differences were carried out using descriptive analyses including means and standard deviation. To examine sample rank means, a Welch t test and a Wilcoxon signed-rank test were specifically used. A probability level of p < .05 was considered statistically significant. Where appropriate, a 95% confidence interval (CI) was applied. For continuous data where residuals fit a normal distribution, linear regression was applied. Where the outcome measure's residuals were not a normal fit, a generalized linear model specifying gamma error distributions to account for skewed data was applied. All analyses were carried out using RStudio v. 4.1.0 (RStudio, PBC, Vienna, Austria) and associated updates.

A total of 212 students were included in this study, of whom 125 were men (58.97%) and 87 were women (41.03%). The 2019 group included 90 (42.45%) students, and the 2020 group included 122 (57.55%) students. The average age was 25 (±3.30, 95% CI = ±0.45) years (Table 2).

The observed means in the 2020 session 1 OSCE 1 (M = 37.51, SD = 3.58) and session 1 OSCE 2 (M = 18.22, SD = 2.10) assessment marks were higher than the means of their 2019 counterparts (Fig. 1). These differences were statistically significant for both OSCE 1, t(166.81) = −50.92, p < .001; W = 48, p < .001, and OSCE 2, t(115.94) = 5.161, p < .001; W = 4209, p < .001 (Table 2). Conversely, the means in the 2019 session 2 OSCE 1 (M = 16.45 ± 2.36) and session 2 OSCE 2 (M = 25.61 ± 3.17) assessment marks were higher than the means of their 2020 counterparts. These differences were also statistically significant for both OSCE 1, t(95.81) −2.84, p = .005; W = 1846.5, p < .001, and OSCE 2, t(75.78) 19.02, p < .001; W = 5732.5, p < .001.

Generalized linear modeling was calculated to predict OSCE assessment marks for the 2019 and 2020 groups. Results of linear modeling indicated that there was a collective significant effect between all OSCE assessments for the 2019 and 2020 groups (Table 3). General trends indicated that the cohort group was a significant predictor of higher marks seen in the 2020 group in session 1 and the 2019 group in session 2. The assessment marks, given year group, were approximately normal, showing a positive linear association between the variables (Fig. 2).

This study aimed to examine the effect of a change in the mode of delivery on assessment marks in the 2020 clinical neurology student cohort at Macquarie University following the COVID-19 pandemic, compared with the cohort of the previous year, 2019. The mode of delivery in 2019 was traditional, classroom-based learning, while remote, online learning was rapidly adopted in session 1 of 2020. The 2020 group then returned to on-campus learning in session 2.

The current study shows that the average means of the 2020 session 1 OSCE marks were significantly higher than that of their 2019 counterparts. However, in session 2, this trend was reversed, and the 2019 group achieved significantly higher marks than the 2020 group. An important distinction between the 2 cohorts was the lack of opportunity for the 2020 students to put into practice the clinical skills being taught under the guidance and supervision of experienced clinicians during session 1. The remote learning format made it impractical to provide students with sufficient and individualized guidance and feedback on clinical skills acquisition and on developing competency in the neuroexamination. This was reflected in the marks for the 2020 session 2 OSCE assessment when students returned to on-campus invigilation. The students were underprepared and lacked the hands-on clinical experience required for the session 2 OSCE when compared with their 2019 counterparts in session 2.

Educators are responsible for providing a supportive learning environment and use feedback (reinforcement) to modify behavior in the desired direction.¹⁰  In addition, the peer support and interaction that occurs in face-to-face teaching also arguably contributes to behavior. The on-campus tutorial environment provides continuous repetition and skill exercises that over time lead to improved competency.¹⁷  Educators play an important role in shaping effective education and should invest time in implementing timely feedback strategies.^11,18  In fact, a recent meta-analysis of 435 studies found that feedback had positive effects on student learning and a higher impact on cognitive and motor skill outcomes than on motivational and behavioral outcomes.¹⁹  Students construct new knowledge through the interaction between their previous skills and knowledge, the skills and knowledge gained from social interaction with peers and teachers, and social activities.^16,20  Knowledge is actively constructed based on a learner's environment: the physical and social world.¹⁶  Much of clinical education occurs within a workplace context, such as supervised clinical internships in private practice or hospitals; therefore, sociological theories of learning (eg, social constructivism) are well suited to this setting.^21,22  A systematic review of 38 meta-analyses investigated 105 correlates of achievement from 2 million students and found a close relationship between social interaction in courses and achievement.¹¹ 

Unpreparedness for clinical practice in recent graduates remains a problem despite changes in curricula from clinical internships to outcome-based designs. This is unsurprising because capability depends on learning from practical experience in supportive learning environments. Indeed, the student's exposure to a clinical learning environment is one of the most important factors affecting the teaching-learning process in clinical settings.²³  Facilitator-supported participation in practice results in real clinician-patient learning and equips clinical students with the identity and capabilities of safe, effective, and compassionate clinicians, who continue to be lifelong learners.^24,25  Furthermore, bedside teaching is rationally necessary for learning clinical skills, such as history taking, examination skills, knowledge of clinical ethics, and professionalism. Evidence suggests that bedside teaching can improve certain clinical diagnostic skills in medical students and residents²⁶  and aid competency-based education models that cannot be replaced by simulation-based education.²⁷ 

However, remote learning has been shown to benefit assessment performance. Gonzalez et al⁴  found that remote education changed students' learning strategies to a more continuous habit, improving their efficiency. The authors concluded that better scores in students' assessments are expected due to remote learning and could be explained by an improvement in their learning performance.⁴  Indeed, students who participate in online video formats have statistically better practical examination scores and final grades when compared with students who undertake traditional, face-to-face learning only.²⁸  This could account for the higher marks seen in the video OSCE assessments of the session 1 2020 group when compared with their 2019 counterparts. Students chose which test they were most comfortable in demonstrating and were able to practice the test multiple times before submitting their video file. This delivery mode is different from an invigilated, timed assessment with a real patient and examiner. Furthermore, the nature of the video OSCE format allowed for comprehensive feedback on performance in the neuroexamination, which is not possible in the invigilated format due to time constraints. This feedback is meaningful in that it can help students understand their strengths and weaknesses.²⁹  Self-assessment has been shown to help students develop metacognitive skills and improve achievement in a range of different academic areas.³⁰  However, online learning holds certain limitations for older students, those who live in rural areas, individuals with work and family responsibilities, and those with limited electronic resources.³¹ 

While online education can provide opportunities for self-study, the main challenges faced by online education in the clinical sciences are how to provide immediate guidance to a student who is conducting a clinical skill and how to model professional behavior. This is the essential social “bundle of goods” that must be passed to students who are to become clinicians. Learning clinician-patient interaction and professional behavior may be successfully modeled under the guidance of experienced tutors with a live patient.^32,33  A completely remote learning mode of delivery may not provide students the necessary clinical skills acquisition required to become competent clinicians. Several recent studies have explored the effects of the COVID-19 pandemic on medical students.^34–37  However, many of these qualitative studies have explored only student feedback, perceptions, and experiences of remote learning.³⁸ 

This study provides a novel insight into the effects of remote learning on assessment marks in clinical students. The results obtained in this study demonstrate that a change in the mode of delivery from face-to-face to remote learning and the change in testing methods in response to the global COVID-19 pandemic had a significant effect on clinical neurology student assessment performance scores. It is speculative but conceivable that this is due to the lack of a teacher-student interaction described here. However, it cannot be denied that further analysis of online delivery is needed to find other and better ways to achieve this kind of knowledge and skill acquisition. It is not the intention of this article to deny the benefits of online delivery but rather point to a blended form of education as a possible solution.

The small sample size of convenience-sampled postgraduate students is a study limitation. Furthermore, the study was performed at only 1 location and examined only 1 subject, and thus these findings may not necessarily be replicable in other subjects or countries. Several factors may have influenced the findings of this study, such as the students' dispositional and situational factors, psychosocial factors, and technological capability and accessibility, to name a few. These factors may have potentially affected the results of this study and were not accounted for in the statistical analyses. Another plausible explanation for the study findings is that the difference in scores may not have been due to the teaching but rather to other factors, such as the way the students were tested. The format of the examination delivery was different for each group and might have been primarily responsible for the difference in scores and not due to teaching quality or delivery. Generalized linear modeling requires large data sets and is sensitive to outliers, which were evident in the analyses. To account for the positive-skewed data, gamma distribution was applied. The study was directly observational of descriptive data, which identified a statistically significant between-group difference in OSCE assessment marks for 2019 and 2020 cohorts. However, causal links cannot be presumed. Given these limitations, the results add to the existing literature with the potential to inform tertiary educational policies and future studies. Future research should focus more holistically on blended learning and identify specific obstacles in web-based clinical education by quantitatively assessing clinical assessments.

The sudden change in the mode of delivery from face-to-face to remote learning and the change in testing methods in response to the global COVID-19 pandemic had a significant effect on the clinical neurology student assessment performance scores. The OSCE marks in the 2020 cohort for session 1 were significantly higher than those in 2019. However, when the 2020 cohort returned to on-campus tutorials in session 2, their marks were significantly lower than that of their 2019 counterparts. The web-based mode of delivery in 2020 may have had a significant impact on student OSCE marks when compared with the 2019 cohort. The level of competency in the OSCE neuroexamination, attention to detail, and professionalism around the patient required revision in session 2 2020 when on-campus tutorials resumed.

We wish to sincerely thank Dr Robert Borotkanics, Dr Reidar P. Lystad, and Mr Jack Sahagian, who contributed with smaller amounts of work, mostly regarding statistical analysis support and general critique.