Going Beyond Grades: Online Learning Assessment: Chiropractic Educators Research Forum (CERF), June 26, 2021

The Chiropractic Educators Research Forum (CERF) holds conferences from time to time that focus on a topic relevant to education and the chiropractic profession. This conference showcases education research, innovations, and best-practices and provides a forum for the presentation of scholarly work in chiropractic education theory and practice. The CERF held a virtual conference on June 26, 2021. The focus of this meeting was on research related to education and learning assessment. After submission to a rigorous peer-review process, the following abstracts were accepted for presentation. As we have done with prior CERF conferences, the abstracts include the video presentations so that they may be more widely distributed.¹ 

The presentations from the June 26, 2021 conference are listed here in alphabetical order by first author's last name. Each abstract includes a link to the video abstract of the presentation that was delivered at the conference.

Ilija Arar

Objectives: It was reported that academic dishonesty had been increasing during the shift to remote administration of courses resulting in an inflationary effect on assessment outcomes. The aim of this study was to determine if recreating traditional assessment conditions through live, remote proctoring prevented third trimester physiopathology final exam score inflation. Methods: Remote proctoring was implemented through a secondary mobile device logged in to a video conferencing service and positioned in a way that the lead instructor could actively monitor each student as they completed their exam on a tablet. Students were provided a detailed set of instructions to establish expectations and a protocol for addressing test item enquiries through a chat feature to decrease audible interruptions to other students. Acceptance of an integrity statement was a mandatory requirement prior to commencement of exams. Real-time technology support was made available through contact information with the college's Academy for Teaching Excellence. An independent samples t test was used to compare final exam grades from the spring and fall 2020 third trimester cohorts (n=78) to students from same period during the previous year prior to the pandemic (n=85). Results: Students who were proctored remotely performed poorer than their upper classmates from the same time period the year prior (79.26±12.05 vs 83.51±9.12, p=.012). Conclusion: Third trimester physiopathology final exam score inflation was not observed. This study suggests that remote proctoring is a feasible method of maintaining academic honesty. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/xDlFtFWvHGQ

Grand Choi, David Starmer, Scott Dunham, Cirene D'Monte, Brendan Corr

Objective: The purpose of this report is to describe the creation of an interactive virtual simulation experience for a radiographic positioning course to assess students' skills. Methods: We developed a simulation using avatars, images/videos of equipment, and proper/improper positioning techniques, a virtual simulation lab experience was built using Articulate 360 software to replicate all components of the in-person assessment. A faculty member was contracted for 40 project hours and worked with a radiographic positioning content expert to develop the experiences in less than 6 weeks. The simulations were created so that students would measure body parts, look up and set exposure factors, select appropriate cassette size, align x-ray tube and cassette, position patient appropriately, and expose films in the interactive virtual simulation. We included software features to assess student's ability to work through different cases. Results: Three interactive modules were the result. They included (1) a tour of equipment and orientation to the lab, (2) demonstrations of proper techniques for different body regions, and (3) allowed students to take plain film radiographs with virtual patients. Conclusion: We created an interactive virtual simulation lab experiences to teach and assess practical skills for radiographic positioning courses. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/8tdGtphUrbE

Grand Choi, David Starmer, Scott Dunham, Vanessa Petrini, David Lee, Kim Ross, Cirene D'Monte

Objective: The purpose of this report is to describe the development and delivery of a virtual formative assessment experience for grand rounds. Methods: Clinical experiences and records of patient-clinician interactions were recorded. With those assets, interactive virtual modules were built using Articulate 360 and Microsoft Forms. Using features within the software, the interactive experiences were built to assess student's clinical reasoning and decision making. Ten project hours per clinical grand rounds were allocated to build the virtual modules to replicate the interactions that would take place in person. The interactive experiences were developed to be self-paced. The program was developed to allow students to review intake forms, test their ability to score outcome measures; and view patient histories/physicals, and report of findings. The formative assessment was developed to allow students to self-assess and compare their differential diagnoses, clinical rationale, and decision-making against those of an experienced clinician. Results: Twelve virtual clinical grand rounds modules were created, delivered, and completed by 586 students. Conclusions: The creation of interactive virtual clinical grand rounds modules was feasible. We created a self-paced, independent, immersive interaction that allowed learners to review content with opportunities for reflection and identification of personal knowledge gaps. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/XHOQzZhl7Dc

Scott Dunham, Vanessa Petrini, David Starmer

Objective: This study compares the number of failures for a virtual administration and in-person delivery of an Objective Structured Clinical Examination (OSCE). Methods: Assessment components were mapped to evaluate clinical competencies in a virtual environment. Training was provided to assessors on electronic evaluation and provision of feedback. Case information was provided before and during the assessment, with students objectively assessed using an internally developed rubric. Student performance scores on the virtual OSCE were then compared to the previous cohort who received the in-person equivalent. Individual case and overall examination class averages were compared between cohorts, as well as overall student failures of the exam. Results: Year-end OSCEs were completed in an online format for 379 students in years I and II. Individual case performance on the virtual exam improved compared to the in-person format. Overall virtual examination performance was higher by 10% and 19% for year I and II students compared to the previous year of in-person exams. There were no failures in the virtual assessment as compared to 4 student failures for the prior in-person format. Conclusions: Performance scores for the virtual OSCE were higher than previous years, and with fewer failures. Because the cases and assessed competencies differed between the formats, comparison of specific components of the exam proved difficult. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/AKO2gnmaqMs

Amberly Ferguson

Objective: The purpose of this report is to describe the development of 4 learning aid activities as low-stakes formative assessments in place of in-class activities due to lecture-based courses being delivered synchronously online during the 2020 Fall term. Methods: The learning aids consisted of 20 to 50 exercises of varying complexity. For example, there were matching sections, true/false sentences, fill-in-the-blank, and critical thinking questions. The course was divided into 4 units. Students were provided with the activity at the start of the unit and it was due prior to the unit exam. On the due date, an answer key was posted for students to self-correct their work. These activities were not graded for accuracy, but for overall completion. Questions on activities 2 and 4 asked for feedback about the course, allowing the students to share what was working well and what could be changed to make it better. Unit exams during the Fall term were 40 questions and the Summer term had 20 questions. Results: Exam scores dropped compared to the previous term. The final calculated grade median for the Summer term (n= 99 students) was 90% and for the Fall term (n= 60 students) was 86.56%. Student feedback suggested the low-stakes assessments were helpful in preparing them for the unit exams, but they would have preferred to do them as smaller collaborative in-class activities. Conclusions: Student performance was not enhanced by comprehensive formative assessments, but were viewed favorably by students. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/TMaJXm6ZJG4

Giulia Fuschini, Pablo Perez de la Ossa

Objectives: The aim of this study was to determine whether there was assessment consistency between online written and face-to-face examinations. Methods: Due to the COVID-19 pandemic, the Barcelona College of Chiropractic (BCC) decided to use online teaching and assessment during the second half of academic year 2019-2020. The use of online examinations is common in distance teaching, but there were some concerns about using it in face-to-face programmes. During the second semester of 2019-20, all written examinations were done using the Inspera platform and a secure browser. Further invigilation was done using webcams controlled by college staff. Only the marks for end of the semester written exams were considered. Face-to-face exam marks were collected for the 3 previous years (2017 to 2019). Online examination marks were obtained from the results of the 2020 year and were compared to previous years' marks using 1-way ANOVA. Bonferroni post-hoc test was used to determine pairwise significant differences. Results: We did not find any significant difference between most 2020 exams with other previous year exams. The only significant difference was found in year 2 (25% increase in marks compared to previous years) written exams. This may be due to changes in teaching content for this specific module. Other modules' content remained without substantial changes, although they all were delivered online. Conclusion: The results of this study suggest that online examination results are consistent compared with face-to-face exams and may be considered as an interesting alternative for the assessment of modules using written examinations. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/pj0xtL0jtZY

Fiona Jarrett-Thelwell, Chris Borgerding

Objectives: The purpose of the study was to compare course performance using virtual positioning software as formative feedback (COVID-19 group) with delivering all instruction in the x-ray positioning laboratory (pre-COVID-19 group). Methods: The formative feedback was a 1-hour pre-laboratory that used Shaderware virtual radiology software and an assignment workbook to address COVID-19 requirements of social distancing and population density in the x-ray laboratory. Students selected a simulated patient and appropriate x-ray cassette, and then manipulated the virtual positioning software to obtain radiographic images of the spine and extremities. Additionally, instructors guided students to enter various suboptimal radiographic factors into the virtual positioning software to provide them with tangible examples of distorted x-ray images. The summative assessments were 12 weekly assignments and midterm and final examinations in the x-ray laboratory. For all summative assessments, a 20 point rubric graded students on the following fundamental principles of plain film radiographic techniques: console panel; radiographic cassettes; mensuration procedures; central ray alignment; collimation; tube tilt; tube-film distance; bucky placement; patient positioning; patient instruction; radiographic anatomical marker placement; gonadal shielding equipment. A Chi Square Analysis tested the null hypothesis. Results: Distributions of final letter grades were similar between COVID-19 group and the comparable pre-COVID-19 cohort (χ2(df=3,n=261) = 3.035, p=0.386). Qualitatively, students expressed satisfaction with using the virtual positioning software. Analyses of Likert-scale ratings of student satisfaction are pending. Conclusion: These data suggested that the virtual positioning software was feasible as formative feedback to assist students with learning the fundamental principles of plain film radiographic techniques. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/kAg-7WGNvtA

Suzanne Lady, Martha Kaeser, Cecelia Martin

Objectives: The objective of this study was to examine any overall differences in chiropractic student performance when transitioning from a face-to-face simulated clinical activity using standardized patients (SPs), to a fully virtual activity. Methods: Using a mixed methods approach, student data (n=211) was compared from Fall 2019 and Fall 2020. t-tests were conducted on final examination scores while qualitative data was analyzed using the constant comparative method. Results: A comparison of 2 different cohorts of students' final examination scores, conducted by SPs in Fall 2019 and Fall 2020 was conducted. Fall 2020 final examination scores (M = 96.91, SD = 5.20) were significantly higher when compared to fall 2019 final examination scores (M = 92.66, SD = 4.21), t(209), -6.50, p<.001; d = .90. Students also completed a written self-assessment of their performance after the final examination. Qualitative evaluation demonstrated more uncertainty about history taking in a virtual environment. Conclusion: The shift to virtual encounters was pivotal for seamlessly transitioning to an online platform that allowed for formative and summative history taking and communication skill assessment. When comparing student performance on the final examinations, a significant difference in student performance was found with students in a virtual environment performing better than their face-face-counterparts. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/GxB9Db8VmsQ

Anne Maurer, Mark Pfefer

Objective: The purpose of this study was to assess student communication skills using a video assignment. Methods: Students enrolled in an online Doctor of Chiropractic program communications course had the option to produce audiovisual content consisting of observed, rubric-led speaking tasks in simulated marketing and patient scenarios. Students across 2 trimesters produced videos for a hypothetical chiropractic practice. Students produced 2 educational videos appropriate for social media use and 1 video addressing a hypothetical patient with a problem. Video content was assessed by a single instructor using a 6-point rubric addressing: 1) student's introduction, 2) synopsis of a musculoskeletal condition, 3) role-played patient concerns, 4) treatment approach to the condition, 5) student qualifications, and 6) a call to action for the patient. Completed assignments were graded with Satisfactory, Incomplete, or Missing categories. Students were asked to provide feedback on the assignment. Results: Forty-nine out of 58 students completed 3 video presentation assignments. Over 80% of students across 2 trimesters received “Satisfactory” scores on all 3 video assignments. Twenty-six percent of students completed only a portion of the video assignment. Student feedback included that they did not submit assignments due to emotional discomfort, disinterest, and/or lack of time. Conclusion: This assignment assessed students on patient-specific speaking tasks. We learned the reasons why some students chose not to complete this audiovisual assignment. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/BpYsKPunYAs

Daniel Moore, Paul Chesterton

Objective: This presentation describes the implementation of a Future Facing Learning (FFL) toolkit initiative, how it has been used to support our hybrid teaching and assessment model and the infrastructure required to deliver this initiative. Method: The FFL toolkit was developed to digitally empower students by supplying an Apple iPad to all full-time students and staff. The iPad was loaded with useful app's as well as direct access to a Virtual Learning Environment (VLE). Assessment models were implemented using a VLE for remote summative tests in customisable formats. The VLE was also used for asynchronous formative assessment. Third-party app's such as Kahoot and Padlet were used for remote formative assessments. Results: The FFL initiative reduced digital barriers to student engagement, such as digital poverty. In module evaluations conducted during the coronavirus pandemic (n=6), overall module satisfaction did not identify concerns regarding implementation of online assessments with a score of 1.2 on average (1 strongly agree, 5 strongly disagree, N=13). On average, students rated the opportunity they had to use digital tools or technology to enhance their modules as 1.5. This was a positive indication digital tools likely enhanced their learning and assessment experience. Conclusion: Students' digital readiness created through the FFL initiative supported the required transition to online learning and assessment. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/EYnbM3RFDu0

Lia Nightingale

Objectives: Online instruction has hindered individualized feedback on a clinical nutrition application project, taking a toll on project quality. The purpose of this report is to describe how the project was redesigned to improve feedback provided to students with an opportunity to earn back points, in hopes of improving the quality of project submissions. Methods: The project was split from 2 parts into 3 smaller sections. The first section included a more thorough patient history, a 24-hour food recall, and data assessment as problem, etiology, and sign/symptom (PES) statements. Following feedback, students were given the opportunity to correct their PES statements and formulate goals utilizing peer-reviewed evidence. Finally, following another round of feedback, students were tasked with piecing the information into a documentation note and Critical Thinking Worksheet (CTW). Project quality was assessed via updated rubrics and final exam PES scores were compared to 3 terms of students prior to the redesign, including face-to-face and online instruction. Results: Average project scores were 8% higher following the redesign (n=98 with redesign; n=264 without). Overall, the documentation notes and CTW demonstrated greater critical thinking compared to several terms prior to the restructure. The portion of the final exam assessing PES formulation was 3% higher following the project redesign, suggesting an improvement in student learning. Conclusion: Redesigning an application project in a clinical nutrition course by providing more feedback opportunities improved the quality of student projects and enhanced student learning. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/wSemiy-jfe0

Cortny Williams, Cecelia Martin

Objective: The purpose of this study was to analyze student responses from peer assessment assignments to determine the similarities and differences of providing feedback in online and on-campus experiences. Methods: Peer assessment assignments are an element of team-based learning (TBL) in year one of a doctor of chiropractic program. The TBL modules were delivered on-campus prior to the COVID-19 pandemic and have since been delivered online. The assessments measure student perception of key performance indicators of their team effectiveness. Qualitative responses were analyzed using the constant comparative method. Means and standard deviations were computed on Likert-scale items and results were compared with qualitative responses. Results: Students valued communication in an online experience more than in an on-campus experience, as evidenced by thematic analysis of open-ended questions. Similarly, when comparing a 5-point Likert-scale item about asking useful questions during TBL module discussions, mean scores were higher in the online cohort (3.82, p<.001) compared to the on-campus cohort (3.57). Preparedness was another valuable key performance indicator for the online delivery of TBL (3.87; p<.001) compared to the cohort that experienced TBL on campus (3.51). Conclusion: Peer assessment provided students the opportunity to develop the necessary soft skills to promote professionalism and interprofessional collaboration. Students self-reported that they valued communication and preparedness more frequently when the TBL experience was delivered online than when it was delivered on-campus. (This is a conference presentation abstract and not a full work that has been published.)

Video Abstract https://youtu.be/OwTQsTyX8tI

Publication of the proceedings was funded by CERF. No conflicts of interest were reported for the publication of these proceedings.