Restructuring of an evidence-based practice curriculum and assessment with structural mapping by course outcome verb

This paper describes how we restructured an evidence-based clinical practice (EBCP) subcurriculum within our overall doctor of chiropractic (DC) training program. Our program is an integrated curriculum. In this curriculum, some courses are a combination of related disciplines instead of covering a discrete discipline. For example, anatomy, physiology, diagnosis, patient management, and philosophy are integrated within most courses for the entire 13 quarter-based curriculum. Our program also has subcurricula, which are main themes (eg, communication or diagnosis) that are weaved throughout the integrated curriculum. Sometimes these themes combine into 1 outcome such as, “The student is able to communicate the diagnosis and treatment to a mock patient using EBCP terms such as absolute risk ratios in a report of findings.”

At the time of the revision of the EBCP subcurriculum it was not fully meshed within the integrated curriculum. The EBCP subcurriculum was limited to the outcomes in a first quarter course. Some professors who were early adopters of EBCP covered related topics in other courses; although, there was not an accounting of those EBCP outcomes. Clinic faculty members had started to integrate EBCP in their teaching as well, yet there was not a known curricular structure where the EBCP topics were intentionally integrated in the curriculum. Given the integrated curriculum, it was a challenge to match discrete subjects with EBCP topics. For example, all classes about diagnostic procedures (eg, physical examination) should have included diagnostic validity (likelihood ratios) and reliability (κ, interclass correlation coefficients) in related course outcomes.

We had to develop a strategy that would provide a basic structural goal, reveal where professors were already teaching EBCP topics, determine where we needed volunteers to teach the topics, and solicit volunteers to teach the material in courses that related to the topics and were not already laden with content. In addition, we had to provide resources for those who were unfamiliar with EBCP topics. The challenges of the integrated curriculum resulted in what we believe is an innovative model for the incorporation of EBCP outcome measures into a health care provider curriculum.

Different outcome verbs carry different “depth of processing” levels. For example, “After this course, the student will be able to define evidence-based clinical practice.” The verb “define” means that the student will memorize the definition, although the outcome does not require critical analysis of the definition. The various depth of processing levels were developed by Benjamin Bloom around 1956. Mary Allen listed the following ordinal Bloom's verb levels: knowledge, comprehension, application, analysis, synthesis, and evaluation and included a table of “relevant verbs” that are related to these levels.¹  The verbs can be used to take a student from memorization of a concept, to application, to critical analysis throughout an entire curriculum. The verbs can also be used to encourage a teacher to assess deeper levels of processing. Thus, students must master the content in order to apply it to their patient's benefit and further the development of the profession through critical analysis. The ability for professionals to question established protocols and taboos enables them to think through nonscripted situations in practice and improve patient management skills.

The background of EBCP has been described by many authors in the past and should be a familiar topic to chiropractic educators. Therefore, we defer to other sources for details on EBCP.^2,3  Basic implementation of EBCP can be achieved by using the 5 As (Ask, Acquire/Access, Appraise, Apply, Assess) model adapted from Sackett.⁴  Basically, a clinician asks a clinically relevant question. The main terms in that question are used in a database search to acquire related articles. The clinician appraises the article for quality and applicability to the clinical situation. If applicable, the article's recommendations are applied to the patient and then an assessment is made to see if the intervention helped. The applicability of mentioning this model in this paper is 3-fold: (1) because the 5 As parallel critical thinking constructs,⁵  then using the process demonstrates critical thinking in student learning with respect to patient care; (2) helps to delineate topics for outcome measures for an evidence-informed curriculum; and (3) provides a descriptive overview for those who are new to developing an EBCP curriculum.

There are many stated benefits to EBCP and the importance to stakeholders (patients,^6,7  students,^8–10  clinician¹¹ , other health professions,^12,13  and accreditors^6,14) is evident. These benefits include improved patient outcomes, common ground, and meeting accrediting requirements. The benefits of EBCP make it a worthy inclusion into the content of educational programs for physicians. Generally, content should be assessed for its usefulness to stakeholders, especially patients. Otherwise, the content becomes a waste of time to teach, manage, and assess for the teacher and an unnecessary barrier for the student to learn more applicable concepts.

Significant barriers exist to the integration of EBCP into curricula, into the practice of EBCP in clinical training settings, and into practice. The following barriers are prominent themes in the literature: time constraints,^{2,4,6,7,15,16}  confidence,^9,17  a lack of adequately trained faculty,^7,15,16,18  teaching methods,^6,19–24  clinical challenges,^{7,8,12,18,25,26}  negative attitudes toward EBCP,^4,7,8,18  and the belief that EBCP might degenerate into “cook-book” clinical practice.⁴ 

Given the above, the overreaching research question during our process was, “How does an educational institution prepare students for their intended profession?” More specifically, we questioned “How do we reevaluate curricula and determine if curriculum design promotes student learning to the benefit of their patients?” We also wanted to know the specific criteria that should be present in the curricular structure. To determine the degree of student learning, we wanted to know the level of depth that content was being taught and the assessment scores compared with the threshold. Finally, we wanted to identify if the content was necessary for students to perform their intended profession. For this project, we concentrated on the connection between the structure and level of student learning.

Because the subcurriculum was complex, visual representation of the way the EBCP topics progressed was desired to avoid gaps and a lack of Bloom's verb progression. We designed a graphical analysis of the curriculum to provide the visual representation. Therefore, our aims with this paper were to (1) describe the process we used to recreate our EBCP subcurriculum; (2) identify if the Bloom's verb level seemed to progress upward from the beginning of the curriculum to the end; (3) using the same plot, determine if there were gaps where a topic was not addressed periodically during the curriculum for learning reinforcement; and (4) determine if our approach to restructuring the EBCP subcurriculum provided adequate integration.

The barriers pointed out earlier had to be lowered to reach our goal of revising the EBCP subcurriculum. Our solutions to these barriers included resources and training. We informed the faculty of various resources, such as clinical practice guidelines. One helpful resource was the Canadian Chiropractic Association's clinical practice guideline Initiative (https://www.chiropractic.ca/guidelines-best-practice/practitioners/). This site has clinical practice guideline for headaches, neck pain, mid back pain, upper and lower extremity disorders, and other conditions.^27,28 

It seems that many of the barriers noted above were partially overcome by training in EBCP provided to our faculty over several years. Faculty members were given opportunities to participate in intensive EBCP training series provided by different schools, including Northwestern States University, Palmer College of Chiropractic, University of Iowa, and McMaster's University. This training was possible through a R-25 grant from the National Institutes of Health.

Based upon the levels of training of our faculty in EBCP through the R-25 grant, an EBCP faculty committee was formed comprising clinical and academic faculty. An EBCP topics list was created with proposed sequencing throughout the curriculum by 2 to 3 topic experts who formed an initial curriculum map. EBCP topics were determined by categorizing content into similar topics. Sequencing was based on where topics related to outcomes in the courses. EBCP committee faculty were asked to review the map and provide feedback. Changes were made based upon this feedback.

Committee members were then asked what EBCP topics they already taught and were willing to teach to complete the proposed sequencing. Learning outcomes were assigned based upon the sequencing and committee faculty feedback. Competencies and learning outcomes were created for each of the evidence-based topics by content experts. Most topics were intended to be introduced in quarter 1. Inclusion of 1 to 3 new EBCP learning outcomes in 1 class per quarter was implemented with reinforcement of each learning outcome in a later quarter. Evidence-based competencies were placed sequentially in courses so that the competency was introduced in first quarter, covered in greater depth in a subsequent quarter and reinforced in a later quarter as exemplified in Table 1. These sequential competencies were added to existing course syllabi as learning outcomes from first to 13th quarter. Learning outcomes were debated, refined, and approved by a learning outcomes committee and then approved by a faculty and administration level curriculum management committee and administrative committee. This project was authorized by the academic dean and granted exemption by the Palmer College of Chiropractic institutional review board (# N2018-10-1-M).

A frequency analysis of the use of Bloom's verbs served as a starting point that we synthesized with statistics software to make a graphical representation. First, the outcomes from the syllabi were paired with relevant EBCP topics. Table 2 lists the EBCP topics and corresponding content. Most outcomes covered multiple EBCP topics. Some of the outcomes were at 2 Bloom's levels in 1 course, although most were at 1 level. Some courses had projects that often incorporated multiple topics and levels of understanding. Microsoft Excel (Microsoft Corp; Redmond, WA) was used to prepare data for analysis (Table 3).

Our structural analysis involved creating graphs of the topics that plotted Bloom's verbs on the y-axis and the quarter on the x-axis. Assessments were structural or learning performance focused. Structural assessments included determining aspects of sequencing, repetition, and level of rigor (Bloom's verb level). Learning performance assessments involved determining the grades achieved and whether they demonstrated clinically meaningful learning.

We used the ggplot2 (Wickham, Hadley; Switzerland) graphics program of R (version 3.4.3, R Foundation for Statistical Computing, Vienna, Austria) to plot our subcurriculum.²⁹  Structural analysis methodologies used by other researchers include Academic Analytics, Walktrap algorithm³⁰  and Bloom's verb based Frequency Analysis.²²  In our analysis, the quarter was assigned to the x-axis, Bloom's level to the y-axis, and the topic number to functions called color and faceting. Figure 1 shows the ggplot2 script.

The EBCP task force supported the faculty by creating teaching tools, including detailed PowerPoint presentations (Microsoft Corp) with a standardized format, for each competency. The teaching resources were then compiled into a single resource for all faculty members to assure consistency in the information being provided. We also implemented a continuing education program about EBCP for all faculty members. During these periodic in-service meetings, concepts that have been identified and that need to be emphasized and taught to students are reinforced.

The mapping of the curriculum quarters versus Bloom's verb level is presented in Figure 2. The value for Bloom's verb should increase as the student goes through the curriculum, showing a progression from memorization to critical analysis. Another feature was the ability to find gaps. A gap in the quarter on the x-axis shows that the topic is not being reinforced during that quarter. This could be an issue or not depending upon how much reinforcement is needed for that topic. A gap in the Bloom's verb level means that the topic is not being assessed at that level. These gaps showed that the outcome depth of processing did not have foundation, mid-level building blocks, or a cap stone, higher level exposure.

Table 3 and Figure 2 were used to determine the amount of EBCP topics per quarter and where they are more concentrated. Table 4 showed that 14 of the 19 topics had increasing linear model trends, 5 had short linear trends, and 1 had no trend. Decreasing, short trends were found in 3 topics. Quarters 5–8 showed decreased attention to EBCP topics, especially quarter 5.

The 14 of 19 increasing linear model trends were preferred patterns because they represent topics assessed at higher levels as the quarters progressed. The 5 short linear trends represent topics that are not assessed later in the curriculum. The 1 topic with no trend showed that the topic is taught only once. The 3 decreasing short trends represent topics that were developed at a higher level in the beginning but at a lower level later in the program. This is not a preferred trend. This structural analysis of the EBCP subcurriculum was helpful for us to detect trends and gaps for further refining of the curriculum. In our assessment plan, we have addressed building a curricular structure that includes early, frequent, repetitive, progressive development of EBCP topics. Based on our analysis, further development of the structure is planned.

Our future plans involve tracking the EBCP competencies tied with the learning outcomes within the courses. We will need to overcome some barriers to assessment, such as the compilation and analysis of data from all the courses into an easily understandable product. As well, it will be important to limit the changing of course outcomes contrary to an adequate process of revision (ie, curricular drift).³¹  It will also be important to identify outcomes that are not sufficiently demonstrated by students if they are intentionally doing well enough on other content to receive a passing grade. For example, a student could do well on the easier topics and not the harder topics.

Specific structural changes may be needed to address shortcomings identified with the structural analysis. We also plan to gather information about the students' mastery of EBCP topics through programmatic assessments. To assess our students' ability to apply EBCP principles to patient care, we can review data on our interns in quarters 9 through 12.

We believe that 1 of the unique aspects of our project was the goal of EBCP outcomes being introduced early, frequent, repetitively, and progressively throughout the entire curriculum. As stated by Bussieres et al⁷  in 2016, “academic programs should incorporate EBCP content early in a program of study and ‘weave' it throughout the 4–5 years of professional education.” We look forward to performing our learning assessment following this “weaving” concept. Our desired end goal is to assure a progression from knowledge to comprehension to application in all topics and lead to successful clinical integration of EBCP by our students. As noted by Nkangienieme³³:

It is of particular interest to note that students have been found to value higher order learning opportunities for EBCP; as noted by Thomas et al³⁴  in 2017, “Students believe that application of EBCP in the clinical context is of utmost importance and that the academic program should design opportunities for this to happen as often as possible.”

Comparison with another method by Whillier et al³¹  is presented in Table 5. We are interested in their more open format, research stream to increase research capacity, and inclusion of recent graduates and chiropractic clinicians in noncollege-related practice.

There were several limitations to this study. There was no learning assessment as part of this project. We did not ask each professor whether any EBCP topics were embedded in their course without an outcome. We also did not address the issue of subjectivity across the level of Bloom's taxonomy. Such that, “appraisal” can be done at a beginner level where the student can “identify” or at an expert level where they “criticize.”³⁵ 

The graphical verb-based, structural analysis seemed useful to identify progression of intended learning complexity and gaps in presentations of topics in our EBCP subcurriculum. Fourteen of the topics had increasing linear model trends indicating verb progression. Insufficient attention to EBCP topics was identified in some quarters. We hope that our experience incorporating EBCP into an integrated curriculum may be helpful to others.