Chiropractic program changes facilitated by the European Council on Chiropractic Education Accreditation reports:A mixed methods audit and thematic analysis

Quality assurance of modern higher and postgraduate health care educational programs is considered crucial in order to produce knowledgeable and competent graduates.^1–4  This is particularly critical for primary contact health care professionals, including chiropractors, in order to protect the graduates' future patients and to instill life-long learning habits.²  Quality assurance policies and procedures should be conducted internally by the college, university, or specific program as well as externally by recognized quality assurance agencies.^4–6  External quality assurance evaluations should provide a unique perspective without the potential blinders that may be present if programs rely solely on internal reflection and assessment. External quality assurance agencies can also compare the relevant programs with other similar programs nationally and internationally, which may facilitate exchanges of best practice as well as international mobility of graduates.^7–11 

One of the primary contact health care professions in Europe that has continued to grow over the past several decades is chiropractic. Although the legal status of this profession varies widely in Europe, from being 1 of the 5 recognized medical professions in Switzerland to still being illegal in Spain, most western European countries have laws that regulate the profession.¹²  This requires that the education and training of chiropractors meet specific “Standards” to prepare graduates to be safe and competent practitioners and to provide evidence-based care to their patients.^7,12 

The European Council on Chiropractic Education (ECCE), which has been in existence since 1981, currently accredits 10 chiropractic programs; 8 in various European countries and 2 in South Africa.¹³  Additionally, several new chiropractic programs have started in Europe to meet the growing demand for this profession, but have yet to undergo ECCE evaluation, pending the graduation of their first cohorts of students. The ECCE is a member of the Council on Chiropractic Education International (CCEI) and has mapped the ECCE “Standards,” “Competencies,” and “Policies and Procedures” to the CCEI “International Framework” as well as with these same documents from the other international member agencies. In addition, the ECCE has also mapped its “Standards” against the European Standards and Guidelines for higher education in Europe.¹³  (Mapping documents are available upon request.) The purpose of these mapping exercises was to identify any gaps/omissions in the ECCE “Standards,” which needed to be addressed. The current version of the ECCE “Standards” includes all updates and changes based on these mapping exercises.⁷  As the primary purpose of accreditation of chiropractic programs/institutions is to ensure safe and competent graduates, thus protecting chiropractic patients, making sure that the “Standards” are appropriate and current is critical. Additionally, high quality, current educational “Standards” may help facilitate international mobility of graduates and basic standardization of the profession.

Although it has always been assumed that undergoing evaluation by the ECCE and receiving accreditation of a program helps the program to grow and improve, detailed investigation as to whether or not this is true as perceived by the specific chiropractic programs has never been done. Obtaining this information is important to assure that the ECCE continues to be an effective accrediting body that provides useful feedback to chiropractic programs. To do this, the ECCE must also evaluate its own processes and procedures to determine whether or not its evaluations are useful to chiropractic programs (and to the profession at large) as well as to identify areas within the ECCE that may need improvement. The ECCE has recently completed and published 2 studies requiring internal reflection resulting in change/improvements to its policies and practices.^14,15  Therefore, the purpose of this third study sought to identify whether feedback from past ECCE accreditation evaluation reports has resulted in changes in the individual chiropractic programs and if so, to identify specific changes made and to what extent.

This study used a mixed-methods audit and thematic analysis approach collecting survey data from the department heads or program directors of each of the 10 ECCE accredited chiropractic programs. An audit was considered the appropriate tool to collect the necessary information as audits focus on evaluating and analyzing existing tasks and procedures in a systematic approach (ie, internal introspection) in order to determine if changes are needed.¹⁶  Similar to many research designs, audits start with important questions, but these are related to current practice and procedures.¹⁶  Questionnaires were designed and evaluated for face and content validity independently by each of the authors as well as the ECCE executive members and suggested changes included in the final version. The final, approved version of the questionnaires included 2 sections. The first section contained 10 questions related to the audit portion of this study. Each question corresponded to 1 of the 10 sections of the ECCE “Standards.” The instructions to the participants for this section were as follows: “To what extent, if any, have past ECCE evaluation reports facilitated your educational institution to make changes/improvements to the components identified in [specific category inserted here (eg, Aims and Objectives)]?” The options included the responses “substantial changes made,” “some changes made,” and “no changes made.” The respondents were instructed the use their own interpretation as to what qualified as “substantial changes” or “some changes” for their programs as this may vary depending upon the country where the program occurs, whether or not they are part of a long-established university, and how long they have had ECCE accreditation. What is substantial for 1 program may not be for another. Therefore, attempting to control for program “maturity” or length of ECCE accreditation was not desired. Discussions with the ECCE executive and ECCE members prior to conducting the study concluded that the 3 options of “substantial change,” “some change,” and “no change” provided adequate differentiation in the responses to avoid confusion for the responders. For each of these 10 closed questions, annotations were also included listing each of the individual “Standards” that fell into the category (Appendix 1). The frequency of responses falling into the 3 categorical options for each question was calculated.

The second part of the survey collected written responses to the statement “This section requests that you list/identify (1) changes that you did make based on feedback from an ECCE evaluation report; and (2) changes that you were unable to make and the reasons why these were not done” (modified thematic analysis section). The respondents were also asked to identify “substantial changes” vs “other” (less substantial) changes done because of feedback from an ECCE evaluation report. The data were collected for 9 of the 10 chiropractic programs on hard-copy forms distributed during the annual ECCE meeting held in November 2019. Participants were aware in advance of the purpose of the study and had opportunities for questions and discussion prior to completing the data collection forms. One program head did not attend the meeting in person and completed the questionnaire electronically.

All written responses listing the specific changes were copied verbatim and placed in a single document to be assessed independently by 3 experienced researchers using a modified thematic analysis approach, similar to a recent study published by the ECCE (Appendix 2).¹⁵  This modified thematic analysis approach used written responses rather than live oral interviews to collect the data. These written responses were analyzed and coded according to thematic analysis protocols using an inductive approach (ie, starting with observations that are specific and limited in scope, proceeding to generalized conclusions that are likely but not certain). Each researcher was instructed to identify recurrent “themes” from the written responses for each of the 3 statements. Once this was completed individually, the researchers met together to discuss their individual findings, explain their rationales, and to agree on the final general “thematic categories” as well as the specific “themes” falling into each category.

Ethical approval was not necessary for this audit study with voluntary participation and no interventions, in accordance with the literature and other recent studies published by the ECCE.^14–16  Returning the completed questionnaire was considered informed consent to participate, and participants were informed of this as well as the fact that their responses would not be anonymous to the researchers conducting this study. Information regarding the purpose and design of this study and lack of anonymity of responses was presented to the institutional representatives during the ECCE annual November 2019 meeting by the ECCE Quality Assurance Consultant with the opportunity provided for questions and clarifications from the participants.

All 10 chiropractic programs returned their completed questionnaires. Table 1 shows the proportion of “no change,” “some change,” and “substantial change” responses for each program for each of the 10 sections of the ECCE Standards. Only 1 program stated that they had made no changes to their program based on feedback from ECCE evaluation reports.

Of the 9 accredited programs reporting changes based on ECCE accreditation report recommendations, 7 reported “some” or “substantial” changes to 6 or more of the 10 sections of the ECCE Standards (Table 1). One program reported that they had made changes to all 10 sections of the ECCE Standards based on feedback from the evaluation reports, with 3 of the areas undergoing “substantial” changes (Table 1).

The most common section of the ECCE Standards for programs to report “some” or “substantial” changes based on feedback from ECCE Accreditation Reports was “Educational Program” (section 2 of the Standards) with 8 of the 10 programs having made changes within this section. Section 6 of the ECCE Standards (Educational Resources) had the largest number of programs reporting “substantial changes” (4 in total) and was the second most common section of the Standards to have reported changes (Table 1). The 3 sections of the Standards where programs were least likely to report any changes were as follows: (1) “Students” with 7 of the 10 programs reporting “no changes”; (2) “Governance and Administration” showing 6 programs making no changes; and (3) “Assessment of Students” also showing 6 programs making no changes (Table 1).

Regarding the responses to the open-ended written questions asking the programs to identify and describe the changes that they had made due to feedback from the ECCE accreditation/evaluation reports (listed in Additional File 2), the researchers identified 3 areas/categories where changes were made (Table 2). These were (1) Improved/Augmented Resources; (2) Increased Evidence-Based Teaching and Learning; and (3) Increased Research and Critical Appraisal. Specific “themes” identified by the 3 researchers relating to each of these 3 areas are listed in Table 2. In particular, written comments underpinning the themes identified in the category regarding “Resources” stated that the ECCE evaluation reports were instrumental in helping programs obtain not only better physical facilities, but more faculty and faculty with higher/specialist qualifications. The following are quotes from Appendix 2:

The primary difference between the other 2 thematic categories (ie, Increased Evidence-Based Teaching and Learning vs Increased Research and Critical Appraisal) was that the latter category included performing actual research studies, whereas “Increased Evidence-based Teaching and Learning” focused on increased integration of subjects, implementing the biopsychosocial model of health care teaching and learning, and providing educational experiences that were more patient and student centered. Three relevant quotes from Appendix 2 are shown.

Although the impact and effectiveness of accreditation on medical education processes have been reported,^1,2  this current study is the first within the chiropractic profession to assess the effects, if any, that accreditation has on the way chiropractic programs evaluated and accredited by a chiropractic specific accrediting agency develop and function. It was very encouraging to discover that 9 of the 10 ECCE accredited programs reported that they had made changes/improvements to their operations based on the feedback from the ECCE accreditation reports. The only program that did not report any improvements/changes stated that because they are also accredited by an agency within their own country, this takes priority over feedback from the ECCE. Consequently, they admitted in the comments section of the questionnaire that some recommendations contained in the ECCE report could not be implemented. However, 6 of the other ECCE accredited programs also undergo accreditation by their individual country's chiropractic, higher or medical education accrediting bodies but were able to address issues that arose from the ECCE feedback reports. This difference is most likely due to the varying regulations and laws governing education throughout Europe and South Africa. Furthermore, there is currently considerable collaboration between the national chiropractic accrediting bodies in the United Kingdom and Switzerland with the ECCE. The accrediting body in South Africa also recognizes the ECCE (personal communication from the ECCE vice president and South African department head).

The most common section of the ECCE Standards where changes were made following feedback from the ECCE accreditation reports was “Educational Resources” (section 6).⁷  This section of the ECCE Standards contains 3 of the 18 “critical Standards” that must be at least “substantially compliant” in order to achieve the maximum 8-year accreditation time period.¹⁴  Thus, chiropractic programs would likely be very cognizant of the relative importance of these particular Standards and very motivated to make changes to their programs in order to achieve a high level of compliance. Indeed, 9 of the 10 accredited programs reported that the ECCE accreditation reports motivated their program's academic leaders to make needed changes in this area. This section also dominated the written comments listing the various changes implemented. Thematic analysis of these written comments found that the most common specific changes/improvements made were in Infrastructure/Physical Facilities and Equipment, increasing the number of faculty members with the appropriate qualifications and experience and increasing the opportunities for interdisciplinary teaching and learning. Having an unbiased, external, professional accrediting body identifying these areas of weaknesses and recommending specific improvements appeared to be instrumental in motivating the leaders of the programs to make the needed changes.¹  This supports the value of external quality assurance councils performing accreditation activities as internal quality assurance procedures alone may not have the broader perspective.^1,3,5 

The second most common section of the ECCE Standards where respondents reported changes to their programs arising from the ECCE accreditation reports was the “Educational Program” itself (section 2).⁷  Thematic analysis of the written comments (Additional File 2) relating to this topic found that several programs increased their focus on evidence-based teaching and learning as well as evidence-based practice, which incorporates the Bio-Psycho-Social model of health care education. Additional themes that arose for this section included improving the integration of subjects, closing the theory-practice gap as well as increasing self-directed learning, which all lead to decreased contact time for students. Five of the 18 “critical” ECCE Standards that must be at least “substantially compliant” for the maximum accreditation time period fall into this category.¹⁴  This most likely provided additional impetus for programs to change these areas.

The researchers also identified a third area in the thematic analysis of the written comments entitled “Research.” Although this may appear to link into the previous thematic section on evidence-based teaching and learning, the actual responses falling into this category were more specifically on teaching faculty and students how to perform research studies, analyze the quality of research publications, as well as how to use existing research in daily practice with patients. This aligns with ECCE “Standard” 2.2 “The Scientific Method,” which is also 1 of the “critical” “Standards” requiring at least a substantially compliant rating for the maximum accreditation time period.¹⁴  The additional theme identified by the researchers for this section of “Standardized Evaluations and Feedback” refers to using questionnaires to obtain feedback within the program regarding course/class evaluations and faculty performance evaluations (Additional File 2). The purpose of this type of feedback is to perform internal quality assurance and facilitate changes and improvements to the program.

Although the ECCE “Standards” section on “Students” showed that 7 of the 10 programs reported no changes to their programs based on evaluation report feedback regarding this area, this is because that particular section of the “Standards” deals only with how students are selected and admitted to the programs and how they are supported and counselled. No other issues regarding students are addressed in this section as they are covered in other “Standards.”⁷  Most accredited programs were confident that their selection criteria were appropriate and that the students were sufficiently supported. Furthermore, the ECCE did not identify these areas as “Critical Standards” in the previous research on this subject.¹⁴  The majority of ECCE accredited programs are part of larger universities, which provide good student support services.

The fact that all 10 ECCE accredited chiropractic programs completed the questionnaire gives validity to the findings obtained in this study. The documentation in the form of written feedback from accredited programs that the accreditation reports have resulted in many changes for 9 of the 10 accredited programs is important feedback to the ECCE that their accreditation processes are useful. In particular, written comments describing improvements in equipment and resources as well as the number and quality of the faculty were frequently mentioned. Increasing evidence-based teaching and practice was also a frequent theme.

The fact that the ECCE is engaged in self-reflection and systematically evaluating their professional activities through audits, subsequently sharing these results in professional journals, demonstrates transparency of this organization and a desire to continuously improve not only the ECCE but the chiropractic profession.^14,15  These activities can also facilitate collaborative projects with national and other accrediting bodies with the potential to reduce the accreditation burden on some chiropractic programs and raise the professional profile of chiropractic in Europe and South Africa.

The most obvious limitation to this study was that the respondents to the questionnaires were not anonymous as they were all heads of the 10 accredited programs, with some having a longer tenure in their position than others. However, 8 of the 10 department heads had been in position for at least 2 ECCE accreditation cycles. Thus, institutional/program memory of past ECCE accreditation evaluation reports was required in order to accurately complete the questionnaire, as it was not specifically requesting information from only the most recent accreditation event. As usual there could be memory lapses or recall bias or even a tendency of the participants to report a biased answer (halo effect) to please the researcher. Further, it is possible that the level of change for 1 school may have been considered “minor” and yet “substantial” to a different school. In the future, these terms should be carefully defined.

A further limitation of this study is that the questionnaires were completed during the ECCE meeting (with 1 exception) rather than at the department heads' institutional offices. Thus, “top of the mind” reflections were likely provided as opposed to reviewing reference materials to support their answers.

Another important weakness to the study is that all responses were in written format rather than having the researchers conduct oral interviews with the participants. Oral interviews would have allowed the respondents to perhaps provide more detailed answers and the opportunity for follow-up questions from the researchers could have clarified some areas.

ECCE accreditation processes and subsequent recommendations facilitated changes to the majority of chiropractic programs that they evaluate. The main categories of “Standards” where these changes were reported included the following: Educational Resources (both human and physical), Research, and Evidence-Based Practice, all of which contain several of the individual ECCE “critical” Standards that require at least “substantial compliance” to achieve the full 8-year accreditation period.

The ECCE will continue to perform internal quality assurance studies, particularly assessing recent changes made in policies and procedures to evaluate effectiveness. These studies can improve the operations of this organization, which would reflect onto the relationship between the ECCE and the programs that they evaluate as well as facilitate joint accreditation activities in Europe.

There were no direct funding sources for this study. The ECCE provided in-kind support for the project. The fact that 2 of the authors are on the ECCE could be considered a conflict of interest and the other 2 authors being recent retirees from ECCE accredited programs, may also be a conflict of interest. Additionally, 2 of the authors have been on past ECCE evaluation teams for some of the programs participating in this study and this could be considered a conflict of interest. However, the heads of the 10 chiropractic programs were aware of this when completing the questionnaires and aware that this study is part of ECCE's internal quality assurance processes. None of the authors received speakers' fees or other fees relating to this study.