Objective

To describe the best evidence on the effectiveness of technology-based learning tools designed to improve knowledge of health care providers about clinical practice guidelines (CPGs).

Methods

We conducted a systematic review, searching MEDLINE, Embase, and CINAHL from inception to July 2018. Included studies investigated the effectiveness of any technology-based learning tools developed to improve knowledge of health care providers about CPGs. We used a 2-phase screening process to determine eligibility. Pairs of reviewers critically appraised relevant studies using the Scottish Intercollegiate Guidelines Network checklist for randomized controlled trials or the National Institutes of Health checklist for pre- and postintervention trials. Evidence from internally valid studies was described using a best-evidence summary. We conducted a sensitivity analysis to determine whether results varied according to methodological quality.

Results

Twenty-five of 8321 articles met our selection criteria. Six studies had a low risk of bias and were included in this review. Spaced education was associated with improvement in knowledge; however, its effectiveness relative to other interventions is unknown. Module-based online educational interventions were associated with improvement in knowledge of CPGs; however, they may not be more effective than paper-based self-learning or in-person workshops. The sensitivity analysis determined that the evidence was similar between the high and low risk of bias studies.

Conclusion

Module-based- and spaced-education interventions may be beneficial for improving health care providers' knowledge of CPGs; however, much of the evidence toward their use is preliminary.

INTRODUCTION

Health care providers are expected to remain current with clinical evidence,1,2  yet the use of evidence in clinical practice is suboptimal.310  There are many ways to enhance the use of research evidence, including clinical practice guidelines (CPGs). CPGs include recommendations developed following an evaluation of the scientific literature.11  CPGs optimize patient care by allowing health care providers and patients to select the best evidence-based care consistent with the unique needs and preferences of patients.11  However, difficulty accessing CPGs and evidence-based information can inhibit health care providers from incorporating this information into their patient care.510 

There is a need for the development of appropriately tailored knowledge translation (KT) activities to facilitate the exchange of evidence to health care providers.1218  KT refers to the exchange, synthesis, and ethically sound application of knowledge to improve health and provide effective services in health care.19  Education is an important aspect of KT because it contributes to all phases from its development to implementation and the final phases of evaluation. Constructivism in education is the theory that learners construct their understanding through experiences and reflection.20,21  In order to learn something new, according to this theory, learners must reconcile new information with their previous knowledge and experiences.20,21 

There is a significant shift toward the use of technology-based learning rather than the traditional in-person, classroom-based learning.22  Technology-based learning tools in health care education can improve access to information to meet the needs of health care providers.2326  Moreover, such tools can be used to adapt information to the health care providers' learning styles and increase intrinsic motivation;2427  however, there is no clear understanding of which technology-based educational interventions are most effective in improving knowledge in the health care provider population.28  In this review, we define technology-based learning tools as instruments of learning that incorporate digital technology as a method for the delivery of information.29  Terms such as Web-based learning, e-learning, computer-assisted learning, and online learning have been used synonymously with technology-based learning.4,30,31  Examples include but are not limited to websites, online courses/modules, and podcasts.

A previous systematic review aimed to identify health care providers' perceived usability and practice behavior change of technology-based educational interventions used in disseminating CPGs.30  They identified 7 types of technology-based interventions, including websites, computer software, web-based workshops, computerized decision support systems, an electronic educational game, e-mail, and multimodal interventions consisting of at least 1 technological component. The results varied by intervention type with no clear superior method of dissemination. This review provides important information; however, additional pedagogical components should be explored to better inform the development of appropriate KT tools for this population.

Our review is set within a larger study that aimed to develop and evaluate a technology-based learning tool designed to improve knowledge of CPGs within chiropractors and chiropractic teaching faculty at a Canadian chiropractic college. To develop a learning tool tailored to the target population, we aimed to determine whether previous learning tools have been developed to disseminate CPGs and then use them to inform the development of this novel tool. Understanding that there is no single right learning tool design for every population, we must integrate information from a variety of resources to develop a tool more likely to be effective. We worked with an Advisory Committee, reviewed pedagogical theories and principles for online learning, and conducted the present review to collectively construct a body of evidence toward identifying the most appropriate and well-informed tool for this population and subject. Improving our understanding of technology-based educational interventions that are effective in improving knowledge is necessary to develop KT strategies for health care providers. Therefore, we aimed to describe the best evidence on the effectiveness of technology-based learning tools designed to improve knowledge of health care providers, in active practice, about CPGs.

METHODS

Registration

We registered our systematic review protocol with the International Prospective Register of Systematic Reviews (PROSPERO) on August 3, 2017 (CRD42017071308).

Eligibility Criteria

Population

We included studies targeting health care providers in active practice including but not limited to physicians, residents, and nurses. We excluded populations, including students not yet in practice.

Interventions

Interventions included technology-based educational interventions aiming to educate individuals about a CPG. Examples of technology-based educational interventions may include web-based modules or smartphone apps. We excluded educational simulation design interventions and clinical decision support systems in this review. Educational simulation decision interventions refer to the artificial representation of a clinical scenario by a 3-dimensional application (eg, SimLab). A clinical decision support tool is any tool that provides clinicians, administrative staff, patients, caregivers, or other members of the care team with information that is filtered or targeted to a specific person or situation. Decision support tools are patient specific and provide more information about care toward the individual patient rather than a general improvement of knowledge of CPGs. These interventions were excluded, as they do not fit within our definition of a technology-based learning tool aiming to disseminate CPGs.

Comparison Groups

Studies that compared technology-based educational interventions to other interventions (technology based or not) or no intervention were considered.

Outcomes

The primary outcome of interest was a measure of knowledge following the use of the educational intervention. We did not use a specific definition of knowledge. Instead, we accepted the authors' definitions and/or means of assessing knowledge and commented on their justification.

Study Characteristics

Eligible study designs met the following inclusion criteria: (1) English language, (2) published in a peer-reviewed journal, and 3) study designs, including randomized controlled trials (RCTs), cohort studies, case control studies, and pre- and postintervention trial study designs. We excluded study designs such as case reports, case series, qualitative studies, literature reviews, biomechanical and laboratory studies, and studies not reporting a methodology and publication types, such as guidelines, letters, editorials, commentaries, reports, book chapters, conference proceedings/abstracts, lectures, and consensus development statements.

Information Sources

Our search strategy was developed in consultation with an experienced health sciences librarian, and a second health sciences librarian reviewed the search for completeness and accuracy using the Peer Review of Electronic Search Strategies (PRESS) Checklist. We searched MEDLINE and Embase (through Ovid Technologies Inc, New York, NY) and CINAHL Plus with Full Text (through EBSCOhost) from inception to July 2018. The search strategies (Fig. 1) were first developed for MEDLINE and subsequently adapted to the other databases. The search strategy combined Medical Subject Headings (MeSH) as well as text words (title and abstract) related to CPGs and technology-based education. We used EndNote X7 to create the bibliographic database.

Figure 1-

MEDLINE search strategy. Search run April 11, 2017, and updated July 1, 2018, in Ovid MEDLINE: Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE® Daily and Ovid MEDLINE® 1946–Present.

Figure 1-

MEDLINE search strategy. Search run April 11, 2017, and updated July 1, 2018, in Ovid MEDLINE: Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE® Daily and Ovid MEDLINE® 1946–Present.

Study Selection

We used a 2-phase screening process to select eligible studies. In phase I, 5 pairs of independent reviewers screened citation titles and abstracts to determine eligibility. Citations were classified as either relevant, irrelevant, or possibly relevant. In phase II, the same pairs of reviewers independently screened the possibly relevant articles from phase I to determine eligibility. Reviewers reached consensus through discussion following each phase.

Quality Assessment and Data Extraction

Four pairs of reviewers independently appraised the internal validity of eligible studies using the Scottish Intercollegiate Guidelines Network checklists for RCTs as well as the National Institutes of Health Checklist for pre- and postintervention trials.32,33  Reviewers reached consensus through discussion. Studies deemed to have a low risk of bias were included in this review. Those with a high risk of bias (presence of methodological fatal flaws, such as selection bias due to improper randomization) were excluded. We contacted the authors when additional information was needed to complete the appraisal. A study was considered to have a high risk of bias if reviewers considered that the study's internal validity was compromised because of biases and methodological flaws.

The lead author extracted data from low risk of bias studies into evidence tables to describe the type of educational intervention, population, the topic of the CPG, follow-up time points, and results of each study. A second reviewer independently verified the accuracy of the extracted data. Any disagreements were discussed until consensus was reached.

Sensitivity Analysis

We conducted a sensitivity analysis (1) to determine whether results varied between low and high risk of bias studies and (2) to assess the possible impact of misclassification bias from our risk of bias assessment. The lead author extracted data from the high risk of bias studies and created evidence tables. We stratified results of high risk of bias studies according to types of educational interventions and identified whether these interventions demonstrated an improvement, a reduction, or no change in knowledge. Finally, we described the similarities and discrepancies between the high and low risk of bias studies.

Data Summary

A narrative summary of the low risk of bias studies was performed. We stratified our results according to types of educational interventions.

Statistical Analysis

The interrater agreement for article screening was computed using the κ coefficient, and percentage agreement for critical appraisal was calculated. We computed the mean difference between groups and 95% confidence intervals (CIs) to quantify the effectiveness of interventions when possible. Where this was not possible, we reported median values and significance, as reported in the studies. More weight was given to results of RCTs. This systematic review complies with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.34 

RESULTS

We identified 8321 articles. We removed 311 duplicates and screened 8010 articles for eligibility (Fig. 2). Phase I screening yielded 97 articles, and 25 articles were relevant following phase II screening. Reasons for exclusion from phase II (n = 72) were (1) ineligible intervention type (n = 10), (2) outcomes not relevant (n = 43), (3) ineligible study design (n = 3), (4) ineligible study population (n =  3), and (5) ineligible publication type (n = 13). The interrater agreement for phase I screening of articles was κ = 0.33 (fair agreement) and for phase II κ = 0.73 (substantial agreement). The percentage agreement for the critical appraisal of articles was 59%. There were no deviations from the protocol.

Figure 2-

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Figure 2-

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Study Characteristics

We critically appraised 25 articles. Of those, 6 had a low risk of bias and were included in our summary. Four low risk of bias studies were RCTs, and the remaining 2 were pre- and postintervention trials. The studies focused on (1) primary care following myocardial infarction in resident physicians;35  (2) detection, evaluation, and treatment of high blood cholesterol in physicians;36  (3) hematuria, priapism, staghorn calculi, infertility, and antibiotic prophylaxis in urologists and urology residents;37,38  (4) health care–associated infections in health care workers (nurses, physicians, and other health care workers, including pharmacists, paramedics, respiratory therapists, and physiotherapists);39  and (5) whiplash management in general practitioners.40  The educational interventions included module-based online education (n = 4),35,36,39,40  spaced-education combined with case studies (n = 1),37  and spaced-education combined with a game (n = 1).38  Module-based online education is a series of online sequentially ordered modules, each focusing on a particular topic. Modules are often combined to teach larger, more complex topics to learners. Spaced education refers to educational interventions delivered over a prolonged period. It includes “spaces” or times without the intervention between learning intervals, which is said to improve long-term memory.41  The prolonged period between learning intervals is variable. No standard length of time appears to exist for this type of intervention. The length of the intervals included in this review is noted.

Risk of Bias Within Studies

The low risk of bias RCTs had (1) clear research questions, (2) adequate randomization processes, (3) baseline similarities between groups, (4) interventions as the only differences between groups, (5) adequate outcome measurement tools, (6) loss to follow-up accounted for in analyses, and (7) intention-to-treat analyses (Table 1, available as online supplementary material attached to this article at https://www.journalchiroed.com).3538  However, concealment methods were not clearly described for 3 studies,3638  and blinding did not occur in 1 study35  and was not clearly described for 2 studies.37,38 

Two pre- and postintervention studies had a low risk of bias.39,40  They had (1) clear research questions, (2) clearly described eligibility criteria, (3) representative study populations, (4) adequate enrollment procedures, (5) adequate sample sizes, (6) clearly described interventions, (7) loss to follow-up accounted for in analyses, and (7) adequate outcome measurement tools (Table 2, online). Both studies did not report on blinding procedures (researcher blinding to participant allocation). All 6 studies justified the selection of their knowledge measurement through content expert review,35,39,40  pilot testing,36,37  or a previous trial.38 

We excluded 19 studies due to important methodological limitations, 7 RCTs (Table 3, online),4248  and 12 pre- and postintervention trials (Table 4, online).4960 

Interventions Involving Spaced Education

Two RCTs aimed to improve knowledge using spaced education in combination with a game or online case studies.37,38  These studies provide preliminary evidence suggesting that spaced education may be associated with improvement in knowledge; however, the effectiveness of spaced education is not well established because it was not compared to a different educational strategy. Additionally, the length of the spacing did not appear to influence knowledge change.

The first RCT randomized 1470 urologists to 1 of 2 spaced-education game intervention groups (n = 735 per group).38  Although knowledge improved in both groups (group A: 52/100 increase; group B: 53/100 increase), the difference between groups was not statistically significant (group A: 100/100 [interquartile range 3.0]; group B: 98/100 [interquartile range 8.0]) (Table 5, online).

The second RCT included urologists and urology residents (n = 240 per group) who received spaced education in combination with case studies focusing on 1 of 2 CPGs.37  The only difference between the 2 interventions was the CPG being instructed. The results of this study cannot be used to determine differences in the effectiveness of spaced education between groups; therefore, we used only within-group results. The results suggest that both groups significantly improved their knowledge following the intervention (cycle 3) (p < .05); within-group difference in means (95% CI) for group A was 29.1/100 (28.0–30.14) and for group B was 24.6/100 (23.73–25.47) (Table 5, online).

Interventions Involving Module-Based Online Education

Four studies (2 RCTs and 2 pre- and postintervention studies) aimed to improve knowledge using module-based online educational programs.35,36,39,40  Based on this review, preliminary evidence suggests that online module-based education may be effective in improving knowledge about CPGs in health care providers; however, they may not be superior to paper-based self-learning or face-to-face workshops.

The first RCT randomized resident physicians to either a module-based education program (n = 83) or a printed guideline group (n = 79).35  The results indicate that participants in the intervention group scored a median of 0.5/20 higher than the control group postintervention (F1) and 1.0/20 4–6 months following the intervention (F2) (Table 5, online). These results were not statistically significant between groups (intervention: F1: 15.0/20 [95% CI 14.0–15.0]; F2: 12.0/20 [95% CI 11.0–13.0]; control: F1: 14.5/20 [95% CI 14.0–15.0]; F2: 11.0/20 [95% CI 10.0–12.0]). Knowledge increased in both intervention groups; however, the statistical significance is unknown (intervention F1: 5.0/20; F2: 2.0/20 increase; control F1: 5.5/20; F2: 2.0/20 increase).

In the second RCT, physicians were randomized to either an online multiformat education group (n = 52) or a live workshop (control) group (n = 51).36  There was no statistical mean difference between groups (F1: 1.01/39 [95% CI: −0.39–2.41]; F2: 0.66/39 [95% CI: −0.65–1.97]) (Table 5, online). However, participants in both groups significantly increased their knowledge (difference in mean test scores: intervention: F1: 11.62/39 [95% CI 10.58–12.66]; F2: 13.89/39 [95% CI: 12.86–14.92]; control: F1: 12.63/39 [95% CI: 11.70–13.59]; F2: 14.55/39 [95% CI: 13.75–15.36]).

The first pre- and postintervention study included 971 health care workers.39  The results indicated that each group (stratified by profession) significantly increased their knowledge immediately after (F1) as well as 3 months following the intervention (F2) (p < .05) (Table 6, online) (nurses: F1: 26/100; F2: 22/100 increase; physicians: F1: 24/100; F2: 15/100 increase; other health care workers: F1: 24/100; F2: 22/100 increase).

The second pre- and postintervention study included 233 general practitioners.40  The results indicated a statistically significant mean difference following the intervention (1.8/9 [95% CI: 1.65–1.95]) (Table 6, online).

Sensitivity Analysis

Nineteen studies (7 RCTs and 12 pre- and postintervention studies) with a high risk of bias were included in the sensitivity analysis. Of the 7 RCTs, 5 investigated the effectiveness of an online module-based educational intervention (Table 7 online). Three RCTs reported that online module-based interventions were superior to the control interventions (wait list/no intervention or printed guidelines).43,45,46  All 5 studies reported a within-group improvement in knowledge following the interventions.4346,48  Moreover, another RCT reported that an electronic guideline (electronic point-of-care tool) was more effective than paper-based learning.42  Finally, the last RCT reported that an electronic guideline and in-person education led to improvements in knowledge compared to in-person education alone.47 

The 12 pre- and postintervention studies (Table 8, online) included 7 studies investigating online module-based educational interventions. All 7 studies reported improvement in knowledge.50,51,54,56,57,60,61  Two studies investigated the effectiveness of a video (narrated presentations) and concluded positive improvements in knowledge.49,59  Finally, 2 studies investigated the effectiveness of social media campaigns.52,55  One study reported that a social media campaign was not associated with consistent improvements in knowledge,55  and the other found that traditional methods (print, email, and internet-based education) followed by a social media campaign were associated with improvement in knowledge following instruction with the traditional methods but not following the social media campaign.52 

DISCUSSION

Summary of Evidence

Our systematic review aimed to examine the best available evidence on the effectiveness of technology-based learning tools designed to improve knowledge of CPGs by health care providers. We found preliminary evidence for the use of spaced education in combination with a game or case studies; however, because this intervention was not compared to a control intervention, the effect of the benefit cannot be accurately determined. Second, online module-based education may be effective in improving knowledge; however, preliminary evidence suggests that this intervention may not be superior to paper-based self-learning or in-person workshops.

The sensitivity analysis determined that the results of online module-based educational interventions did not differ across methodological quality. This analysis also provided preliminary information about the possible effectiveness of electronic guidelines with and without in-person workshops, short videos, and social media campaigns. These results could be used to generate hypotheses for future studies.

Additionally, considerable gaps in the quality of this literature were apparent throughout the conduct of this review. This limitation has led to a minimal quantity of quality literature on which to draw conclusions. We recommended that future research in the area address this concern.

Theories of Learning

The constructivist theory of learning influenced our main objective of this review and the overall larger study. We considered that learners' previous knowledge and experiences might play a significant role in the uptake of new information. Therefore, health care providers (the learners) will have some previous knowledge and clinical experiences of CPGs on which they can build. Further, constructivist-style learning supports the construction of knowledge rather than learners regurgitating a series of facts.21  If we want health care professionals to retain and engage in using the information from a guideline into clinical practice, the construction of this knowledge is paramount.

Informing KT

KT is an essential component of health research. It is imperative when developing KT strategies to understand end users and their needs, barriers, and preferences so that the activity/intervention can be tailored to their unique needs and preferences.

The previous systematic review by De Angelis et al30  provides insight into the use of technology-based educational interventions that may be beneficial in improving the perception of usability and practice behavior changes about CPGs. Our systematic review builds on this information and provides new evidence to further support the use of technology-based educational interventions.

Strengths and Limitations

This study has several strengths. We implemented a rigorous search strategy that was developed with an experienced health sciences librarian to help minimize errors. All reviewers were trained to screen and critically appraise to minimize error and bias. We eliminated studies of low quality to minimize the risk of bias; however, we also conducted a sensitivity analysis to understand the possible impact of misclassification bias on our results.

Some limitations are present in this study. We limited our search to studies published in the English language, and this may have excluded relevant studies; however, this is an unlikely source of bias.6266  In addition, the critical appraisal process involves scientific judgment varying between reviewers. However, this methodology is widely used in systematic reviews minimized by training reviewers a priori.6769  Our review is limited to the quality of the outcome measurements used in the low risk of bias studies. We restricted our review to studies that assessed knowledge following the use of a technology-based learning tool. We did not include studies assessing other measures, such as behavioral change and clinical outcomes. While we recognize that a change in knowledge does not guarantee an eventual implementation of a new practice, a change in knowledge is an important antecedent of behavior change and is typically needed if the implementation of a new practice is expected.70  Finally, due to the limited number of articles included in this review, our findings may not be generalizable.

CONCLUSION

Health care providers need to remain current with CPGs; however, the use of CPGs in clinical practice is suboptimal. A learning tool that incorporates findings from this systematic review stands to improve the use of CPGs. We have summarized the best available literature on educational interventions and their effectiveness in improving knowledge about CPGs in health care providers. This evidence will be used to inform the development of a novel technology-based educational intervention used to disseminate a CPG to chiropractic teaching faculty. This review may also be used to inform the development of other technology-based education KT interventions.

ACKNOWLEDGMENTS

The authors would like to acknowledge the invaluable contributions to this review from Ms Kristi Randhawa, Ms Kathleen Smith, Dr Hainan Yu, Dr Heather Shearer Dr. Jessica Wong, Dr Mana Rezai, and Ms Nancy Fynn-Sackey.

FUNDING AND CONFLICTS OF INTEREST

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The authors have no conflicts of interest to declare relevant to this work.

REFERENCES

1. 
Ellery
J,
McDermott
RJ,
Ellery
PJ.
Computers as a formal continuing education tool: Moving beyond intention
.
Am J Health Behav
.
2007
;
31
(3)
:
312
322
.
2. 
Murad
MH,
Coto-Yglesias
F,
Varkey
P,
Prokop
LJ,
Murad
AL.
The effectiveness of self-directed learning in health professions education: a systematic review
.
Med Educ
.
2010
;
44
(11)
:
1057
1068
.
3. 
Juanes
JA,
Ruisoto
P.
Computer applications in health science education
.
J Med Syst
.
2015
;
39
(9)
:
97
.
4. 
Sinclair
PM,
Kable
A,
Levett-Jones
T,
Booth
D.
The effectiveness of Internet-based e-learning on clinician behavior and patient outcomes: a systematic review
.
Int J Nurs Stud
.
2016
;
57
:
70
81
.
5. 
Brown
LC,
Johnson
JA,
Majumdar
SR,
Tsuyuki
RT,
McAlister
FA.
Evidence of suboptimal management of cardiovascular risk in patients with type 2 diabetes mellitus and symptomatic atherosclerosis
.
CMAJ
.
2004
;
171
(10)
:
1189
1192
.
6. 
FitzGerald
JM,
Boulet
LP,
McIvor
RA,
Zimmerman
S,
Chapman
KR.
Asthma control in Canada remains suboptimal: the Reality of Asthma Control (TRAC) study
.
Can Respir J
.
2006
;
13
(5)
:
253
259
.
7. 
McKenzie
JE,
O'Connor
DA,
Page
MJ,
et al.
Improving the care for people with acute low-back pain by allied health professionals (the ALIGN trial): a cluster randomised trial protocol
.
Implement Sci
.
2010
;
5
:
86
.
8. 
Walker
BF,
French
SD,
Page
MJ,
et al.
Management of people with acute low-back pain: a survey of Australian chiropractors
.
Chiropr Man Therap
.
2011
;
19
(1)
:
29
.
9. 
Scurlock-Evans
L,
Upton
P,
Upton
D.
Evidence-based practice in physiotherapy: a systematic review of barriers, enablers and interventions
.
Physiotherapy
.
2014
;
100
(3)
:
208
219
.
10. 
Chan
S,
Clough
A.
A critical evaluation of evidence-based practice of physiotherapy in musculoskeletal medicine
.
Int Musculoskelet Med
.
2010
;
32
(4)
:
163
166
.
11. 
Alonso-Coello
P,
Irfan
A,
Sola
I,
et al.
The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies
.
Qual Saf Health Care
.
2010
;
19
(6)
:
e58
.
12. 
Mairs
K,
McNeil
H,
McLeod
J,
Prorok
JC,
Stolee
P.
Online strategies to facilitate health-related knowledge transfer: a systematic search and review
.
Health Info Libr J
.
2013
;
30
(4)
:
261
277
.
13. 
Grol
R,
Grimshaw
J.
From best evidence to best practice: effective implementation of change in patients' care
.
Lancet
.
2003
;
362
(9391)
:
1225
1230
.
14. 
Jones
CA,
Roop
SC,
Pohar
SL,
Albrecht
L,
Scott
SD.
Translating knowledge in rehabilitation: systematic review
.
Phys Ther
.
2015
;
95
(4)
:
663
677
.
15. 
Scott
SD,
Albrecht
L,
O'Leary
K,
et al.
Systematic review of knowledge translation strategies in the allied health professions
.
Implement Sci
.
2012
;
7
:
70
.
16. 
Menon
A,
Korner-Bitensky
N,
Kastner
M,
McKibbon
KA,
Straus
S.
Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review
.
J Rehabil Med
.
2009
;
41
(13)
:
1024
1032
.
17. 
Al Zoubi
FM,
Menon
A,
Mayo
NE,
Bussieres
AE.
The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review
.
BMC Health Serv Res
.
2018
;
18
(1)
:
435
.
18. 
Bussieres
AE,
Al Zoubi
F,
Stuber
K,
et al.
Evidence-based practice, research utilization, and knowledge translation in chiropractic: a scoping review
.
BMC Complement Altern Med
.
2016
;
16
:
216
.
19. 
Canadian Institutes of Health Research.
Knowledge translation
.
2019
.
20. 
Olusegun
S.
Constructivism learning theory: a paradigm for teaching and learning
.
IOSR J Res Method Educ
.
2015
;
5
(6)
:
66
70
.
21. 
Educational Broadcasting Association.
Workshop: constructivism as a paradigm for teaching and learning
.
2004
.
22. 
Ward
JP,
Gordon
J,
Field
MJ,
Lehmann
HP.
Communication and information technology in medical education
.
Lancet
.
2001
;
357
(9258)
:
792
796
.
23. 
Cook
DA.
Web-based learning: pros, cons and controversies
.
Clin Med (Lond)
.
2007
;
7
(1)
:
37
42
.
24. 
Lam-Antoniades
M,
Ratnapalan
S,
Tait
G.
Electronic continuing education in the health professions: an update on evidence from RCTs
.
J Contin Educ Health Prof
.
2009
;
29
(1)
:
44
51
.
25. 
Nelson
EA.
E-learning. A practical solution for training and tracking in patient-care settings
.
Nurs Adm Q
.
2003
;
27
(1)
:
29
32
.
26. 
Rohwer
A,
Young
T,
van Schalkwyk
S.
Effective or just practical? An evaluation of an online postgraduate module on evidence-based medicine (EBM)
.
BMC Med Educ
.
2013
;
13
:
77
.
27. 
Odom
SE,
Barnes
K,
Wicker
M.
Incorporating pedagogical principles in developing an online nursing research course
.
Comput Inform Nurs
.
2005
;
23
(3)
:
146
152
.
28. 
Grimshaw
JM,
Eccles
MP,
Lavis
JN,
Hill
SJ,
Squires
JE.
Knowledge translation of research findings
.
Implement Sci
.
2012
;
7
:
50
.
29. 
Koller
V,
Harvey
S,
Magnotta
M.
Technology-Based Learning Strategies
.
Washington, DC
:
US Department of Labor, Employment and Training Administration;
2006
.
30. 
De Angelis
G,
Davies
B,
King
J,
et al.
Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review
.
JMIR Med Educ
.
2016
;
2
(2)
:
e16
.
31. 
Clark
RC,
Mayer
RE.
E-Learning and the Science of Instruction. 4th ed
.
Hoboken, NJ
:
John Wiley & Sons;
2016
.
32. 
Harbour
R,
Miller
J.
A new system for grading recommendations in evidence based guidelines
.
BMJ
.
2001
;
323
(7308)
:
334
336
.
33. 
National Institutes of Health, National Heart Lung and Blood Institute.
Quality assessment tool for before-after (pre-post) studies with no control group
.
2014
.
34. 
Moher
D,
Linerati
A,
Tetzlaff
J,
Altman
DG,
the PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement
.
PLoS Med
.
2009
;
6
(7)
:
e1000097
.
35. 
Bell
DS,
Fonarow
GC,
Hays
RD,
Mangione
CM.
Self-study from web-based and printed guideline materials. A randomized, controlled trial among resident physicians
.
Ann Intern Med
.
2000
;
132
(12)
:
938
946
.
36. 
Fordis
M,
King
JE,
Ballantyne
CM,
et al.
Comparison of the instructional efficacy of internet-based CME with live interactive CME workshops: a randomized controlled trial
.
JAMA
.
2005
;
294
(9)
:
1043
1051
.
37. 
Kerfoot
BP,
Kearney
MC,
Connelly
D,
Ritchey
ML.
Interactive spaced education to assess and improve knowledge of clinical practice guidelines: a randomized controlled trial
.
Ann Surg
.
2009
;
249
(5)
:
744
749
.
38. 
Kerfoot
BP,
Baker
H.
An online spaced-education game for global continuing medical education: a randomized trial
.
Ann Surg
.
2012
;
256
(1)
:
33
38
.
39. 
Labeau
SO,
Rello
J,
Dimopoulos
G,
et al.
The value of E-learning for the prevention of healthcare-associated infections
.
Infect Control Hosp Epidemiol
.
2016
;
37
(9)
:
1052
1059
.
40. 
Rebbeck
T,
Macedo
L,
Paul
P,
Trevena
L,
Cameron
ID.
General practitioners' knowledge of whiplash guidelines improved with online education
.
Aust Health Rev
.
2013
;
37
(5)
:
688
694
.
41. 
Kerfoot
BP,
DeWolf
WC,
Masser
BA,
Church
PA,
Federman
DD.
Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial
.
Med Educ
.
2007
;
41
(1)
:
23
31
.
42. 
Rhode
H,
Mash
B.
The effect of an automated integrated management of childhood illness guideline on the training of professional nurses in the Western Cape, South Africa
.
S Afr Fam Pract
.
2015
;
57
(2)
:
100
105
.
43. 
Houwink
EJ,
van Teeffelen
SR,
Muijtjens
AM,
et al.
Sustained effects of online genetics education: a randomized controlled trial on oncogenetics
.
Eur J Hum Genet
.
2014
;
22
(3)
:
310
316
.
44. 
Wolpin
S,
Lee
JA,
Glenny
RW,
Wittkowsky
AK,
Wolf
FM,
Zierler
BK.
Evaluation of online training on the prevention of venous thromboembolism
.
Vasc Endovascular Surg
.
2011
;
45
(2)
:
146
156
.
45. 
Stewart
M,
Marshall
JN,
Ostbye
T,
et al.
Effectiveness of case-based on-line learning of evidence-based practice guidelines
.
Fam Med
.
2005
;
37
(2)
:
131
138
.
46. 
Cook
DA,
Dupras
DM,
Thompson
WG,
Pankratz
VS.
Web-based learning in residents' continuity clinics: a randomized, controlled trial
.
Acad Med
.
2005
;
80
(1)
:
90
97
.
47. 
Carson
JA,
Gillham
MB,
Kirk
LM,
Reddy
ST,
Battles
JB.
Enhancing self-efficacy and patient care with cardiovascular nutrition education
.
Am J Prev Med
.
2002
;
23
(4)
:
296
302
.
48. 
Bell
DS,
Harless
CE,
Higa
JK,
et al.
Knowledge retention after an online tutorial: a randomized educational experiment among resident physicians
.
J Gen Intern Med
.
2008
;
23
(8)
:
1164
1171
.
49. 
Goodman
HA,
Pacheco
CL,
Loescher
LJ.
An online intervention to enhance nurse practitioners' skin cancer knowledge, attitudes, and counseling behaviors: a pilot study
.
J Dermatol Nurses Assoc
.
2018
;
10
(1)
:
20
28
.
50. 
Delaney
MM,
Friedman
MI,
Dolansky
MA,
Fitzpatrick
JJ.
Impact of a sepsis educational program on nurse competence
.
J Contin Educ Nurs
.
2015
;
46
(4)
:
179
186
.
51. 
Slaughter
AL,
Frith
K,
O'Keefe
L,
Alexander
S,
Stoll
R.
Promoting best practices for managing acute low back pain in an occupational environment
.
Workplace Health Saf
.
2015
;
63
(9)
:
408
415
.
52. 
Narayanaswami
P,
Gronseth
G,
Dubinsky
R,
et al.
The impact of social media on dissemination and implementation of clinical practice guidelines: A longitudinal observational study
.
J Med Internet Res
.
2015
;
17
(8)
:
e193
.
53. 
Harris
JM,
Salasche
SJ,
Harris
RB.
Can internet-based continuing medical education improve physicians' skin cancer knowledge and skills?
J Gen Intern Med
.
2001
;
16
(1)
:
50
56
.
54. 
Sisson
SD,
Rastegar
D,
Rice
TN,
Prokopowicz
G,
Hughes
MT.
Physician familiarity with diagnosis and management of hypertension according to JNC 7 guidelines
.
J Clin Hypertens
.
2006
;
8
(5)
:
344
350
.
55. 
De Sousa
F,
Jackson
J,
Knight
R,
et al.
A social media intervention to improve hypoglycemia management at a multicenter hospital: a quality improvement pilot for clinical nurses
.
Contemp Nurse
.
2018
;
54
(1)
:
44
51
.
56. 
Robson
J.
Web-based learning strategies in combination with published guidelines to change practice of primary care professionals
.
Br J Gen Pract
.
2009
;
59
(559)
:
104
109
.
57. 
Ginis
MKA,
Tomasone
JR,
Welsford
M,
et al.
Online training improves paramedics' knowledge of autonomic dysreflexia management guidelines
.
Spinal Cord
.
2017
;
55
(2)
:
216
222
.
58. 
Hong
SJ,
Lee
E.
Effect of evidence-based postoperative pain guidelines via web for patients undergoing abdominal surgery in South Korea
.
Asian Nurs Res (Korean Soc Nurs Sci)
.
2014
;
8
(2)
:
135
142
.
59. 
Valerio
TD,
Heaton
K.
The effects of an online educational program on nurse practitioners' knowledge of obstructive sleep apnea in adults
.
J Am Assoc Nurse Pract
.
2014
;
26
(11)
:
603
611
.
60. 
Dy
SM,
Hughes
M,
Weiss
C,
Sisson
S.
Evaluation of a web-based palliative care pain management module for housestaff
.
J Pain Symptom Manag
.
2008
;
36
(6)
:
596
603
.
61. 
Harris
JM
Jr,
Salasche
SJ,
Harris
RB.
Using the Internet to teach melanoma management guidelines to primary care physicians
.
J Eval Clin Pract
.
1999
;
5
(2)
:
199
211
.
62. 
Juni
P,
Holenstein
F,
Sterne
J,
Bartlett
C,
Egger
M.
Direction and impact of language bias in meta-analyses of controlled trials: empirical study
.
Int J Epidemiol
.
2002
;
31
(1)
:
115
123
.
63. 
Moher
D,
Fortin
P,
Jadad
AR,
et al.
Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews
.
Lancet
.
1996
;
347
(8998)
:
363
366
.
64. 
Moher
D,
Pham
B,
Lawson
ML,
Klassen
TP.
The inclusion of reports of randomised trials published in languages other than English in systematic reviews
.
Health Technol Assess
.
2003
;
7
(41)
:
1
90
.
65. 
Sutton
AJ,
Duval
SJ,
Tweedie
RL,
Abrams
KR,
Jones
DR.
Empirical assessment of effect of publication bias on meta-analyses
.
BMJ
.
2000
;
320
(7249)
:
1574
1577
.
66. 
Morrison
A,
Polisena
J,
Husereau
D,
et al.
The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies
.
Int J Technol Assess Health Care
.
2012
;
28
(2)
:
138
144
.
67. 
Carroll
LJ,
Cassidy
JD,
Peloso
PM,
et al.
Methods for the best evidence synthesis on neck pain and its associated disorders: the Bone and Joint Decade 2000–2010 Task Force on Neck Pain and Its Associated Disorders
.
J Manipulative Physiol Ther
.
2009
;
32
(suppl 2)
:
S39
S45
.
68. 
Carroll
LJ,
Cassidy
JD,
Peloso
PM,
Garritty
C,
Giles-Smith
L.
Systematic search and review procedures: results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury
.
J Rehabil Med
.
2004
(suppl 43)
:
11
14
.
69. 
Côté
P,
Cassidy
JD,
Carroll
L,
Frank
JW,
Bombardier
C.
A systematic review of the prognosis of acute whiplash and a new conceptual framework to synthesize the literature
.
Spine
.
2001
;
26
(19)
:
E445
E458
.
70. 
Greenhalgh
T,
Robert
G,
Macfarlane
F,
Bate
P,
Kyriakidou
O.
Diffusion of innovations in service organizations: systematic review and recommendations
.
Milbank Q
.
2004
;
82
(4)
:
581
629
.

Author notes

Leslie Verville is a Research Manager - Knowledge Translation at the Centre for Disability Prevention and Rehabilitation at Ontario Tech University (2000 Simcoe Street N, Oshawa, Ontario, L1H 7K4, Canada; leslie.verville@uoit.ca). Pierre Côté is a professor on the Faculty of Health Sciences, director of the Centre for Disability Prevention and Rehabilitation, and Canada research chair in Disability Prevention and Rehabilitation at Ontario Tech University (2000 Simcoe Street N, Oshawa, Ontario, L1H 7K4, Canada; pierre.cote@uoit.ca). Diane Grondin is an associate professor in the Department of Research and Innovation at the Canadian Memorial Chiropractic College (6100 Leslie Street North, Toronto, Ontario, M2H 3J1, Canada; dgrondin@cmcc.ca). Silvano Mior is the director for research partnerships and health policy at the Canadian Memorial Chiropractic College and a scientist at the Centre for Disability Prevention and Rehabilitation at Ontario Tech University (6100 Leslie Street North, Toronto, Ontario, M2H 3J1, Canada; smior@cmcc.ca). Keshini Moodley is a master's degree student at the Centre for Disability Prevention and Rehabilitation at Ontario Tech University (2000 Simcoe Street N, Oshawa, Ontario, L1H 7K4, Canada; keshini.moodley@uoit.ca). Robin Kay is a professor on the Faculty of Education at Ontario Tech University (2000 Simcoe Street N, Oshawa, Ontario, L1H 7K4, Canada; robin.kay@uoit.ca). Anne Taylor-Vaisey is health sciences librarian for the Centre for Disability Prevention and Rehabilitation at Ontario Tech University (2000 Simcoe Street N, Oshawa, Ontario, L1H 7K4, Canada; Anne.TaylorVaisey@uoit.ca).

Concept development: LV, PC, RK. Design: LV, PC, RK. Supervision: LV, PC, RK. Data collection/processing: LV, PC, KM. Analysis/interpretation: LV, PC, DG, SM, RK. Literature search: ATV. Writing: LV, PC. Critical review: LV, PC, DG, SM, KM, RK, ATV.

Supplementary data