Objective

The purpose of this study was to evaluate the perceived effectiveness and learning potential of 3 Web-based educational methods in a postgraduate radiology setting.

Methods

Three chiropractic radiology faculty from diverse geographic locations led mini-courses using asynchronous discussion boards, synchronous Web conferencing, and asynchronous voice-over case presentations formatted for Web viewing. At the conclusion of each course, participants filled out a 14-question survey (using a 5-point Likert scale) designed to evaluate the effectiveness of each method in achieving specified course objectives and goals and their satisfaction when considering the learning potential of each method. The mean, standard deviation, and percentage agreements were tabulated.

Results

Twenty, 15, and 10 participants completed the discussion board, Web conferencing, and case presentation surveys, respectively. All educational methods demonstrated a high level of agreement regarding the course objective (total mean rating >4.1). The case presentations had the highest overall rating for achieving the course goals; however, all but one method still had total mean ratings >4.0 and overall agreement levels of 70%–100%. The strongest potential for interactive learning was found with Web conferencing and discussion boards, while case presentations rated very low in this regard.

Conclusions

The perceived effectiveness in achieving the course objective and goals was high for each method. Residency-based distance education may be a beneficial adjunct to current methods of training, allowing for international collaboration. When considering all aspects tested, there does not appear to be a clear advantage to any one method. Utilizing various methods may be most appropriate.

Chiropractic radiology residency training has traditionally been performed through full-time programs affiliated with chiropractic colleges or universities.1  In addition to clinical, research, and teaching responsibilities, many programs utilize face-to-face didactic educational conferences and case-based laboratory sessions to guide residents in training. While these sessions are an important component of residents' education, delivering them efficiently can be a challenge for both faculty and residents, especially if they are located at different sites. In addition, some programs have too few radiology residents to generate helpful peer-to-peer discussions or too few faculty members available to lead these sessions. Distance learning, or e-learning, has been postulated as a useful complement to traditional teaching methods in professional residency programs,2  and there appears to be a high level of acceptance using Internet-based sessions for learning activities.3  With that in mind, a collaborative approach between the chiropractic radiology residency programs using e-learning technologies has been proposed to resolve these challenges.

Multiple studies have demonstrated the efficacy of computer-mediated teaching methods in medical residencies based in the United States,46  in undergraduate programs, or in continuing medical education,710  and in several other countries.2,3,11  These latter studies have implications for programs in remote regions of the world or where access to experts in specific specialty areas is limited.

However, there are still gaps in our knowledge: No previously reported studies have evaluated the effectiveness of postgraduate distance learning in chiropractic radiology residencies; there is still a need to investigate learning effectiveness between different Internet-based distance education formats;11  and no consensus has been reached on the optimum format for distance learning models within the various areas of professional residency education.12 

The primary purpose of this prospective, controlled pilot study was to evaluate the perceived effectiveness of 3 different Web-based educational delivery methods by both radiology residents (students) and chiropractic radiologists (observers) in achieving the course objective and goals. Additionally, the potential for educational interactions, the perceived learning potential for each method, and key logistical issues were evaluated.

After obtaining approval from the New York Chiropractic College institutional review board to conduct this study, we invited the chiropractic radiology community via e-mail to take part. Specifically, invitations were sent to the members of the American Chiropractic College of Radiology (ACCR) and the members of the academic committee of the ACCR, asking the latter to forward the message to their radiology residents. Volunteer respondents who agreed to participate in the study were divided in two groups: residents (students) and radiologists (observers). No other inclusion or exclusion criteria were considered.

During a 6-week period, 3 experienced chiropractic radiology faculty members from different geographic locations led 3 diverse radiology mini-courses using different distance learning methods. The courses, developed by each instructor, concentrated mainly on the diagnostic evaluation of radiology cases, and they were presented using a Web-based format. All courses shared the same objective and goals (Figure 1). Instructors received appropriate training by personnel of the host college to use their respective computer-mediated Web technologies. For each session, the residents were asked to be the main participants while the radiologists would observe the course without prompting responses or helping residents. The inclusion of radiologists as observers in the design of the research allowed for the comparison of data from the students' and the radiologists' perspective.

Figure 1.

Course objective and goals.

Figure 1.

Course objective and goals.

The first instructor, located in Australia, directed a 4-week course using an asynchronous discussion board inside the learning management system (LMS) Desire2Learn (D2L, Desire2Learn Inc, Kitchener, ON, Canada). A total of 7 forums were created based on 7 imaging cases, during which the facilitator would ask the residents individual and group questions to promote a student-centered learning environment. The second instructor, located in the United States, directed a 4-week course using a synchronous (live) Web conferencing tool (GoToWebinar, Citrix Online, Goleta, CA). This online application allowed the instructor to facilitate live discussions of radiology cases by the participants who could visualize the computer screen of the facilitator and, using a microphone, communicate verbally with all other participants. Invitations to attend the live session were sent to the participants by e-mail a few days before the conference. A total of 4 sessions lasting between 1.5 to 2 hours were held at 1-week intervals. The third instructor, located in France, directed a series of 4 asynchronous case presentations using voice-over PowerPoint (Microsoft Office 2007, Microsoft Corp, Redmond, WA). The presentations were converted to a Flash (Adobe Systems Inc, San Jose, CA) format for Web viewing. Each case presentation lasted approximately 20 minutes and demonstrated a series of skills used to analyze imaging findings and reach a correct differential diagnosis or final diagnosis. A short didactic explanation of the conditions followed each case presentation. This course followed a self-directed format. It was not scheduled so that participants could start and finish at any time. No points/scores were tabulated for any of the 3 courses.

At the conclusion of each course, the participants were sent an electronic link to an evaluation survey comprising 14 qualitative questions (Figure 2). The answers were rated along a 5-point Likert scale (1 = very ineffective or very low to 5 = very effective or very high). Each question also displayed an option of “nonapplicable” or “don't know” as a potential answer. The online survey instrument application used was Zoomerang (MarketTools, San Francisco, CA). The survey was designed to evaluate several areas, including the perceived effectiveness of the teaching and course delivery methods in achieving the course objective (question 1) and goals (questions 2–7) and the satisfaction rate for the use of the different educational methods when considering the learning potential of each session (questions 9–12). Additional questions were inserted to evaluate the level of participation (question 8), the ease of use of the Web applications, and the willingness of the participants to use these technologies in future distance education courses (questions 13, 14). An invitation to write short open-ended comments was included at the end of the survey.

Figure 2.

Survey questions 1–14 for students and observers.

Figure 2.

Survey questions 1–14 for students and observers.

The survey was developed by consensus agreement between 2 of the authors. One had over 5 years of experience as a diagnostic imaging residency director, while the other was a full-time chiropractic college educator with multiple years of course delivery experience in online education. The survey was sent to 1 additional full-time diagnostic imaging faculty member at a chiropractic college for review. The comments provided by the reviewer were used to edit the survey. The survey responses were tabulated by a designated research person in the Office of Institutional Quality Assurance in order to maintain anonymity of the respondents.

Survey responses were separated in different categories: observers, students, and total (combined responses). Measures of mean and standard deviation were tabulated. Responses of 4 or 5 on the scale were considered “positive” for analysis. Because of the small number of participants overall, statistical comparisons between methods were not undertaken.

A total of 46 individuals (14 students, 32 observers) demonstrated an interest in participating in the pilot study. Thirteen students were located in the United States and enrolled in 6 different programs. Another student was located in New Zealand. All but 4 of the observers were located in the United States, with 2 located in Canada, 1 in England, and 1 in Australia. The inclusion of these latter participants (and the international instructors) allowed the researchers to consider the logistical issues of an international audience as part of the project.

Thirty-one participants (14 students and 17 observers) took part in the asynchronous online discussion board. Nineteen (13 students and 6 observers) participated in 1 or more synchronous Web conferences. The level of participation for the asynchronous voice-over PowerPoint presentations (number of downloads) was not made available. The combined results for the 3 different distance-learning education methods are displayed in Table 1. Twenty participants (14 students, 6 observers), 15 participants (11 students, 4 observers), and 10 participants (8 students, 2 observers) completed the discussion board, Web conferencing, and PowerPoint presentations surveys, respectively. One observer did not rate question 12 of the PowerPoint presentations survey, and one student did not rate question 7 of the Web conferencing survey. Although all 14 participants in the asynchronous discussion sessions filled out the survey, only a small number of these students fully participated during the whole 4-week period, with some students participating in only 1 case; however, all students would have been able to continue to observe any of the sessions without directly participating.

Table 1.

Participant Learning Method Ratings (With Percentage Agreement) per Survey Question*

Participant Learning Method Ratings (With Percentage Agreement) per Survey Question*
Participant Learning Method Ratings (With Percentage Agreement) per Survey Question*

All educational methods demonstrated a high level of agreement for effectively achieving the course/session objective. The highest mean (4.45 for total participants) was seen in the asynchronous discussion board method, with overall agreement ranging from 86% to 100%.

Each method had a total mean rating of >4.0 for questions 2–7 (exploring the effectiveness of the educational methods in reaching the goals of the course), with one exception: question 6 for Web conferencing rated 3.93 overall. Overall agreement levels ranged from 70% to 100%. The lowest agreement percentage was for question 7 for the asynchronous discussion board, and the highest agreement percentage was for questions 2–4 and 7 for the PowerPoint presentations. The asynchronous discussion board method trended lower than the other 2 methods in regard to the positive ratings across most goals.

The lowest total ratings across all methods were found in question 8, which assessed each participant's level of perceived participation for his or her activities. These results were expected since, as observers, the radiologists generally rated their participation level as low, negatively influencing the total ratings. In addition, given their busy schedule, the residents were not able to participate in all sessions, also negatively influencing their perceived level of participation. Each method had a perceived level of student participation rating <4.0, except for the Web conferencing, which rated 4.0. The synchronous nature of the Web conferencing method required participation from all students present, potentially influencing in a positive way the student's participation rating.

Questions 9–11 looked at the potential for interactive learning opportunities, with the strongest potential coming from the synchronous Web conferencing, followed closely by the asynchronous discussion boards. PowerPoint case presentations were rated very low in this regard, particularly for question 10 (<3.0 mean ratings for each group and <50% positive rating overall), which would be expected as no direct interpersonal interactions took place with this method.

When looking at learning potential versus time commitment (question 12), the asynchronous discussion board rated lowest, while the other two methods rated very high. This was anticipated, as the discussion board required regular logging in to D2L over a 4-week period, tracking down any new discussions or cases, researching answers to various questions, and typing all responses into the system. For ease of use, there was a low agreement rating by observers for the webinar method (25%), but this did not appear to affect their willingness to use this method (75%). Finally, the open-ended comments collected from the respondents did not provide any additional information related to the purposes of the research beyond the data collected in the surveys.

Distance learning, or distance education, is defined as “a separation in time and/or space between the learner and the instructor.”5  Recent advances in Web technologies and applications have led to an increase in the popularity of distance learning programs in health care education. Technical, logistical, and cost-related limitations still exist (particularly in international collaborations),11,13,14  but even so, the specialty of radiology, because it is largely image based, seems well positioned to transfer a portion of its course delivery to an online format.

The most commonly utilized methods of distance learning include asynchronous and/or synchronous presentations of course material and discussions that require the application of online educational best practices.5,6  There are challenges and benefits to both methods for online content delivery. Asynchronous methods provide learners with the opportunity to participate in learning activities when it is convenient for their schedule, while synchronous delivery can be challenging when collaborating with numerous individuals and accounting for time zone teaching.15,16 

A systematic review of Internet-based continuing medical education by Cook et al found that Internet-based learning is educationally beneficial and can achieve results similar to those of traditional instructional methods.17  It has demonstrated favorable outcomes across a wide variety of learners, learning contexts, clinical topics, and learning outcomes.9  Our results support the observations made by other studies that evaluated the utilization of asynchronous discussion boards inside an LMS,6,8,9  synchronous Web conferencing,4,5  and asynchronous computer-based presentations and courses.7,10  The use of distance learning methods in residency programs can be seen as a beneficial educational adjunct to the current face-to-face training. As trends change in the delivery methods of health care education to meet the demands of the millennial student, online education is expected to become more widely used.18 

In this study, all of the teaching methods used were generally considered to be effective by all groups of participants. The PowerPoint presentation was rated as having the highest level of effectiveness across the broadest number of questions (7 total, mostly related to the objective and goals); however, the success of this method was offset by having the lowest ratings for questions relating to the ability to interact.

Observers generally tended to have a higher level of agreement than did the students across the various survey questions, which may be due in part to the smaller number of observers that responded to the survey, the perspective they bring when considering the process as a teacher, and/or the amount of time spent observing these activities.

Prominent differences in ratings between methods (i.e., >30% difference in positive rating between at least 2 of the methods) were noted by at least 1 subgroup in 8 of the survey questions (4, 7–13). While this did not trend in favor of any one method, participants ultimately indicated that they were most willing to use the PowerPoint method. It may also be noted that for the asynchronous discussion board, the autonomy of access to sessions and its designed open-endedness may have worked against this method—particularly for students who are challenged with effective time-management or prioritization skills. Additionally, the design of the LMS made it difficult to find and follow the flow of discussions, which may have negatively affected the participants' learning experience. In the near future, many of the issues identified with LMS could be overcome by implementing an independent standardized technical system, such as a sharable content object reference model, which may also provide better functionality for teaching internationally.19 

Limitations

With any pilot study, there will be limitations in design and the application of outcomes. First, the use of mean survey scores as a main response indicator in any 5-point Likert ordinal scale in surveys has to be interpreted cautiously, especially with a small number of participants. In this study, the percentage of respondents who answered positively to a question (i.e., answered 4 or 5) was provided as an additional response indicator to supplement the mean scores.

In addition, the relatively small number of survey respondents, especially for the “observers” category, precluded generalization of the results to the entire chiropractic radiology community. Some students participated across all 3 courses, while others were only involved in 1 or 2. While this had the advantage of providing a reference point for comparison between teaching methods for some students, it also had the potential to allow an upward or downward rating bias compared to those students who participated in only 1 course. The unequal number of surveys filled out between groups may have skewed the results (e.g., the lower the number of participants, the greater the chance for artificially high—or low—ratings and agreement). A larger study would allow for less impact from participant drop out and the ability to statistically compare rating scores. The time required for participation in the courses was more than some participants could allow within their respective workload. Increasing the allowed study time period may have improved the participation level. Lastly, the study would have benefited from the collection of supplemental data, including the seniority level of the residents and the number of viewings (downloads) of the PowerPoint presentations.

In the distance education setting, interaction with educators or other students during or after a learning session (in any form) is perceived by residents as better than no interaction, with more immediate, direct, or integrated interactions appearing to provide the highest satisfaction and most effective outcomes for achieving learning goals.13,14,20,21  Prompt feedback from the educator is also felt to be one of the most effective ways to reinforce learning behaviors and to encouraging reflection on mistakes.22,23  It is therefore interesting to note that in this study the learning method perceived by students as being most effective in achieving the stated goals was the method with the least amount of direct interaction, possibly owing to the design of the presentations and/or the ability of the students to review the information at their own pace or on multiple occasions.

Distance education can provide opportunities for residents to interact with experts in their field in virtually any part of the world, and interactions with residents in other programs is perceived as an enhancement of the overall learning experience3,13 ; however, there are technical, logistical, and cost-related limitations that still need to be overcome to provide a more seamless educational experience. The goal will be to find the proper balance between effective and efficient teaching methods in distance training opportunities.

In this pilot study, online distance education methods such as asynchronous discussion boards inside an LMS, synchronous Web conferencing, and asynchronous voice-over PowerPoint presentations all demonstrated a high level of perceived effectiveness in achieving the course objective and most of the goals. The majority of the participants positively rated the learning potential of the distance education methods and their willingness to use these methods in future residency-based diagnostic imaging training. When balancing the desire for student–teacher and student–student interaction against the perceived effectiveness of each method used, the results demonstrated that there does not appear to be a clear advantage in using any particular method. Further studies including a larger number of participants could prove beneficial to determine if the results obtained in this pilot evaluation could be generalized to the entire chiropractic diagnostic imaging residency community.

Funding for this research was obtained through a seed faculty grant program from New York Chiropractic College, a nonprofit institution of higher education.

CONFLICT OF INTEREST

The authors have no conflict of interest to declare.

1
Mestan
MA
,
Taylor
JA
,
Blackshaw
GL
,
McDonald
JC
.
Commentary: establishing an accredited master of science in diagnostic imaging degree at a chiropractic college
.
J Manipulative Physiol Ther
.
2006
;
29
:
410
413
.
2
Carriero
A
,
Beomonte Zobel
B
,
Bonomo
L
,
et al
.
E-learning in radiology: Italian multicentre experience
.
Radiol Med
.
2011
;
116
:
989
999
.
3
Obura
T
,
Brant
WE
,
Miller
F
,
Parboosingh
IJ
.
Participating in a community of learners enhances resident perceptions of learning in an e-mentoring program: proof of concept
.
BMC Med Educ
.
2011
;
11
(
3
):
1
6
.
4
Markova
T
,
Roth
LM
,
Monsur
J
.
Synchronous distance learning as an effective and feasible method for delivering residency didactics
.
Fam Med
.
2005
;
37
(
8
):
570
755
.
5
Jeun
BS
,
Javan
R
,
Gay
SB
,
Olazagasti
JM
,
Bassignani
MJ
.
An inexpensive distance learning solution for delivering high-quality live broadcasts
.
Radiographics
.
2008
;
278
:
1251
1258
.
6
Sparacia
G
,
Cannizzaro
F
,
D'Alessandro
DM
,
D'Alesssandro
MP
,
Caruso
G
,
Lagalla
R
.
Initial experiences in radiology e-learning
.
Radiographics
.
2007
;
27
:
573
581
.
7
Burnette
K
,
Ramundo
M
,
Stevenson
M
,
Beeson
MS
.
Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students
.
Acad Emerg Med
.
2009
;
16
:
46
50
.
8
Howlett
D
,
Vincent
T
,
Watson
G
,
et al
.
Blending online techniques with traditional face to face teaching methods to deliver final year undergraduate radiology learning content
.
Eur J Radiol
.
2011
;
78
(
3
):
334
341
.
9
Curran
VR
,
Fleet
LJ
,
Kirby
F
.
A comparative evaluation of the effect of internet-based CME delivery format on satisfaction, knowledge and confidence
BMC Med Educ
.
2010
;
10
(
10
):
1
7
.
10
Davis
J
,
Crabb
S
,
Rogers
E
,
Zamora
J
,
Khan
K
.
Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomized controlled trial
.
Med Teach
.
2008
;
30
(
23
):
302
307
.
11
Pelayo
M
,
Cebrian
D
,
Areosa
A
,
Agra
Y
,
Izquierdo
JV
,
Buendia
F
.
Effects of online palliative care training on knowledge, attitude and satisfaction of primary care physicians
.
BMC Fam Pract
.
2011
;
12
:
37
47
.
12
Reiter
DA
,
Lakoff
DJ
,
Trueger
NS
,
Shah
KH
.
Individual interactive instruction: an innovative enhancement to resident education
.
Ann Emerg Med
.
2013
;
61
:
110
113
.
13
Bednar
ED
,
Hannum
WM
,
Firestone
A
,
Silveira
AM
,
Cox
TD
,
Proffit
WR
.
Application of distance learning to interactive seminar instruction in orthodontic residency programs
.
Am J Orthod Dentofacial Orthop
.
2007
;
132
:
586
594
.
14
Miller
KT
,
Hannum
WM
,
Proffit
WR
.
Recorded interactive seminars and follow-up discussions as an effective method for distance learning
.
Am J Orthod Dentofacial Orthop
.
2011
;
139
:
412
416
.
15
Vonderwall
S
,
Liang
X
,
Alderman
K
.
Asynchronous discussions and assessment in online learning
.
J Res Technol Educ
.
2007
;
39
(
3
):
309
323
.
16
Andresen
MA
.
Asynchronous discussion forums, success, failures, outcomes, assessments and limitations
.
Educ Technol Soc
.
2009
;
12
(
1
):
249
257
.
17
Cook
DA
,
Levinson
AJ
,
Garside
S
,
Dupras
DM
,
Erwin
PJ
,
Montori
VM
.
Internet based learning in the health professions
.
JAMA
.
2008
;
300
(
10
):
1181
1196
.
18
Borges
NJ
,
Manuel
S
,
Elam
CL
,
Jones
BJ
.
Comparing millennial and generation X medical students at one medical school
.
Acad Med
.
2006
;
81
(
6
):
571
576
.
19
Njuguna
N
,
Flanders
AE
,
Kahn
CE
Jr.
Envisioning the future of e-learning in radiology: an introduction to SCORM
.
Radiographics
.
2011
;
31
:
1173
1179
.
20
Klein
KP
,
Hannum
WM
,
Koroluk
LD
,
Proffit
WR
.
Interactive distance learning for orthodontic residents: utilization and acceptability
.
Am J Orthod Dentofacial Orthop
.
2012
;
141
:
378
385
.
21
Zhang
D
.
Interactive multimedia-based e-learning: a study of effectiveness
.
Am J Distance Educ
.
2005
;
19
(
3
):
149
162
.
22
Arogundade
RA
.
Adult learning principles for effective teaching in radiology programmes: a review of the literature
.
WAJM
.
2011
;
30
(
1
):
3
10
.
23
Williamson
KB
,
Gunderman
RB
,
Cohen
MD
,
Brank
MS
.
Learning theory in radiology education
.
Radiology
.
2004
;
233
:
15
18
.

Author notes

Jean-Nicolas Poirier is dean of academic programs and services at New York Chiropractic College in Seneca Falls, New York. Jeffrey Cooley is an associate professor in the School of Health Professions at Murdoch University in Western Australia. Michelle Wessely is a clinical radiologist in St Paul, Minnesota. Gary Guebert is an assistant professor at Logan University in Chesterfield, Missouri. Kristina Petrocco-Napuli is an associate professor at New York Chiropractic College. Address correspondence to Jean-Nicolas Poirier, Dean of Academic Programs and Services, New York Chiropractic College, Seneca Falls, NY 13148; npoirier@nycc.edu. This article was received October 4, 2013, revised January 15, 2014, and accepted January 24, 2014.