Abstract
Beginning a graduate medical education training program is associated with a steep learning curve for incoming residents.
To compare the efficacy and efficiency of live versus webcast formats for Institutional Orientation.
This 2-year non-blinded study, with a nonrandomized cohort, compares outcomes for trainees oriented Summer 2005 in a ‘‘live-lecture’’ format with trainees oriented Summer 2006 using a webcast format. Outcomes include posttest success, the time required, presentation quality and utility, and cost.
In 2005, 249 trainees attended the live orientation. Of the 211 who completed the posttest; 132 (63%) passed it within 3 attempts. Of the 241 trainees in 2006, 236 completed the posttest. Of these, 215 (91%) passed it within 3 attempts. Compared to the live-lecture cohort, the webcast cohort rated the posttest as more difficult. Despite performing better, significantly fewer trainees in the webcast cohort rated the posttest as “appropriate” (χ2 = 5 28.57, df 5 1, P , .001). There were no significant differences between the 2 groups on their perceptions of quality and utility of the presentations. While the first year cost of the webcast exceeded that of live lectures, the amortized cost was nearly identical to the live-lecture costs.
As corroborated by resident comments, the web-based approach was more effective because it provided trainees flexibility regarding when to study, options on how to view the material, and opportunities to review it if needed for mastery. We plan to continue using the webcast strategy, revising the content as needed.
Introduction
Beginning a graduate medical education (GME) training program is associated with a steep learning curve for incoming residents and fellows. The transition from medical school to a GME program or from one program to another requires acculturation to the new learning environment of the program and its sponsoring institution. For first-year residents transitioning from medical school, this includes the additional role change associated with employment.
Formal orientation programs for new employees are a proven tool in the business world, yet there is little literature on GME orientation. Articles on the use of this approach in GME are limited in scope1–3 or restricted to a single department.4–8 A few institutions have developed “basic training” for incoming interns, using simulation.9,10 Beyond these, we are not aware of descriptions of institutional GME orientation programs in the United States.
The ideal institutional orientation would be an effective, efficient introduction for trainees to their new work environment. Institutional content should (1) be consistent over various trainee start dates, (2) be reproducible, (3) utilize the best faculty, (4) provide enduring documentation of content, and (5) demonstrate knowledge acquisition.
Barriers to the ideal institutional orientation include variations in start dates, diversity in prior training, reproducibility of content, and time pressures. Unlike in college and medical school, there is frequently no single uniform start date for residents and fellows. Acquisition of visas or extended duration of prior training due to leave time or remediation make it impossible to for all residents and fellows to begin on a single start date. The diverse educational and personal backgrounds of residents necessitate adaptability in content and presentation. Organizing consistent live lectures from busy health system leaders is a challenge because of their time constraints and the difficulty of obtaining consistency over successive lectures, even from the same speaker. Moreover, programs frequently exert pressure to get trainees to work with patients immediately, frequently to ease the call schedule.
Orienting trainees with finite resources and limited time requires flexibility. Web-based training has been shown to be at least as effective as traditional teaching.11 It allows learner control over content, sequence, pace, time, and sometimes even media.12 It allows learning “anywhere, anytime.” Reported disadvantages of web-based training include (1) availability of infrastructure for computer connections and software, (2) variable quality, and (3) isolation of students from teacher and peers.13 In addition, excessive cognitive load that accompanies visual and auditory stimuli can be detrimental to the learning process.14
The Orientation
Each year, Duke University Hospital orients 270 to 300 residents and fellows entering one of more than 70 Accreditation Council for Graduate Medical Education (ACGME) and 60 non-ACGME training programs. Approximately 240 trainees start between late June and early July, and approximately 30 residents enter between August and January. Approximately 140 residents are making the transition from medical school to their first postgraduate training. The others have had some, if variable, prior GME training. Most come from other institutions and are unfamiliar with Duke. Since 2000, the Office of Graduate Medical Education has oriented residents to the institution—its physical space, legal requirements, safety issues, and culture. Each year the challenge has been to schedule presentations and activities 3 times each summer and once in January. In the past, live presentations from June or July were videotaped, edited, and converted to webcasts to allow individuals starting later to view the orientation content. In 2005, Duke developed a list of essential topics, many of which were drawn from the ACGME core competencies.15,16 Duke's GME educators collaboratively developed a posttest with the presenters to determine trainees' retention of key material. In 2006, the authors (K.A., G.M., and T.T.) contrasted the relative benefits of the live presentations versus webcasts. These webcasts replaced live presentations beginning in August 2005 (although the data from the small 2005 pilot are not included in the analyses reported here).
The webcast platform displays a video of the lecturer and slides. The user can opt to watch the content in multiple ways: (1) lecturer side by side with the slides, (2) the slides with a thumbnail image of the lecturer, (3) slides alone, or (4) slides plus thumbnail list of the slides. The user can select audio or no audio. A multimedia vendor created the platform and high-quality web presentations. Learners can move back and forth among slides and view the presentations in any order. The cohort that received live lectures had only one chance to interact with the content and was not given printed copies of the slides. In 2005, the posttest included 29 questions. In 2006, 3 questions were deleted and 17 questions were added for a total of 43 questions. Since 2006, the webcasts have been used for the majority of orientations for all groups.
Methods
This 2-year study is a nonblinded, nonrandomized cohort study comparing outcomes for trainees oriented in June and July 2005 in a “live-lecture” format (live-lecture cohort) with those for trainees oriented in June and July 2006 using an orientation webcast (webcast cohort). Outcomes include success at passing the posttest within 3 attempts, the amount of time required, participant perception of presentation quality and utility, and cost.
The questions used in this analysis were formulated after the first full year of webcast implementation. The Duke University School of Medicine Institutional Review Board exempted this study from informed consent. Participants include 249 trainees in June through July 2005 and 241 in June through July 2006. Table 1 shows the demographics of the 2 cohorts.
Evaluation of the Orientation
Data from the posttest and an anonymous evaluation survey were used to compare orientation outcomes from the live lecture versus the webcast, focusing on knowledge acquisition, posttest difficulty, presentation quality and utility, and suggestions for improvements. Outcomes include posttest passing rates, trainee assessment of posttest difficulty, trainee assessment of quality and utility of the presentations and workshops, and the cost of the training. Some of the data, particularly from the 2005 (live-lecture) cohort, are missing. The missing data from the posttests appears to be random. We do not believe that the missing data has biased the analysis.
Posttests
A total of 25 questions were identical for the posttests in 2005 and 2006. All posttests were completed using the same Internet platform. Trainees could not open the webcasts and posttest simultaneously. Participants were allowed 3 attempts to select the correct answers to all questions. Questions missed previously were re-presented to the trainee for a second attempt. Questions still missed were presented for a third try. If on the third try there were still questions missed, the trainee had to contact the associate director or the educator of the GME office via phone or e-mail to discuss the material, and, after confirming understanding, receive a “verbal pass.” By the completion of this fourth attempt, all the trainees passed the test. At the end of the posttest, the trainees were asked to indicate whether they thought the posttraining assessment was easy, appropriate, or difficult.
Time Required for Training
The time for the entire orientation was tallied by counting trainee live activities and lecture or required webcast time.
Trainee Quality and Utility Ratings
Trainees rated the quality and utility of each live or webcast session, whether sufficient time was devoted to each topic, and their perceptions of the overall orientation. All questions used a 5-point Likert scale, and trainees were asked for suggestions to improve the presentations.
Costs of Orientation
To determine the difference in cost between live presentations and webcasts, we calculated the costs of both formats. We included the cost of food, handouts, or other materials; token gifts to the presenters; and the webcast production costs.
Results
Posttest
In 2005, 249 trainees attended the live orientation. We have data from 211 who completed the 29-question posttest; 132 (63%) passed the test within 3 attempts. Of the 236 trainees in 2006 who participated in the webcast orientation, 215 (91%) passed the posttest within 3 attempts.
In 2005, the mean posttest score for first attempts was 93.2%, but only 9 trainees (4%) obtained a score of 100%. Results from the second attempt showed cumulatively that 57 trainees (27%) achieved a perfect score, and after the third attempt, 133 trainees (63%) had perfect scores. The remaining 78 (37%) received a verbal pass.
In 2006, 241 trainees attended the webcast orientation. Of these 236 completed the posttest. For the webcast cohort, the mean score on the posttest on the first attempt was 92.5%, and 33 trainees (14%) achieved a perfect score. After the second attempt, 118 trainees (50%) had achieved perfect scores. After 3 attempts, 215 trainees (91%) had perfect scores. Only 21 (9%) of the trainees received a verbal pass (table 2).
We computed a χ2 test to determine whether there was a significant difference in the proportion of 2005 (live-lecture) versus 2006 (webcast) trainees passing the posttest after 3 attempts (χ2 = 156.90, df = 1, P < .001). The 2006 trainees who participated in the webcast orientation significantly outperformed those who participated in the live 2005 presentations.
Perceptions of Posttest Difficulty
The response rate for the survey following the posttest was 63% (158 of 249) in 2005 and 95% (230 of 241) in 2006. The majority of the trainees judged the posttest to be “appropriate.” In the live-lecture cohort 128 (81%) rated the posttest as appropriate whereas in the webcast cohort, only 177 (77%) rated it as appropriate. We computed a χ2 test to determine whether there was a significant difference in the proportion of 2005 versus 2006 trainees rating the posttest as appropriate (χ2 = 28.57, df = 1, P < .001). The live-lecture cohort had a significantly higher proportion of trainees who rated the posttest as appropriate (table 3).
Trainee Quality and Utility Ratings
The trainees were asked to rate the quality and utility of the individual presentations on a 5-point scale with 5 being “excellent” and 1 being “poor.” Their averaged ratings for quality were 3.79 ± 0.28 and 3.94 ± 0.12 in 2005 and 2006, respectively. Their averaged ratings for utility were 3.77 ± 0.30 and 3.95 ± 0.12 in 2005 and 2006, respectively. We computed a χ2 test to determine whether there was a significant difference in the proportion of 2005 versus 2006 quality scores (χ2 = 0.23, df = 1, P = .63) and utility scores (χ2 = 0.54, df = 1, P = .46). There were no significant differences between the live-lecture and webcast cohorts on perceived quality or utility of the content (table 4).
Trainee Overall Evaluation
In 2005, 42 trainees provided free-text evaluations of the overall program. The majority were complaints that the training was too long (n = 11), the room was uncomfortable (n = 9), and the computer training could have been improved (n = 10). Two applauded the orientation.
In 2006, 69 trainees provided a free-text evaluation of the overall program. About a third commented that there was redundancy in the presentations and that the webcasts should be shortened. Ten complained that there was insufficient time to complete the webcasts. Ten trainees reported difficulty obtaining adequate Web access, indicating they would have preferred more interaction with people or that they would have liked to ask questions of the speakers. Ten trainees applauded the technology and expressed appreciation that they did not have to spend 2 days in a lecture hall. Six indicated they would have preferred having the schedule ahead of time so they could better plan their time.
Time Required for Training: Efficiency of Webcasts
Each year the webcast and live presentations have been improved by providing feedback to the presenters. Initially, the lectures consisted of 12.5 hours of lecture and 3.5 hours of other large-group sessions. This and hands-on computer training (2 hours) left little time for small-group sessions. In 2006, the webcasts were reduced to 11.9 hours of didactics. In addition there was a live welcome session (2.8 hours) and hands-on computer training (2 hours), shown in table 5. In 2006, a small-group session, “How to give educational feedback,” was added.
Costs of Orientation
The cost of orientation in 2005 was $29 520 ($119/person) without room charges. The majority of this cost was for food for 2.5 days. In 2006, the cost of orientation was $43 630 ($181/person). The majority of the cost in 2006 was due to the development of the webcast. Food costs in 2006 were about 3% of what they were in 2005. These charges do not cover faculty or staff costs.
Discussion
Institutional orientation introduces incoming trainees to their new environment. It complements specialty-specific orientation, presenting expectations and regulations of the institution and resources available for trainees' personal and professional development. It provides an opportunity for socializing with faculty, staff, and peers, including those from different specialties.
At Duke University Hospital, we developed a successful orientation in which 56% (in 2005) to 92% (in 2006) of trainees felt prepared for their new roles. Although some felt the orientation was long and covered a lot of content, both quality and utility of the sessions were rated highly in both cohorts. We know from our previous work that our trainees prefer a web-based format to a live format, and the lecture content of orientation is now almost entirely webcast. Each year we review the evaluations from trainees, and we review the content and posttest questions with the faculty, revising as necessary.
While the 2005 cohort had a slightly better mean score on first attempt, the 2006 trainees performed significantly better on the posttests based on attempts to criterion. This is an interesting result, especially since the 2005 cohort comprised 47% fellows and the 2006 cohort comprised only 41% fellows. We would assume that fellows, who have already finished residency, would be more successful in passing the test than residents. In addition, there were more posttest questions in 2006 compared to 2005, which also should have made it more difficult for the 2006 cohort. However, our data are consistent with other studies17 which demonstrate superior posttest scores with web-based learning.
In 2007 and 2008, because we only had to retape 4 to 6 presentations and update the database, the cost decreased from $43 630 to $22 350 per year. By 2007, we were able to reduce the webcast content from 12 hours to 8 hours. With less time needed for the webcasts, we added more small-group sessions to increase interaction and specific skill development. By 2008, the amortized cost was $122 per person, essentially the same as the cost of the live orientation in 2005. If only a few presentations need to be “redone” each year, the amortized cost will continue to decline. Additionally, the webcasts and posttests provide an enduring product for documentation for regulatory agencies such as the Joint Commission. We were gratified when during an unscheduled regulatory visit, the site visitors queried specifically what 3 of our house staff had been taught concerning the Emergency Medical Treatment and Active Labor Act. The site visitor was shown the specific pertinent content and the passing grades for those trainees in the posttest questions, which contained the key points from the presentation. The webcast also provides consistency for orienting trainees who begin off-cycle.
As corroborated by resident comments, the web-based approach was more effective because it provided trainees with flexibility in when to study the material, options on how to view the material, and opportunities to review it if needed in order to master it. It also allowed flexibility to incoming residents, who are frequently busy with personal activities such as moving, obtaining a driver's license, registering to vote, securing schools for children, and finding employment for significant others. The Web format allowed trainees to choose a time and location most convenient to them. When we converted to a webcast format, we recognized that we had previously only counted “seat time” in a lecture hall as the outcome of our orientation. That we now require successful completion of a posttest ensures key concepts were learned.
We plan to continue using the webcast strategy with yearly revisions of content as needed. In 2008, we added a podcast option in addition to the webcast option. As we shorten the didactic portion, we hope to add more opportunities for hands-on learning; social activities to facilitate interaction with peers, faculty, and staff, within and across programs; and more “unscheduled time.” We will continue to explore alternate strategies (DVDs, virtual training, social networking), believing we need to increasingly match the preferred learning styles of our increasingly diverse and increasingly “millennial” trainees.
References
Author notes
Kathryn M. Andolsek, MD, MPH, is Professor of Community and Family Medicine at Duke University School of Medicine and Associate Director, Graduate Medical Education at Duke University Hospital; Gwen Murphy, PhD, RD, is Assistant Consulting Professor at the Division of Community Health, Community and Family Medicine, and Specialist Graduate Medical Education, Duke University Hospital; Sandro Pinheiro, PhD, is Assistant Professor at the Department of Medicine, Center for the Study of Aging and Human Development, Duke University Medical Center; Emil Petrusa, PhD, is Professor of Medical Education and Administration, Director of the Office for Teaching and Learning in Medicine, and Director of the Center for Outcomes Research in Education at Vanderbilt University School of Medicine; Tammy Tuck, BA, is Chief of Staff at the Graduate Medical Education Office, Duke University Hospital; John Weinerth, MD, is Designated Institucional Oficial, Associate Dean, and Director of Graduate Medical Education, Duke University Hospital.