Context.—

Developing skills related to use of computer-based tools is critical for practicing genomic pathology. However, given the relative novelty of genomics education, residency programs may lack faculty members with adequate expertise and/or time to implement training. A virtual team-based learning (TBL) environment would make genomic pathology education available to more trainees.

Objective.—

To translate an extensively implemented in-person TBL genomic pathology workshop into a virtual environment and to evaluate both knowledge and skill acquisition.

Design.—

Using a novel interactive simulation approach, online modules were developed translating aspects of the TBL experience into the virtual environment with a goal of acquisition of necessary computer-related skills. The modules were evaluated at 10 postgraduate pathology training programs using a pre-post test design with participants deidentified. A postmodule anonymous survey obtained participant feedback on module quality and efficacy.

Results.—

There were 147 trainees who received an email request to voluntarily participate in the study. Of these, 43 trainees completed the pretest and 15 (35%) subsequently completed the posttest. Mean overall scores were 45% on the pretest compared with 70% on the posttest (P < .001; effect size = 1.4). Posttest improvement of results was similar for questions testing acquisition of knowledge versus skills. Regarding the 19 participants who took the survey, 18 (95%) would recommend the modules to others and believed they met the stated objectives.

Conclusions.—

A simulation-based approach allows motivated pathology trainees to acquire computer-related skills for practicing genomic pathology. Future work can explore efficacy in a nonvoluntary setting and adaptation to different specialties, learners, and computer tools.

Pathologists must acquire genomic medicine–related knowledge and skills to provide the best possible patient care.1,2  Specifically, to adequately analyze genomic data, it is critical to develop proficiency in the use of computer-based genomics tools. These Web sites and programs facilitate a number of tasks, including genetic test development, interpretation of variants, and analysis of genomic data sets. Didactic formats do not provide practical hands-on instruction on how to use these tools.

There are accumulating data that a team-based learning (TBL) approach allows for better outcomes compared with traditional methods, especially with regard to skills-based objectives.3,4  To allow for an understanding of genomics and effective use of computer tools we developed a TBL genomics workshop for pathology residents at national meetings, which has been published previously.5  These sessions allow participants to work in teams as they discuss management of a breast cancer patient and use online tools to analyze genomic data and answer clinically relevant questions.

Developing and implementing a TBL session, however, is time-consuming and requires significant expertise. In addition, given the relative novelty of genomics education, many residency programs may lack faculty members with adequate background and/or time to implement similar sessions. Hence, creating a virtual TBL environment would enable a larger number of learners to take advantage of the benefits of this approach without the need for extensive local faculty input.

To better disseminate genomic pathology training, we translated our in-person workshop approach into a virtual format, using computer-based simulation, to accomplish both knowledge-based and skills-based objectives. In a multisite study involving pathology residents and fellows, we examined the efficacy of these modules using a pre-post test study design. We also surveyed participants on module quality in comparison with in-person TBL and lecture formats. This study demonstrates the benefits of a simulation-based approach for teaching genomic pathology that can be potentially applied to other computer-related skills in any specialty.

Study Design and Participants

Between March 2016 and July 2017, 10 postgraduate pathology training programs participated in the study (4 sites with residents and fellows and the remaining 6 with residents only). Participants had to have a previous or current molecular pathology rotation. Trainees were invited to voluntarily take the modules and participate in the study through a standardized email. To link the results from the pretests and posttests but preserve anonymity, participants were asked to provide the last 4 digits of their phone number at the beginning of each test. Prior to distribution to the program directors, all test results were paired with removal of the phone numbers by evaluation staff not affiliated with the training programs. The postmodule survey was anonymous. The study was deemed exempt from further review by the Institutional Review Boards at participating sites.

Team-Based Learning Workshop Development

The development of the genomic pathology workshop has been described previously.5  Briefly, a working group made up of experts in genetics, pathology, and medical education developed a series of 4 TBL exercises: single-gene testing; prognostic gene-panel testing; cancer gene panels; and whole-exome sequencing. The curriculum was designed for residents with some training in molecular pathology. Each exercise consisted of the following:

  1. Presession reading and preparation questions.

  2. An instructor-delivered 15- to 30-minute PowerPoint (Microsoft Corporation, Redmond, Washington) lecture reviewing answers to the preparation questions and relaying other content needed for the TBL activity.

  3. A 60-minute activity consisting of teams of 3 to 6 residents answering a series of questions, including those necessitating the use of online genomic tools.

  4. An instructor-delivered 15- to 30-minute PowerPoint lecture presenting answers to the questions and incorporating a discussion of team responses.

The workshops were extensively implemented, with 25 iterations across multiple national pathology meetings. Evaluation results from the first 3 workshops have been published.5  This learning model has also been adapted to other specialties.6 

Online Module Development

After introductory slides, links were provided for the preactivity lecture reading. For the preactivity and postactivity lectures, individuals who previously served as workshop faculty provided narration for slides that was captured directly through the Articulate (New York, New York) Storyline tool.7,8  Later edits made use of Audacity, an open-source recording and audio editing program. Interactive preparation questions (ie, selection of an answer with real-time feedback) were included in the preactivity lecture.

The activity portion of each module was developed with Articulate Storyline. In the beginning of each activity, learners were informed that they were part of a team, with pictures of their “teammates.” The learner was then asked to input the answers to each workshop question with the follow-up screen showing the answers of the teammates. Simulations of genomics tools were created through screen recording of workshop faculty using the tools. Interactions were added in which the learner was challenged to make a logistical decision or clinical insight. For example, the learner might be asked to click on an area or write an answer to a Web site query. If the wrong information was typed, the program notified the learner and asked for a correction.

The final component of each module was the postactivity PowerPoint lecture presenting answers to the activity questions. Although with the online modules it was not possible for the learner to have a discussion of their answers with the instructor, the learner's answers were available for viewing during the postactivity lecture to compare with the expert answers.

The modules were intended to be taken by single individuals, with the first online module piloted by 9 trainees who, using a survey, provided feedback. Based on these recommendations, all of the modules were modified. Subsequently, modules 2, 3, and 4 were piloted by 3 trainees, again providing survey-based feedback, to make final edits prior to study distribution. Technical development of each module, subcontracted to the American Society for Clinical Pathology, took approximately 115 hours at a cost of $10,500.

The full modules are available at no charge, after a registration process to allow tracking of use, at pathologylearning.org/trig. To provide an example, access to module 3 on cancer gene panels is provided.9 

Exam and Survey Design

A 19-question exam, consisting of both knowledge- and skills-based questions related to core workshop objectives, was developed by workshop faculty and individuals with expertise in evaluation. Skills-based questions asked participants to identify the appropriate Web site for a task and then, given a specific task, report results through real-time use of that Web site. Demographic questions were also included (eg, previous knowledge of online genomics tools). Two molecular pathologists familiar with the workshop curriculum reviewed the exam for clarity and to assess content validity. The exam was first piloted with a group of 3 residents with feedback obtained in a focus group format by an individual not known to the participants. After incorporating suggested changes, the exam was piloted at an in-person workshop. Demonstrating evidence of discriminative validity, there were 3 individuals taking the pretest, with an average score of 23% (range, 16%–26%) and 6 taking the posttest, with an average score of 60% (range, 37%–74%). The survey was based on one previously published by the authors for workshop evaluation.5 

Statistical Analysis

After assessing the data for normality, we compared pretest to posttest scores using paired t-tests. Descriptive statistics were used for survey results.

There were 10 pathology programs included in the study. A total of 111 residents and 36 fellows were sent an email requesting participation. Of these, 43 completed the pretest, with 15 (35%) having a matching posttest result. Of those completing the matched pretests and posttests, there was 1 postgraduate year 1 (PGY-1) resident, 3 PGY-2 residents, 4 PGY-3 residents, 4 PGY-4 residents, and 3 fellows. The average amount of self-reported training in molecular and genomic pathology was 3.5 weeks (range, 0.5–12 weeks) and 0.5 weeks (range, 0–3 weeks), respectively. Regarding prior knowledge of genomics Web sites, 6 of the 15 participants (40%) indicated they had previously used ClinVar (National Center for Biotechnology Information, Bethesda, Maryland), 2 (13%) had previously used PolyPhen (http://genetics.bwh.harvard.edu/pph2/, accessed June 8, 2018), 8 (53%) had previously used COSMIC (Wellcome Sanger Institute, London, United Kingdom), 5 (33%) had previously used mycancergenome.org (Vanderbilt University, Nashville, Tennessee), 7 (47%) had previously used OMIM (National Center for Biotechnology Information), and 7 (47%) had previously used clinicaltrials.gov (National Library of Medicine, Bethesda, Maryland).1015 

The mean score on the pretest was 45% (range, 20%–85%) compared with 70% on the posttest (range, 30%–90%; P < .001; effect size = 1.4). All participants had an increase in pretest versus posttest scores (Figure). The mean score on the 9 knowledge-based questions was 52% on the pretest and 75% on the posttest (P < .001). For the skills-based questions regarding the use of online genomics tools, the mean score was 41% on the pretest and 73% on the posttest (P = .001). Almost every question showed an improvement in scoring in the posttest compared with the pretest, with an average 27% increase in the number of participants answering correctly (range, −7% to 60%; Table 1). The average increase was 24% (range, −7% to 60%) for knowledge-based questions and 30% for skills-based questions (range, 13%–53%).

Comparison of individual pretest versus posttest scores. Each symbol represents a different study participant.

Comparison of individual pretest versus posttest scores. Each symbol represents a different study participant.

Close modal
Table 1

Pretest Versus Posttest Comparison for Individual Questions

Pretest Versus Posttest Comparison for Individual Questions
Pretest Versus Posttest Comparison for Individual Questions

There were 19 trainees who completed the postmodule survey. Although 4 respondents (21%) reported some technical issues with the modules, results were still uniformly positive, with 18 (95%) indicating they would recommend the modules to others and that the modules were a valuable use of time (Table 2). Regarding comparison with a “typical lecture,” 11 (58%) indicated it was a better use of time (with the remainder indicating it was a similar use of time and no one reporting that it was a worse use of time). When asked if the modules were more engaging than a typical lecture, 12 (63%) responded “yes,” 6 (32%) responded “somewhat,” and 1 individual (5%) selected “no.” When asked whether the modules were an adequate simulation of TBL, for the 16 participants who had previously experienced this teaching modality, 6 (38%) selected “yes” and 6 (38%) selected “somewhat,” with the remaining 4 (25%) selecting “no.” In narrative comments related to simulating TBL, most noted that the aspect that was lacking was the opportunity to interact in-person with peers. In other comments, participants enjoyed the self-directed learning with the ability to individualize their experience. The interactivity was a prominent positive theme. Areas for improvement included the technical issues, and for several respondents, the modules were too long.

Table 2

Postmodule Survey Results (n = 19)

Postmodule Survey Results (n = 19)
Postmodule Survey Results (n = 19)

The online modules enabled acquisition of both knowledge and the practical ability to use genomics Web sites. There were significant differences in pretest versus posttest scores, with some very dramatic increases (eg, <30% on the pretest to more than 80% on the posttest) and an effect size of 1.4. Survey results were uniformly positive, with a high percentage of participants indicating they would recommend the modules to others. Feedback related to technical issues and module length will be addressed in future iterations. Not surprisingly, a lower percentage of participants thought that the modules at least somewhat captured the team-based learning experience. Simulating the personal interactions of a team is challenging, requiring exploration of additional techniques. All participants, however, agreed the modules held their interest and were engaging.

There are several limitations of our study. First, the analysis included only 15 trainees. This likely reflects that taking the modules, let alone participating in the study, was not mandatory and required taking time out of the busy trainee schedule. We chose this approach to ensure that learners did not feel undue pressure to participate. Even with the voluntary nature, 35% who took the pretest also took the posttest, indicating interest in the material and educational approach. Our results also showed a large effect size even with this small group and demonstrated efficacy at geographically diverse US residency programs of varying sizes. Additionally, because the exams were essentially anonymous, there was limited impetus to use outside resources during the testing to boost scores.

Given the small number and volunteer nature, participants who completed the study were likely more motivated and may not represent the broader pool of trainees. In the future as the modules are more freely available, residency program directors can be surveyed regarding adoption of modules in a nonvoluntary setting. Developing the modules also took significant time and, as there was need for external contractors, expense to develop. This approach, however, could lead to overall savings by reducing the number of faculty required to develop and lead in-person TBL sessions at individual institutions. Finally, given time constraints, we were not able to test for extinction of the educational effect.

Our study demonstrates the possibility of using a virtual environment to teach skills-based objectives related to the use of online genomics tools. This innovative simulation-based approach has applications beyond genomics and pathology. Screen recording and simulation could be effective in teaching the use of other computer-based tools to different learner groups. Advantages include allowing the learner to proceed at his or her own pace and saving faculty time and resources. Given the difficulty in simulating personal team interactions, one possible area to explore is to have, as opposed to individual use, a team of trainees access the online modules together and determine the answers to activity questions as a group. In this way, the benefits of both TBL and the online modules could be achieved. Plans are also underway to adapt this approach to a medical student genomics curriculum.

This work was supported by the National Institutes of Health (2R25CA168544).

1
Laudadio
J
,
McNeal
JL
,
Boyd
SD
, et al.
Design of a genomics curriculum: competencies for practicing pathologists
.
Arch Pathol Lab Med
.
2015
;
139
(
7
):
894
900
.
2
Haspel
RL
,
Olsen
RJ
,
Berry
A
, et al.
Progress and potential: training in genomic pathology
.
Arch Pathol Lab Med
.
2014
;
138
(
4
):
498
504
.
3
Parmelee
D
,
Michaelsen
LK
,
Cook
S
,
Hudes
PD
.
Team-based learning: a practical guide: AMEE guide no. 65
.
Med Teach
.
2012
;
34
(
5
):
e275
e287
.
4
Twenge
JM
.
Generational changes and their impact in the classroom: teaching Generation Me
.
Med Educ
.
2009
;
43
(
5
):
398
405
.
5
Haspel
RL
,
Ali
AM
,
Huang
GC
.
Using a team-based learning approach at national meetings to teach residents genomic pathology
.
J Grad Med Educ
.
2016
;
8
(
1
):
80
84
.
6
Musunuru
K
,
Haspel
RL
.
Improving genomic literacy among cardiovascular practitioners via a flipped-classroom workshop at a national meeting
.
Circ Cardiovasc Genet
.
2016
;
9
(
3
):
287
290
.
7
Articulate Global Inc
.
Storyline 3
.
March
4,
2018
.
8
Jacquez
RJ
.
7 reasons why the new Articulate storyline is a game-changer for eLearning
.
March
4,
2018
.
9
TRIG Working Group
.
TRIG online module 3: design of a multigene assay
. ,
2018
.
10
National Center for Biotechnology Information
.
ClinVar
. ,
2018
.
11
Adzhubei
IA
,
Schmidt
S
,
Peshkin
L
, et al.
A method and server for predicting damaging missense mutations
.
Nat Methods
.
2010
;
7
(
4
):
248
249
.
12
Forbes
SA
,
Beare
D
,
Boutselakis
H
, et al.
COSMIC: somatic cancer genetics at high-resolution
.
Nucleic Acids Res
.
2017
;
45
(
D1
):
D777
D783
.
13
Vanderbilt University
.
Mycancergenome.org
.
March
4,
2018
.
14
National Center for Biotechnology Information
.
Online Mendelian Inheritance in Man (OMIM)
. ,
2018
.
15
National Library of Medicine
.
Clinicaltrials.gov
.
https://clinicaltrials.gov/. Accessed March 4
,
2018
.

Author notes

The authors have no relevant financial interest in the products or companies described in this article.