Background

Procedural skill is predicated on knowledge. We used a previously validated test to evaluate the impact of a web-based education program on medical residents' knowledge of 2 advanced medical procedures.

Methods

We enrolled 210 internal medicine residents at 3 residency programs in a randomized, controlled, educational trial. Study participants completed a 20-item, validated online test of their knowledge of central venous and arterial line (CVL and AL, respectively) placement at baseline and after performing their next 2 procedures (test 1 and test 2). Between test 1 and test 2, participants were randomized to online educational material for CVL insertion, AL insertion, both, or neither. The primary outcome of the study was the difference in test scores between test 1 and test 2 by randomization group.

Results

Though residents in the baseline cohort were confident about their knowledge of procedural technique, their mean test scores were low (62% and 58% in the CVL and AL tests, respectively). Baseline test score correlated with the number of prior procedures performed. Sixty-five residents completed all 3 CVL tests, and 85 residents completed all 3 AL tests. Access to the web-based procedure education was associated with a significant improvement in scores for both the CVL test (effect size, d  =  0.25, P  =  .01) and AL test (d  =  0.52, P < .001).

Conclusions

Web-based procedure training improves knowledge of procedures to a significantly greater extent than performing the procedure alone. Web-based curricula can effectively supplement other methods of skill development.

Editor's Note: The online version of this article contains the survey instrument used in this study.

Complications following medical procedures are a significant cause of morbidity and mortality among hospitalized patients.1,2 Recommended strategies to improve procedure outcomes include standardization of procedure training and establishment of minimum competency standards prior to unsupervised practice. The American Board of Internal Medicine (ABIM)3 changed the requirement for procedural competence and now requires candidates for licensure to be competent in performing a more limited number of procedures prior to certification. For more advanced procedures such as placement of central venous lines (CVL) and arterial lines (AL), the ABIM and the Accreditation Council for Graduate Medical Education have discontinued use of a general expectation of procedural proficiency, instead expecting licensure candidates to demonstrate that they know, understand, and can explain a procedure's indications and contraindications and the approach to and technique of the procedure, and be able to anticipate and manage procedural complications.

The current model of procedure training generally involves personalized education in small groups in which residents are instructed by their colleagues.4,5 Rather than employing a standardized approach, training uses the apprenticeship model of “see one, do one, teach one.”5 Procedure training remains largely unstructured, and its quality is rarely formally evaluated in most residency programs.6 As a result, there is substantial variation among residents in their confidence in performing medical procedures.7,8 Efforts to improve procedure training have prompted the use of simulators and the creation of comprehensive inpatient procedure services.5,9,10 Both have demonstrated an improvement in self-reported confidence and in the case of simulation training for central venous line insertions, resulted in a decrease in procedure related complications.9,11 However, both methods of imparting procedural training are resource intensive and may not be feasible for many programs.

Learning materials available online can provide a uniform educational content across sites, can be available anytime and anywhere (even at the bedside), and may provide a central mechanism to evaluate and record the knowledge of residents. web-based teaching can be an effective educational tool for delivering curricular content, and the interactivity of online learning may improve retention.1214 Video can be efficiently incorporated into web-based curricula and may be especially useful for procedure skills training. The effectiveness of video on procedural training may be due to the existence of a “mirror neuron” function in humans, particularly important for manipulative skills and learning.15 

The primary aim of this study was to determine whether scores on a previously validated test of procedure-related knowledge would improve to a greater degree following access to a standardized web-based procedure-training curriculum and completion of the procedure than after performing the procedure without access to the curriculum. The secondary aims of this study were to determine if access to the web-based curriculum resulted in a decrease in the number of self-reported procedure-related complications and to determine whether such a curriculum for arterial line and central line training was acceptable to residents.

Curriculum Development

We convened a panel of procedure-based specialists and educators and identified procedure-related knowledge as a key deficiency of inexperienced operators. Since many procedures in the intensive care unit are time critical, this knowledge deficiency limited the effectiveness of the skill teaching that could occur at the time of the procedure.

Peer-reviewed procedure videos that were not publicly available at the time of this study teaching central venous line (CVL) and arterial line (AL) insertion technique were selected based on their educational value to residents and established content validity.16,17 Two curricular modules incorporating a procedure-specific video (approximately 10 minutes in duration), a written summary of the procedure, and access to selected articles describing CVL and AL placement were made available online. Access to the separate AL and CVL curricula was restricted by randomization group using a secure website that was accessed through a personalized login for each participant. We piloted the videos and test with a small specialist audience to refine the program and secured the funding for implementation.

Development of a Validated Test

We utilized a previously validated test of procedural knowledge developed by our group.18 Multiple-choice questions were designed to test essential aspects of medical procedures including indications, contraindications, procedure technique, and complications and their management. The subject of each question was addressed in the respective curricular module. Content validity of the examination was established by a panel of 4 critical care specialists. Instrument validity and reliability was confirmed by administering the test online to a series of students, residents, and specialty clinicians, as previously reported.18 Answers were not provided between tests.

Participants

Residents in 3 internal medicine residency programs (Brigham and Women's Hospital [BWH], Massachusetts General Hospital [MGH], and North Shore Medical Center [NSMC]) were invited to participate in the study. Residents were beginning a critical care rotation in any 1 of 4 medical centers in Massachusetts affiliated with Harvard Medical School/Partners Healthcare: BWH and MGH (university hospitals), Faulkner Hospital (community hospital affiliated with BWH), and NSMC (community hospital). Informed consent was obtained and the study was approved by the Institutional Review Board at all sites.

Study Design

Participation was defined as submission of the baseline data and completion of the baseline test. The baseline pretest included questions on both AL and CVL technique. Baseline data included demographic data such as sex, residency program and track, experience with each procedure, anticipated career path following residency training, and self-reported confidence in placing ALs and CVLs.

Participants were stratified by level of training. Subjects were then randomized using a random number generator to 1 of 4 groups granting access to the video and associated text for (1) both CVL and AL procedures, (2) CVL but not AL, (3) AL but not CVL, or (4) neither procedure (figure). Participants were asked to complete a procedure report and take the corresponding procedure-specific test (CVL test 1 or AL test 1) after performing the corresponding line insertion. Participants were asked to complete a second procedure report and complete an additional procedure-specific test (CVL test 2 or AL test 2) when they completed a second procedure (figure). With each procedure report, participants also provided the duration of the procedure, the number of attempts, and associated complications. Procedure reports were cross-referenced with the given medical record number to confirm that the procedures had been performed. Completers were residents who completed the pretest, test 1, and test 2. Residents randomized to access the web-based curriculum were asked to rate it using a 5-point Likert scale that comprised extremity anchors of strongly agree and strongly disagree.

Figure

Flowchart of Randomized, Controlled Educational Trial. Following Completion of First Arterial or Central Line, Subjects Were Block Randomized to 1 of 4 Possible Combinations With Access to Both, 1, or Neither Video for Arterial Line and Central Venous Line Placement

Figure

Flowchart of Randomized, Controlled Educational Trial. Following Completion of First Arterial or Central Line, Subjects Were Block Randomized to 1 of 4 Possible Combinations With Access to Both, 1, or Neither Video for Arterial Line and Central Venous Line Placement

Close modal

The primary outcome measure was the difference in test scores between test 1 and test 2 on each of the 2 procedural knowledge tests (AL and CVL), comparing participants who had access to the educational material to those residents who did not. Secondary outcomes included (1) the difference in self-reported complications between residents with access to the web-based procedural curriculum and those without access and (2) acceptability of the web-based educational program.

Statistical Analysis

We analyzed the data for individuals who completed the baseline test and procedure-specific tests 1 and 2 (completers). Baseline and procedure-specific test scores were normalized to a percentage scale (0%–100%). We estimated that we would need 64 completers in each arm to demonstrate a Cohen effect size (d) of 0.5 for an α of 0.05 and a power of 0.8. Cohen's d expresses the difference between the means in terms of SD units, with 0.2 generally considered a small effect, 0.5 a moderate effect, and 0.8 (and above) a large effect.19 Chi-square tests were used to examine the univariate relationship of gender, program, anticipated career path, and year of training among those randomized to access procedure videos compared to those who were not, for both ALs and CVLs. The Wilcoxon signed rank test was used to determine if the change in knowledge scores for each procedure was statistically significant between groups. Repeated measures analysis of variance was used to detect changes in score over time. Statistical analyses were performed using SPSS 15.0 (SPSS Inc., Chicago, IL).

A total of 210 residents enrolled in the study between February and June 2006 (table 1). There was no significant difference in subject characteristics including residency program, year of training, sex, and residency track (categorical/preliminary/medicine-pediatrics) between those with and without access to the online curriculum.

Table 1

Subject Characteristics of Residents Who Provided at Least 1 Procedure Report (n  =  210)

Subject Characteristics of Residents Who Provided at Least 1 Procedure Report (n  =  210)
Subject Characteristics of Residents Who Provided at Least 1 Procedure Report (n  =  210)

The most frequent time period for completing procedures was between noon and 6:00 pm (71/150, 47% of all reports), with procedures distributed approximately equally among the 3 other 6-hour periods of the day; time of procedure was similar for AL and CVL. The median reported time to complete an AL was 15 minutes (interquartile range 6–30 minutes), and for CVL, 30 minutes (interquartile range 15–40 minutes). Residents reported requiring a median of 2 passes to access the artery (interquartile range 1–3) and a median of 2 passes to access the central vein (interquartile range 1–3).

Residents expressed a high degree of confidence in their ability to perform procedures with and without supervision (table 2), yet baseline scores on both knowledge tests were low. The mean baseline scores in the initial cohort were 58% (SD 23%) for AL insertion and 62% (SD 20%) for the CVL test. A higher number of procedures performed was significantly associated with a higher score on the baseline knowledge test (data not shown). There were no significant differences between baseline scores by gender, residency track, training program, or year of training.

Table 2

Baseline Self-Reported Confidence in Performance of Each of the Listed Tasks (n  =  210)

Baseline Self-Reported Confidence in Performance of Each of the Listed Tasks (n  =  210)
Baseline Self-Reported Confidence in Performance of Each of the Listed Tasks (n  =  210)

Eighty-five (40.5%) subjects completed all 3 AL tests, and 65 (30.9%) completed all 3 CVL tests and were included in the per-protocol analysis. Residents without access to the online curriculum had no significant change in test score between test 1 and test 2. Residents with access to the online educational program improved their scores significantly for both the AL test (from 58% to 70%, absolute score difference 12%; d  =  0.52, P < .001) and the CVL test (from 65% to 70%, absolute score difference 5%; d  =  0.25, P  =  .01) (table 3). A repeated measures analysis of variance confirmed a significant interaction between randomization group and time within both the AL (P  =  .002) and CVL (P  =  .031) studies.

Table 3

Mean Test Scores for Arterial Line and Central Venous Line Tests at Baseline and Following the Performance of the Next 2 Procedures (Completers)

Mean Test Scores for Arterial Line and Central Venous Line Tests at Baseline and Following the Performance of the Next 2 Procedures (Completers)
Mean Test Scores for Arterial Line and Central Venous Line Tests at Baseline and Following the Performance of the Next 2 Procedures (Completers)

The overall reported success rate for each second procedure did not differ between groups (for AL, the success rates were 78% (28/36) and 78% (38/49) with and without the curriculum, respectively; for CVL, the success rates were 89% (25/28) and 89% (33/37), respectively).

Residents who had access to the educational materials rated the online curricula for AL and CVL insertion highly, with a substantial majority indicating that they would recommend it to their peers [97% (35/36) and 96% (27/28), respectively] and that the video had helped them learn 89% (32/36) and 82% (23/28), respectively. A majority believed that the video had helped them troubleshoot [63% (23/36) and 53% (15/28), respectively]. A significant minority of residents indicated that access to the curriculum had changed the way they performed the procedure [42% (15/36) and 29% (8/28), respectively].

Overall, 7% (15/207) of AL insertion attempts were reportedly complicated by a hematoma. The self-reported complication rate was higher with CVL insertion than with AL insertion; complications included bleeding [15% (26/169)], arterial puncture [7/169 (4%)], and catheter malposition [5% (9/169)]. There were no significant differences in the complication rate between residents randomized to the online curriculum and those without access for both AL (12 versus 11 complicated procedures; P  =  .90) and CVL (15 versus 18 complicated procedures; P  =  .20) placement. However, compared with participants who reported no complication, participants who self-reported a complication had a significantly lower score on the AL test (59% versus 66%; P  =  .047) and a lower score on the CVL test (63% versus 68%; P  =  .187).

This randomized, controlled educational study demonstrated that residents across 3 residency programs exhibited significant deficits in their knowledge of key medical procedures on a validated test containing items deemed important by a panel of critical care clinicians. Overconfidence was widespread at baseline. However, participants who had more experience at baseline demonstrated greater knowledge of the procedures. In this study, we have shown that access to an online curriculum resulted in an improvement in the participants' self-reported ability to perform these procedures. Furthermore, participants with access to the curriculum demonstrated a modest but statistically significant improvement in procedural knowledge compared with participants performing the procedure without any additional curriculum.

Randomized, controlled trials of educational innovations are uncommon and difficult to perform but offer the rigor necessary to answer important questions.2022 Prior studies have demonstrated the utility of a dedicated “procedure-related service” and use of simulators in medical procedure training. In a single-center study,23 a 2-week inpatient medical procedure rotation (consisting of lectures, instructional videos and text, supervised practice on mannequins, and direct supervision of inpatient procedures by a faculty physician) resulted in subjective improvement in participating residents' comfort and self-rated knowledge. Studies that have used more objective outcomes, such as a decrease in procedure-related complications, have been limited by a before-versus-after study design and therefore lack a control group.24,25 

Although proficiency standards have not yet been established, the ABIM3 expects that medical residents demonstrate procedural knowledge regarding placement of CVL and AL in order to be eligible for certification. However, the number of inpatient medical procedures is declining, and opportunities for experiential learning are becoming more limited.26 As residents prepare themselves to engage in supervised practice, program directors and supervising physicians need to be confident that their residents have the necessary knowledge to learn the procedural technique. The curriculum described here aims to improve residents' knowledge of procedures, a prerequisite for skill development. The improvement in procedural knowledge generated through access to an online curriculum in this study suggests that video-based education is acceptable to learners and can be effective for this purpose. Online learning is wholly insufficient if procedural skill is the goal, and additional methods of procedural training such as simulation, supervision, practice, and feedback are necessary.27 Given the attrition of procedure knowledge and skills over time, online education may be especially useful for maintaining and reinforcing procedural knowledge in an immediately accessible manner when needed.

The strengths of this study include its randomized, controlled study design that involved multiple centers and distinct university- and community-based residency programs. Our findings have the potential for generalizability since the videos used in this study are publicly available.28 Importantly, since knowledge of procedures increases with experience, we linked completion of the tests to the time each procedure was performed, thereby excluding any bias associated with experiential learning. Finally, we assessed the efficacy of the web-based curriculum using an objective rather than subjective primary outcome—namely an improvement in scores of a previously validated test.16,17 

Several limitations should be considered when evaluating the results of this experiment. Though the study was powered to anticipate a low completion rate, the majority of participants completed only a single procedure report. The low number of second-procedure reports may be attributed to the low number of overall procedures performed by internal medicine residents, with vascular access increasingly obtained by emergency room staff and with residents spending limited continuous time in critical care units as a consequence of duty hour restrictions.29 Although we found that participants who did not complete 2 procedures were demographically no different than those who did, we cannot exclude a completer bias. Open-label trials can be subject to residual confounding that persists despite appropriate randomization. These differences may account for the low completion/large dropout rate in this study.

Although a testing effect could account for changes in test performance over time, this is unlikely because a testing effect would not account for the differences in test scores between randomization groups, and because participants did not have access to the answers until the entire study was completed. We did not evaluate whether changes in procedural knowledge were sustained over time because of practical constraints of our residents' procedural exposure. Residents in both arms of the study across study sites were likely to have received additional bedside and/or didactic teaching that was not part of this study. Although there were no differences in our results by study site, we cannot rule out residual confounding due to site differences in procedural training. Though the differences between groups achieved statistical significance, the changes in test scores that we have documented following access to an online curriculum were modest. The limited effect of our online curriculum is reason to be cautious about the role of this method as the sole means of procedural training in residency, particularly with respect to its role in skill development.

Though knowledge is a prerequisite for skill development, increased procedural knowledge may not increase procedural skill accordingly, and an assessment of competence of procedural skill remains essential prior to the performance of these procedures on critically ill patients. Finally, this study does not attempt to suggest that an online curriculum is a substitute for simulation or other training when procedural competence is desired.10 

Our study suggests that additional efforts to improve resident knowledge of procedural techniques are justified since overconfidence is common and the trainees in this cohort demonstrated important knowledge gaps. Our findings demonstrate that web-based curricula can be a moderately effective approach to improving residents' knowledge of procedures. When the educational goal is increasing knowledge, our study confirms that online tools can be an appropriate choice, but educators need to consider the costs and time required to create video-based curricula, since the benefit to knowledge accrual is modest. In addition, skill development and procedural competence would necessitate further training and practice. Additional studies are needed to evaluate the role of online education in long-term retention of knowledge, its utility in facilitating the development of procedural skill, and the impact on patient safety.

1
Brennan
,
T. A.
and
L. L.
Leape
.
Adverse events, negligence in hospitalized patients: results from the Harvard Medical Practice Study.
Perspect Healthcare Risk Manage
1991
.
11
(
2
):
2
8
.
2
Leape
,
L. L.
,
T. A.
Brennan
,
N.
Laird
, et al
.
The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II.
N Engl J Med
1991
.
324
(
6
):
377
384
.
3
American Board of Internal Medicine
Policies and procedures for certification in internal medicine.
4
Jonas
,
H. S.
,
S. I.
Etzel
, and
B.
Barzansky
.
Educational programs in US medical schools.
JAMA
1991
.
266
(
7
):
913
920
.
5
Smith
,
C. C.
,
C. E.
Gordon
,
D.
Feller-Kopman
, et al
.
Creation of an innovative inpatient medical procedure service and a method to evaluate house staff competency.
J Gen Intern Med
2004
.
19
(
5, pt 2
):
510
513
.
6
Fincher
,
R. M.
Procedural competence of internal medicine residents: time to address the gap.
J Gen Intern Med
2000
.
15
(
6
):
432
433
.
7
Hicks
,
C. M.
,
R.
Gonzalez
,
M. T.
Morton
,
R. V.
Gibbons
,
R. S.
Wigton
, and
R. J.
Anderson
.
Procedural experience and comfort level in internal medicine trainees.
J Gen Intern Med
2000
.
15
(
10
):
716
722
.
8
Huang
,
G. C.
,
C. C.
Smith
,
C. E.
Gordon
, et al
.
Beyond the comfort zone: residents assess their comfort performing inpatient medical procedures.
Am J Med
2006
.
119
(
1
):
71.e17
24
.
9
Barsuk
,
J. H.
,
W. C.
McGaghie
,
E. R.
Cohen
,
J. S.
Balachandran
, and
D. B.
Wayne
.
Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit.
J Hosp Med
2009
.
4
(
7
):
397
403
.
10
Millington
,
S. J.
,
R. Y.
Wong
,
B. O.
Kassen
,
J. M.
Roberts
, and
I. W.
Ma
.
Improving internal medicine residents' performance, knowledge, and confidence in central venous catheterization using simulators.
J Hosp Med
2009
.
4
(
7
):
410
416
.
11
Barsuk
,
J. H.
,
E. R.
Cohen
,
J.
Feinglass
,
W. C.
McGaghie
, and
D. B.
Wayne
.
Use of simulation-based education to reduce catheter-related bloodstream infections.
Arch Intern Med
2009
.
169
(
15
):
1420
1423
.
12
Kerfoot
,
B. P.
,
H.
Baker
,
T. L.
Jackson
, et al
.
A multi-institutional randomized controlled trial of adjuvant web-based teaching to medical students.
Acad Med
2006
.
81
(
3
):
224
230
.
13
Kerfoot
,
B. P.
,
P. R.
Conlin
, and
G. T.
McMahon
.
Comparison of delivery modes for online medical education.
Med Educ
2006
.
40
(
11
):
1137
1138
.
14
Ruiz
,
J. G.
,
M. J.
Mintzer
, and
R. M.
Leipzig
.
The impact of e-learning in medical education.
Acad Med
2006
.
81
(
3
):
207
212
.
15
Paas
,
F.
,
P.
Van Gerven
, and
P.
Wouters
.
Instructional efficiency of animation: effects of interactivity through mental reconstruction of static key frames.
Appl Cogn Psychol
2007
.
21
:
783
793
.
16
Graham
,
A. S.
,
C.
Ozment
,
K.
Tegtmeyer
,
S.
Lai
, and
D. A.
Braner
.
Videos in clinical medicine. Central venous catheterization.
N Engl J Med
2007
.
356
(
21
):
e21
.
17
Tegtmeyer
,
K.
,
G.
Brady
,
S.
Lai
,
R.
Hodo
, and
D.
Braner
.
Videos in clinical medicine. Placement of an arterial line.
N Engl J Med
2006
.
354
(
15
):
e13
.
18
Grover
,
S.
,
P. F.
Currier
,
J. M.
Elinoff
,
K. J.
Mouchantaf
,
J. T.
Katz
, and
G. T.
McMahon
.
Development of a test to evaluate residents' knowledge of medical procedures.
J Hosp Med
2009
.
4
(
7
):
430
432
.
19
Maxwell
,
S.
and
H.
Delaney
.
Designing Experiments and Analyzing Data: A Model Comparison Approach
.
Belmont, CA
Wadsworth
.
1990
.
20
Todres
,
M.
,
A.
Stephenson
, and
R.
Jones
.
Medical education research remains the poor relation.
BMJ
2007
.
335
(
7615
):
333
335
.
21
Cook
,
D. A.
and
T. J.
Beckman
.
Reflections on experimental research in medical education.
Adv Health Sci Educ Theory Pract
2010
.
15
(
3
):
455
464
.
22
Prystowsky
,
J. B.
and
G.
Bordage
.
An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction.
Med Educ
2001
.
35
(
4
):
331
336
.
23
Lenhard
,
A.
,
M.
Moallem
,
R. A.
Marrie
,
J.
Becker
, and
A.
Garland
.
An intervention to improve procedure education for internal medicine residents.
J Gen Intern Med
2008
.
23
(
3
):
288
293
.
24
Martin
,
M.
,
B.
Scalabrini
,
A.
Rioux
, and
M. A.
Xhignesse
.
Training fourth-year medical students in critical invasive skills improves subsequent patient safety.
Am Surg
2003
.
69
(
5
):
437
440
.
25
Sherertz
,
R. J.
,
E. W.
Ely
,
D. M.
Westbrook
, et al
.
Education of physicians-in-training can decrease the risk for vascular catheter infection.
Ann Intern Med
2000
.
132
(
8
):
641
648
.
26
Wigton
,
R. S.
and
P.
Alguire
.
The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians.
Ann Intern Med
2007
.
146
(
5
):
355
360
.
27
Gaies
,
M. G.
,
S. A.
Morris
,
J. P.
Hafler
, et al
.
Reforming procedural skills training for pediatric residents: a randomized, interventional trial.
Pediatrics
2009
.
124
(
2
):
610
619
.
28
McMahon
,
G. T.
,
J. R.
Ingelfinger
, and
E. W.
Campion
.
Videos in clinical medicine—a new journal feature.
N Engl J Med
2006
.
354
(
15
):
1635
.
29
Rivers
,
E.
,
B.
Nguyen
,
S.
Havstad
, et al
.
Early goal-directed therapy in the treatment of severe sepsis and septic shock.
N Engl J Med
2001
.
345
(
19
):
1368
1377
.

Author notes

Shilpa Grover, MD, MPH, is an Associate Physician in the Division of Gastroenterology, Hepatology, and Endoscopy of the Department of Medicine at Brigham and Women's Hospital and Instructor in Medicine at Harvard Medical School; Paul F. Currier, MD, is an Instructor in Medicine at Harvard Medical School and Associate Physician in the Pulmonary and Critical Care Unit at Massachusetts General Hospital; Jason M. Elinoff, MD, is a Clinical Fellow at Harvard Medical School and a Resident in the Department of Medicine, Massachusetts General Hospital; Joel T. Katz, MD, is an Associate Physician in the Division of Infectious Diseases at Brigham and Women's Hospital and Assistant Professor of Medicine at Harvard Medical School; and Graham T. McMahon, MD, MMSc, is an Associate Physician in the Division of Endocrinology, Diabetes, and Hypertension at Brigham and Women's Hospital and Assistant Professor of Medicine at Harvard Medical School.

Acknowledgments: Innovation Grant in Instructional Computing, Harvard University Provost. Program in Cancer Outcomes Research and Training R25 CA092203.

Supplementary data