To practice evidence-based medicine, physicians should have a solid understanding of fundamental epidemiological and biostatistical concepts. Research suggests that only a minority of physicians have such an understanding of biostatistics.
To collect pilot data on a novel biostatistical educational tool, a hyperlink-embedded journal article, which is aimed at improving knowledge in biostatistics.
Forty-four physicians-in-training participated in this pilot study. Participants completed a pretest consisting of 5 questions about biostatistical terms that would be encountered in the article. They were randomized to either an unmodified journal article or to the same article with hyperlinked biostatistical terms. All participants then completed a posttest that was identical to the pretest.
Having access to hyperlinked information had a positive association with the number of improved test answers (P = .05). Use of hyperlinks varied, and were seemingly dependent on user comfort with terms; well-understood definitions (“average”) were clicked on a few times (5.5% of participants), whereas more obscure method terms (“Lexis diagram”) were clicked on by 94% of participants. While only 42% of participants stated they would have looked up definitions of the biostatistical terms if they had not been provided in the hyperlinked article, 94% of participants identified the hyperlink tool as something they would use if readily available to them when reading journal articles.
Results of this pilot study of a novel educational intervention suggest that embedded hyperlinks within journal articles may be a useful tool to teach biostatistical terms to physicians.
Editor's Note: The online version of this article contains the survey instrument used in the study.
To provide informed, up-to-date patient care, physicians must be able to critically analyze the medical literature and accurately decipher its clinical implications. In 1998, recognizing an educational gap in this area and looking to reform medical education overall, the Accreditation Council for Graduate Medical Education's Outcome Project stressed physician competency in practice-based learning, which was defined as the ability to “locate, appraise, and assimilate evidence from scientific studies.”1 Subsequently, much emphasis has been placed on teaching evidence-based medicine (EBM) in graduate medical education.
A natural consequence of the EBM movement is that physicians must improve their knowledge of biostatistics. However, many physicians do not understand basic biostatistical principles.2–5 Despite recognition of this deficit, a recent Mayo Clinic study found that only 9% of surveyed students and physicians felt that their current level of biostatistical training was adequate, while 87% recognized that a better understanding of biostatistics would benefit their careers.6 These bleak numbers suggest deficiencies in how physicians are currently taught critical appraisal skills (eg, classroom lectures and journal clubs).
There has been a call to research the best educational approaches to improve physicians' knowledge of EBM and biostatistics.5,7 Just-in-time instructional scaffolding and interactive web-based learning are proven educational techniques. While evidence of their utility with regard to biostatistical and EBM training are lacking to date, these approaches are believed to hold promise.8 We therefore piloted a randomized trial of a novel computer-based intervention, a hyperlink-embedded journal article (HEJA), geared toward improving the understanding of biostatistical terms by physicians-in-training.
Participants were physicians-in-training who were randomized into groups A and B and were exposed to 2 different versions of an open-access journal article.9 Group A was exposed to a manuscript with embedded hyperlinks that, when clicked, provided clinician-oriented definitions of the biostatistical terms (figure). Group B was given the same manuscript without hyperlinks. All participants completed an identical pretest and posttest assessment tool.
Of 58 internal medicine and internal medicine-pediatrics interns in the 2013–2014 class, 23 (40%) volunteered to participate. Of a possible 30 fellows in the hematology and medical oncology fellowship for the same year, 23 (77%) volunteered to participate. Two individuals completed the pretest, but not the posttest. Recruitment was conducted via e-mail.
We selected a short, open-access online journal article9 and inserted 36 web browser–based hyperlinks to 9 distinct biostatistical terms in the body of the article. Embedded hyperlinks were only available to individuals in Group A (the intervention group). When clicked on, the hyperlinks opened pop-up browser windows containing biostatistical terminology definitions targeted to the literacy level of postgraduate medical learners (figure). Biostatistical definitions were written by 1 of the authors (A.T.P.), who is an internal medicine physician with a PhD in statistics. Each definition took between 10 to 60 minutes to compose and edit.
Residents' and fellows' knowledge of selected biostatistical terms was measured by a pretest and posttest consisting of 5 unique test items. These tests were specifically designed for this trial by a group of physicians at our institution (including a physician-statistician) and were focused on the terms found in the selected journal article. Pretest and posttest questions were identical. Beyond that, the pretest included demographic questions, and the posttest included questions on ease of use and perceived utility of the intervention to assess participants' satisfaction with the HEJA (provided as online supplemental material). For participants in Group A, data about the number of hyperlinks clicked also were collected.
Participants were randomized to either Group A or B, stratified by postgraduate year. E-mails were sent to 44 physicians-in-training in Group A and 44 in Group B. One of the authors (D.S.) administered the survey, and was not blinded to group assignments. Surveys were administered between December 2, 2012, and April 25, 2014, using Qualtrics software, available on an internal website. Participants completed the pretest, read the provided manuscript, and completed the posttest in a single session. Open-ended responses were blinded and graded by a statistician.
The study protocol was reviewed by the University of Michigan Institutional Review Board and was determined to be exempt. Residents and fellows indicated assent by taking the pretest.
We calculated the percentages of correctly answered questions before and after reading the article, with or without access to the hyperlink-based biostatistical definitions (table). In general, the proportion of correct answers increased for both groups. The largest improvement in correct answers was noted for “how to use a Lexis diagram,” which increased from 22% (4 of 18) to 72% (13 of 18) with hyperlink access. Similarly, Group A participants who could correctly define Akaike information criterion (AIC) increased from 11% (2 of 18) to 56% (10 of 18), while 0% (0 of 18) of individuals in Group B were able to correctly define AIC in the posttest.
We also assessed whether access to our HEJA term glossary improved test responses. For each participant, the total number of incorrect-to-correct test responses were tallied on the pretest and compared to those in the posttest (range 0 to 3). We then used a 1-way analysis of variance to contrast the number of improved responses to hyperlink access. While we did not power this pilot study to detect a difference between groups, the improvement in correct responses approached biostatistical significance (P = .05).
Every hyperlinked definition was accessed by at least 1 participant in Group A. Use varied relative to the familiarity of biostatistical terms. Proportional use ranged from a low of 6% (for “average”) to a high of 94% (for “Lexis diagram”). Only 41% (7 of 17) of all respondents said that they would have spent time to look up terms had they not been provided. Of the respondents in Group A, 94% (16 of 17) stated that they would use embedded hyperlinks of biostatistical terms if widely available in journal articles. Additionally, 82% (14 of 17) stated that they found the inclusion of hyperlinks useful in answering the posttest questions.
We performed a pilot randomized trial of a hyperlink-embedded journal article (HEJA), which is a newly developed biostatistics education tool. Our findings suggest that physicians-in-training view this learning aid as beneficial, and that it appears to improve knowledge of biostatistical terms in a real world context.
There were several limitations to this study. A small sample size of physicians-in-training at 1 academic institution was studied. Furthermore, the pretest and posttest questions did not have evidence of validity and were designed to test comprehension of a small number of biostatistical terms.
There is an acknowledged need to move evidence-based medicine learning outside the classroom and into the clinical setting.8 The high level of acceptability of HEJAs in this pilot study supports the notion that this tool has the potential to fulfill an important educational need. Furthermore, HEJAs also align with the American Medical Association's goal of “optimizing the learning environment.”10
Future steps for this project include the creation of a biostatistical term glossary and an associated program that will automatically insert hyperlinks into journal articles. We envision incorporating and studying a more robust version of this tool through journal clubs, continuing medical education activities, and online journal articles.
Our pilot data suggest that embedding journal articles with definitions of biostatistical terms may improve knowledge of more advanced terms. Additionally, this educational intervention was regarded as a potentially useful tool for learning biostatistics by physicians-in-training.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
Dr. Saxon and Dr. Pearson contributed equally to this work.
Data were presented at the University of Michigan Medical Education Day in 2012.
The authors would like to thank the University of Michigan fellows and residents who participated in this study. The authors would also like to thank John Del Valle, Raj Patel, Javier Valle, and Caren Stalburg for their support of the project.