An Evidence-Based Medicine Curriculum Improves General Surgery Residents' Standardized Test Scores in Research and Statistics Open Access

The application of evidence-based medicine to patient care requires physicians to cultivate unique skills. To advance residents' knowledge of evidence-based medicine and its application to patient care, residents must understand fundamental research concepts. Research methodology, epidemiology, and statistics topics can be an area of uncertainty to residents who have had minimal exposure to such concepts. After examining our institution's surgical residents' standardized test scores since 2007, the program director noted opportunities for improvement in the basic science questions relating to research methodology and statistics.

The American Board of Surgery In-Training Examination (ABSITE) measures residents' competence in basic science and clinical management. The ABSITE scores have been shown to correlate with pass rates on the American Board of Surgery qualifying examination.1,2 Educational strategies to improve scores have included review courses,2 increased study hours, regular conference attendance,3,4 and programmed regular reading.5–7 Four of 5 surgical residency programs (80%) do not offer institutional or regional ABSITE review courses, but 60% (39 of 65) of program directors believe that review courses improve resident ABSITE performance.8 The majority of published ABSITE preparatory programs take a global approach, encompassing numerous educational topics, and there are no published ABSITE educational programs that focus specifically on research methodology and statistics.

Many residency programs hold regular “journal club” sessions, and there is substantial variability in the formats and objectives of these sessions.9 Reviewing the current literature is a common reason for holding a journal club; only 52% (119 of 227) of program directors report that research education is a motivating factor for a journal club.9 Given residency programs' varying motivations and diverse implementation strategies, a journal club alone may not be adequate for educating residents in research methodology and statistics.

The purpose of our study was to evaluate whether scores on the ABSITE in research and statistics improved after implementation of a 5-hour curriculum. We hypothesized that residents' research and statistics question scores on the ABSITE would significantly improve after implementation of a curriculum focused on evidence-based medicine research methods.

The curriculum was implemented in an Accreditation Council for Graduate Medical Education–accredited general surgery residency program at an independent academic medical center. The program grew substantially from 2 to 4 chief residents graduating annually during the study period. Residents maintain a relatively high caseload, typically completing more than 1000 procedures during training. The faculty provides Surgical Council on Resident Education (SCORE) curriculum lectures and participates in skills training and mock oral examinations. A monthly journal club focuses on literature from the American College of Surgeons continuing medical education program, Selected Readings in General Surgery.

In response to noted opportunities for improvement in scores on research-related ABSITE items, the faculty explored additional curricula to enhance resident education in epidemiology and statistics. Most ABSITE review materials do not focus on these topics, and a curriculum consisting of five 1-hour lectures that focused on research methods, epidemiology, and statistics was developed by 2 of the authors (A.W.T. and M.E.C.). Lectures, tutorials, practice questions, and examples required approximately 4 hours preparation per session in the first curriculum year. Updating presentations for the second-year curriculum typically required 1 hour per session. The ABSITE review materials,10–12 statistics and epidemiology resources,13–18 and prior ABSITE reports were used to design the curriculum and create practice questions. Lectures were given monthly from September through January before the 2012 and 2013 examinations, conducted at the end of January. With support from the research program coordinator (M.E.C.), the department's research statistician (A.W.T.) presented the curriculum during scheduled resident conference times, administered separately for junior residents (postgraduate year [PGY]–1 and PGY-2) and senior residents (PGY-3, PGY-4, and PGY-5). Sessions were structured, but discussion of topics flowed freely with flexibility for questions. Five practice questions were given at the end of each session, and answers were discussed among the group. A cumulative review session of all prior lecture material was given in January, 2 weeks before the ABSITE administration. Residents were assigned to 2 teams to compete in answering practice questions (similar to monthly lecture practice questions). The winning team received $15 in cafeteria coupons, and the 3 top scoring individuals received $20 gift cards (total cost, $75 × 2 sessions  =  $150). An outline of the curriculum is presented in table 1. The lecture series occurred during protected educational time, which is mandatory for all residents. During protected educational time, 1 morning per week, residents are excused from clinical duties to attend lectures and skills training. During other weekdays at that time, residents maintain regular clinical responsibilities.

The hospital's Institutional Review Board determined that this educational study was exempt from review.

The effects of the curriculum were determined by comparing precurriculum and postcurriculum ABSITE scores for topics related to research and statistics. Forty general surgery residents from 2007 to 2013 were included in the analysis. Two investigators (A.W.T. and M.E.C.) independently reviewed ABSITE question topics from individual score sheets to determine which items were related to research and statistics. Both investigators identified the same 44 question topics, and there were no discrepancies. Residents' correct and incorrect responses for each identified ABSITE question were collected from the Surgical Residency Program Coordinator.

A multivariable linear regression model was calculated to estimate the level of improvement in ABSITE scores after the educational curriculum was implemented. The primary outcome variable was the percentage of correct answers on research and statistics questions per individual per year. The main predictor variable was the year of examination. The regression model included adjustments for PGY level (1–5), and repeated measures correlations between each individual's annual ABSITE scores over time were accounted for using clustered sandwich variance estimators. A second model evaluated whether residents were more likely to answer 100% of statistics questions correctly after the statistics review lectures. A multivariable logistic regression model was developed using a binary outcome (yes/no), representing 100% correct answers. The model included adjustment for examination year and resident level (PGY). Clustered sandwich variance estimators were again used to account for correlations between individual's scores over time. Statistical analyses were performed using Stata version 12 software (StataCorp LP). Statistical significance was assessed at α  =  0.05.

A total of 109 ABSITE scores from 40 residents over 7 years were included in the analysis. Forty-four examination questions relating to evidence-based medicine, research, or statistics were included (table 2). In the postcurriculum ABSITE administration, junior residents (PGY-1 and PGY-2) had a mean score of 84% correct answers in 2012 and 94% correct in 2013. Seniors (PGY 3–5) had a mean score of 89% correct in 2012 and 84% correct in 2013 (figure). Linear regression results estimate that, compared with scores from 5 previous years, overall correct responses for postcurriculum ABSITE administrations increased by 27% (95% CI, 17%–37%, P < .001) after adjustment for PGY level and individuals' scores over time.

Before the curriculum was implemented, 10% (3 of 30) of the junior residents achieved perfect scores on ABSITE research and statistics items, compared with 62% (10 of 16) postcurriculum. Similarly, 30% (13 of 43) of the senior residents correctly responded to all research items, compared with 55% (11 of 20) postcurriculum. Logistic regression results indicate that residents had more than 5 times the odds of achieving a perfect score on research and statistics items postcurriculum, as compared with previous examination years (OR, 5.2; 95% CI 2.1–13; P < .001).

A series of five 1-hour lectures on statistical and research concepts was presented to surgical residents monthly before the 2012 and 2013 ABSITE administrations, with cumulative review sessions in January before the examinations. The results of this study demonstrate that residents dramatically improved ABSITE scores on research and statistics questions after receiving formal instruction on these examination concepts. Junior and senior residents demonstrated significant improvements in correct responses to research and statistics items, including overall percentage of correct answers and proportion of residents achieving 100% correct answers.

Results of this study are consistent with previous reports of improved ABSITE scores after implementation of review courses,2 and support the perception by most program directors that review courses improve ABSITE scores.8 To the investigators' knowledge, this is the first ABSITE study program focusing exclusively on research methods, epidemiology, and statistics. Other successful methods to improve ABSITE scores, including increased study hours,3,4 regular conference attendance,3,4 and programmed regular reading,5–7 have been reported.

There are multiple reasons why the score improvements reported in this study should be of interest. True subject matter comprehension is critical to all residents regardless of their ultimate intended specialty. This curriculum led to reproducible subject matter comprehension, as evidenced by 2 years of accomplished improvements in ABSITE topics, which the study investigators had no influence in selecting. Whether the intent is to create or appraise research, epidemiology and statistics are requisite lifelong skills for all surgeons. To that point, questions on these topics will persist on the revised ABSITE, which reunites the junior and senior examinations. Given the limited amount of available protected resident educational time, this study provides evidence to justify the allocation of time for the purpose of providing enduring knowledge on these important topics. The success of this curriculum can serve as a model for residency programs in other disciplines as well. By reproducing the efficiency, comprehensiveness, and repetition of this design, the same improvements in ABSITE scores could theoretically be seen with other clinical and basic science topics.

This study has some limitations. Because this curriculum was designed and implemented for 1 general surgery residency program, there is no comparison group. We have limited information on knowledge retention, spanning only 2 years. The study hospital experienced growth on many levels during the period of the studied cohort, including increases in resident complement as well as scholarly activity. One cannot rule out the possibility that improvements in examination scores could be the result of other changes over time. However, during the 5-year baseline period, there were no obvious trends for score improvement, suggesting that the effect of other program changes over time is small. We were unable to compare residents' scores to a national average for these evidence-based medicine topics because ABSITE scores provided to program directors are summarized across broad subject categories (eg, Body as a Whole/Basic Science). National averages for ABSITE subscores within the basic science section, for research methodology, statistics, patient safety, and other categories would be beneficial for evaluating educational programs.

Surgical residents demonstrated significant improvement in research and statistics ABSITE scores after receiving five 1-hour lectures in these concepts. Junior and senior residents improved both the overall percentage of correct answers and the proportion of residents achieving 100% correct answers. Further studies are warranted to evaluate long-term knowledge retention and translation to clinical practice, and to expand the program to other disciplines. If the adaptation to other clinical areas produces similar results, this type of curriculum could be used by program directors to create a more effective didactic schedule and to guide future revisions of the SCORE curriculum.