The ACGME Duty Hour Standards and Board Certification Examination Performance Trends in Surgical Specialties Open Access

In July 2003, the Accreditation Council for Graduate Medical Education (ACGME) implemented the residency duty hour standards. Part of the rationale for limiting resident duty hours was the relationship between sleep and the quality of resident education. It was believed that duty hour limitations would enhance resident education.1,2 In surgical specialties, studies have shown the limits have improved resident attitudes regarding educational activities in surgical specialties; the limitation of duty hours has allowed for more independent content reading, literature review, conference preparation, and a sense of improvement in the effectiveness of the didactic curriculum.3,4 

However, there are few studies evaluating the long-term impact of the ACGME duty hour standards on resident educational outcomes, including standardized test performance. The most comprehensive review of the literature identified 10 studies in surgical subspecialties evaluating performance on standardized tests, using a pre-post design comparison.5 Studies reported regional data or findings from single institution studies. In 1 study, intern performance on the American Board of Surgery In-Training Examination (ABSITE) improved in basic science, clinical management, and overall test scores. Second-year residents had higher basic science scores after the ACGME duty hour standards.6 In another study, performance on the ABSITE improved in the domains of basic science and overall test scores without a significant improvement in clinical management scores.7 For 6 of the remaining studies, board examination performance remained stable, and for 2 studies it declined after the implementation of the duty hour limits. There is a paucity of evaluation of standardized test performance at the national level with the exception of 1 study showing a recent decreasing trend in performance on the American Board of Surgery (ABS) Certifying Examination.8 

The goal of this study was to describe the physician examinee performance trends on surgical specialty certifying examinations retrospectively with respect to the implementation of the ACGME duty hour standards in 2003. With local and regional improvement and the potential benefit of duty hour limitations leading to improved physician education, the hypothesis is that physician examinees have demonstrated increasing pass rate trends on board certification examinations after the implementation of the ACGME duty hour standards.

In this retrospective cohort study, the 24 medical specialty boards composing the American Board of Medical Specialties (ABMS) were evaluated. The ABMS is the organization involved in the development and utilization of standards for the ongoing certification and recertification of physicians for 24 medical specialty boards.9 In the ABMS, there are 10 primary surgical boards.

For each surgical board, the publicly available board website content was explored for descriptive statistics on certifying examinations. Board newsletters and other electronic publications were included in the search for examination statistics. Examination statistics were also found in published peer-reviewed manuscripts. Specialty board examinations were included in the primary analysis if electronic data were present for any 3 consecutive years after 2003. Subspecialty examinations were excluded from the analysis. Each member board with an examination satisfying inclusion criteria was contacted and each board agreed that electronically published pass rates were able to be quoted and used for research purposes. When available, subgroup examination results were included for first-time examinees or repeated-attempt examinees. Independent double entry and calculation were performed to ensure the accuracy of the extrapolated data. Cross-checking and validation between separate references was performed.

The primary analysis consisted of regression analyses. Simple linear regression was performed by using the examination year as the independent variable and the pass rate as the dependent variable. This regression occurred from the 2003–2004 to the 2010–2011 academic years. When available, subgroup examination trends were evaluated for first-time examinees or repeated-attempt examinees. Examinations were considered to have statistically significant pass rate trends if the 95% confidence interval of the slope of the regression line did not include zero. Examinations with significant pass rate trends were plotted to visually show these trends over time. A Pearson correlation coefficient between pass rate and year was also performed for these examinations. To supplement the primary analysis, an interrupted time series using segmented logistic regression was performed to evaluate differences in pass rate trends leading to and subsequent to the ACGME duty hour standards.10 This method models the trend in pass rates before the introduction of standards and tests if the trend in pass rates will change after the standards are introduced.

The secondary analysis consisted of a pre-post design comparison of passing and failing examinee performance surrounding the initiation of the ACGME duty hour standards in 2003. The raw numbers of physician examinees that passed and failed each board-specific examination were obtained during data acquisition. Data were included between the 1995–1996 academic year and the 2010–2011 academic year; this time frame centers on the 2003–2004 academic year and the initiation of the ACGME duty hour standards. Board examinations were included in the secondary analysis if they satisfied criteria for primary analysis and if data were available before the 2003–2004 academic year. For each board examination that satisfied secondary analysis inclusion criteria, the total numbers of physician examinees that passed and failed before and after the initiation of the ACGME duty hour standards were compared with a 2×2-χ² test. An examination was considered to have statistically significant pass rate differences if the P value from the χ² test was < .05. All statistical tests were performed with Stata 11.1 statistical software (StataCorp LP, College Station, TX) except for the segmented logistic regression, which was performed with SAS version 9.2 (SAS Institute Inc, Cary, NC).

Of the 10 primary surgical boards of the ABMS, 8 specialty boards (80%) had at least 1 examination with publicly available electronic statistics satisfying the inclusion criteria: American Board of Neurological Surgery,11–20 American Board of Obstetrics and Gynecology,21–35 American Board of Ophthalmology,12–14,21,36 American Board of Orthopaedic Surgery,11,22–24,37–43 American Board of Plastic Surgery,11–13,24,44,45 American Board of Surgery,14,21–23,46–56 American Board of Thoracic Surgery,11,14,22,23,57 and American Board of Urology.12,13,21,24,58,59 

Physician examinee performance during the study period is given in table 1. There were 16 unique board examinations that satisfied inclusion criteria. These included 8 (50%) written examinations and 8 (50%) oral examinations. Three (19%) of these examinations had first-time examinee performance and/or repeated-attempt examinee performance available for subgroup analysis. There was a total of 72 482 unique examination results that were included in the analyses, and the median number of examinees per surgical board was 5031 (interquartile range [IQR], 3290–17 211).

Linear and segmented logistic regression results are given in table 1. The simple linear regression analysis showed that 4 (25%) examinations had a linear regression line with a nonzero slope (P < .05). All 4 of these slopes were less than zero. Significant examination trends with the accompanying Pearson correlation coefficients are shown in figure 1.

The segmented regression analyses showed that 7 (44%) examinations had a change in trajectory after the initiation of the ACGME duty hour standards. Four examinations had an upward deflection in trajectory, while 3 examinations had a downward deflection in trajectory. These 7 examination trends are shown in figure 2.

The results of the secondary analyses are given in table 1. There were significant differences in 2×2-χ² tests comparing passing and failing examinees before and after 2003 on 8 (50%) of the examinations. The magnitude and direction of the 8 statistically significant certifying examinations are presented in table 2. Secondary analyses showed that there were 5 examinations with lower pass rates after 2003 and 3 examinations with higher pass rates after 2003. The median absolute difference in pass rates was 7.6% (IQR, 4.7%–9.4%).

Overall, there were 13 (81%) examinations that had either a significant pass rate trend on regression analyses and/or a significant pre-post pass rate surrounding the initiation of the ACGME duty hour standards in 2003 in the secondary analysis (P < .05). The American Board of Thoracic Surgery Part 2 (Oral) Examination had decreasing results on the simple linear regression analysis, segmented regression analysis, and χ² analysis (P < .05).

Our findings show that most certifying examinations had either a significant pass rate trend since 2003 on regression analyses and/or a significant pre-post pass rate surrounding the initiation of the ACGME duty hour standards in 2003. However, none of the data from the simple linear regression analyses satisfied our study hypothesis that duty hour limits would improve performance on the surgical certification board examination. On the contrary, data for 4 certification board examinations showed significant decreasing rate trends since 2003 on simple regression analysis. All of these trends are fairly consistent, with Pearson correlation coefficients > 0.80. The decrease on the ABS General Surgery Certifying Examination is consistent with the findings of a prior study.8 That study examined 2006–2010, evaluating only first-time examinees, and did not use linear regression.

Segmented regression is an excellent method to evaluate outcomes after an intervention.10 This method of regression evaluates the deflection in performance in the setting of established trends. For example, the 4 examinations in figure 2A have upward deflections after the initiation of the ACGME duty hour standards. These examinations, while not having significant positive slopes on simple linear regression, do show a clear change in trend after 2003. These examinations, therefore, are consistent with the study hypothesis, as the downward trends before 2003 were blunted after the initiation of the ACGME work hour standards. In figure 2A, a notable visual difference in performance between all examinees and first-attempt examinees is seen in the American Board of Obstetrics and Gynecology Principal Written Examination. First-attempt examinees outperform the population of examinees that include repeated test-takers.

In figure 2B, there are 3 examinations with downward deflections after the initiation of the ACGME duty hour standards in 2003. The American Board of Urology Certifying Examination most notably shows an interesting phenomenon. The examination trends appear to be increasing before 2003. After 2003, the trend still increases, but at a decreasing rate. The ceiling of performance is theoretically 100%, but this examination shows a unique finding possible with segmented regression analyses that directly contrasts the χ² results. In contrast, the ABS Certifying Examination, while having a clear decreasing pass rate on linear regression, has a segmented regression that suggests there is no significant trend change since 2003. This suggests the downward trend started before 2003, and that it is unclear how this trend was affected by the ACGME duty hour standards. With missing data before 2003, some of the trends from the period before the implementation of the ACGME duty hour standards may not be representative.

There were 8 examinations with significantly different pass rates on secondary analysis. The pass rate changes are mixed. The increase in pass rates seen for the American Board of Orthopaedic Surgery Part I (Written) Examination and the American Board of Urology Qualifying Examination and Certifying Examination are consistent with our hypothesis. The temporal nature of these conclusions is also consistent. These changes, however, are not associated with linear changes in pass rates from the simple linear regression analysis. The decreases in pass rates seen in the secondary analyses are inconsistent with our hypothesis. Interestingly, the χ² analyses demonstrate some potential bias introduced by repeated-attempt examinees. While first-attempt examinees have higher performance after 2003, repeated-attempt examinees, in direct contrast, have lower performance after 2003 on the American Board of Ophthalmology Written Examination.

Overall, there was only 1 examination that was statistically significant on the simple linear regression analysis, the segmented regression analysis, and χ² analysis. The American Board of Thoracic Surgery Part 2 (Oral) Examination has a decreasing pass rate on simple linear regression since 2003, and in the segmented analysis, it has a pass rate that is lower after 2003, compared with data from earlier years.

The results of our study of national performance on surgical board certification examinations are consistent with the published literature of studies using data on local and regional performance. The most complete review of the literature on educational outcomes surrounding the initiation of the ACGME duty hour standards identified 9 studies assessing board examination performance in surgical specialties. Two studies reported an improvement in performance surrounding the initiation of the ACGME duty hour standards. Five studies showed no change, and 1 study found a decrease in standardized examination scores after the implementation of the duty hour standards.5 

The segmented regression analyses and secondary analyses are more powerful from a statistical standpoint. The linear regression analyses can have only a maximum of 8 data points for seeing any linear trends in examination pass rates since 2003 and do not consider trends before 2003. In contrast, the segmented logistic regression analyses and secondary data analyses account for temporal relationships and for thousands of physician examinees across more years. Using multiple statistical approaches allows for a more robust evaluation of the data as opposed to any method alone.

Our study has several limitations. There was a considerable amount of missing data, particularly before 2003, and this makes it difficult to assess the impact of ACGME duty hour standards on the trajectory of pass rates with the segmented logistic regression and also limits conclusions that can be drawn from secondary analyses. Additionally, having first-attempt performance would make for a more homogenous sample, as including repeated-attempt examinee data adds bias to the examination results. If the data supported our study hypothesis, there would also be higher pass rates for repeated-attempt examinees and total examinees. There is evidence that performance for repeated-attempt examinees has worsened since 2003, as demonstrated on the American Board of Ophthalmology Written Examination. Having a randomized prospective design would minimize the bias, but is not possible with board examination data. Finally, retrospective studies are more difficult for drawing cause-and-effect conclusions from the data. At the same time, the methods used in this study are the most appropriate way to show temporal differences surrounding the initiation of the ACGME duty hour standards.

The causes of the trends in board examination performance in the specialties in our study are unclear. Examinee performance is complex and can be affected by the quality of the candidates, improved surgical curricula, and the attractiveness of specialties. Trends in medical school applicants' grade point average, United States Medical Licensing Examination performance, and the competitiveness of surgical specialties represent an area that should be examined in future research in this domain.

Overall, the initiation of the ACGME duty hour standards has been associated with both positive and negative changes in certifying board examination pass rates across surgical disciplines and in both the written and oral examinations. The results of this study are important for individual physician examinees, surgical training programs, and surgical boards. The findings may be relevant to examination preparation by individuals, curriculum development by training programs, and examination development by surgical boards.