In 2013, the Accreditation Council for Graduate Medical Education (ACGME) implemented a new accreditation model that emphasized annual data-driven reviews and accreditation decisions, targeted feedback, and the freedom for thriving programs to innovate.1,2 Review Committees (RCs) are provided annual data in September and October of each year that allow them to identify overall performance, assess trends, and identify specialty and subspecialty programs that may be underperforming. Data reviewed include program characteristics, participating teaching sites, the clinical learning and work environment, changes in faculty and program leadership, resident attrition, scholarly activities, faculty and resident surveys, resident clinical experience, and program-level performance on the certification examination of the American Board of Medical Specialties member boards. RCs use this information to give each program an updated accreditation status annually. When the annual data suggest potential problems in a program, the RC may request clarifying information or schedule the program for a site visit. If problems are confirmed, RCs may issue citations for noncompliance or suggest areas for improvement. Programs are required to write a formal response to each citation, which is reviewed by the RC the following year.
The ACGME implemented the Next Accreditation System (NAS) using a phased-in approach. The 2015–2016 annual review cycle is the third review cycle for the 7 Phase I RCs, and the second review cycle for the 20 RCs in Phase II. It represents the first year of a stable state for NAS, as every RC had issued accreditation decisions through the annual review process for at least 1 prior year. Our work previously captured the outcomes associated with the transition of accreditation decisions in NAS and the presence or absence of citations.2 In this article, we study the accreditation decisions and citation resolution and/or extension for the 2015–2016 annual review cycle.
Methods
The annual review of programs begins in September of each year, and the vast majority of accreditation decisions are finalized by May 1. For this analysis, we determined the accreditation status of programs prior to the 2015–2016 annual review (the status of the programs in September 2015). To assess whether programs were identified as improving and moving to Continued Accreditation, or whether their performance was declining and moving to Accreditation with Warning (warning) or Probationary Accreditation (probation), we examined the status change of individual programs during the 2015–2016 annual review. Programs with a status of Initial Accreditation were excluded, as they do not yet participate in the annual data review. Additionally, we assessed the rate at which RCs issue and resolve citations by comparing the number and percentage of programs without citations upon entering the annual review to the number and percentage of programs with no citations following the annual review by RC. For the remaining analysis of citations, we looked at differences by specialty group: hospital-based, medical, and surgical RCs. Grouping RCs in this way is consistent with common themes and educational outcomes defined in the program requirements and organizational structure within the ACGME.
In the same way that we examine transitions in accreditation status, we studied how the population of programs with or without citations changed during the annual review process. We also reported the rate of citation resolution over a single annual review period at the program-level and for individual citations. Finally, we looked at the distribution of citations by status and specialty group.
Results
The number of ACGME-accredited programs on continued accreditation entering the 2015–2016 annual review cycle was 8915. Of these programs, 8729 (97.9%) had a status of Continued Accreditation, 153 (1.7%) were on warning, and 33 (0.4%) were on probation. Table 1 shows the change in status after the 2015–2016 annual review. Of the 153 programs initially on warning, improvement occurred in 54.5% of programs, 40.5% remained on warning, and 4.5% moved to probation. Of the 33 programs on probation at the start of the 2015–2016 annual review cycle, 18 programs (54.5%) improved to Continued Accreditation, 10 programs (30.3%) improved to warning, 4 programs (12.1%) remained on probation, and 1 program (3.0%) had its accreditation withdrawn. Across all specialties, the data showed a trend of improvement with 85% of programs initially on Probationary Accreditation progressing to Continued Accreditation or Continued Accreditation with Warning. The distribution of decision outcomes shows that RCs are consistent in their use of probation from year to year, with the percentage of all accredited programs remaining at less than 0.4% (33 programs in 2014–2015 versus 31 programs in 2015–2016). Decisions to remove, extend, or newly issue probation status occurred for 12 RCs during the 2015–2016 annual review. Of the 31 programs placed on probation in 2015–2016, 27 (87.1%) were newly identified through the annual review process (20 moving from Continued Accreditation to Probationary Accreditation, and 7 moving from Continued with Warning to Probationary Accreditation). In addition, 1 program previously on Probationary Accreditation and 2 programs previously on Continued Accreditation had their accreditation withdrawn. All decisions for probation or withdrawal follow a data-prompted site visit and review by the RC.
In addition to accreditation status transitions, our analysis of citations showed consistent patterns in the preannual review presence/absence of citations, and the resulting presence/absence of citations by the RC. Table 2 shows the percentage of programs entering the annual review cycle with no citations, and the percentage of programs with no citations following the annual review for each RC. Overall, approximately 90% of programs completed the annual review with no new citations issued and no citations extended. Table 3 shows the majority of programs (86.3%) do not have citations before or after the annual review (89.9%, 91.9%, and 68.4% among hospital-based, medical, and surgical programs, respectively). Of the remaining programs, 355 (3.9%) programs have citations entirely resolved, 313 (3.5%) programs have newly issued citations, and 552 (6.2%) programs had citations prior to and following their annual review. The specialty with the highest use of citations is neurological surgery, and the specialties with the lowest use are internal medicine, neurology, and psychiatry. RCs in the surgical group use citations to a somewhat greater extent than RCs in medical or hospital-based specialties, with RCs in surgical specialties newly issuing or extending citations for 23% of programs.
Table 4 shows the number of total prior citations during the 2015–2016 cycle (2701 citations across 907 programs), with 1920 (71.1%) of them declared resolved by the RCs and 781 (28.9%) extended during the review process. This pattern is consistent across the 3 specialty groups. In addition, 1641 new citations were issued during the 2015–2016 reviews. A closer look at the specific citations for the 33 programs entering the 2015–2016 review cycle on Probationary Accreditation showed that RCs were more likely to resolve citations than to issue new citations, with a total of 318 citations resolved versus 44 citations extended and 32 citations newly issued. This mirrors the overall tendency toward improvement in accreditation status. Overall, programs on probation commonly had citations in the area of evaluation, other than board certification performance (27 programs and a total of 61 citations), patient and procedural volume (21 programs and a total of 39 citations), and board certification performance (14 programs with a citation in this area). Review of citations codes and citation text for programs on probation revealed no specialty patterns across the 3 groups. The exception is the greater prevalence of deficiencies in procedural experience for surgical programs on probation. Seven of 10 programs on probation in the surgical group at the start of 2015–2016 had 1 or more citations for deficiencies in procedural experience, compared to 2 of 15 programs in the medical group, and 2 of 8 programs in the hospital-based program. The vast majority of these citations were resolved in the 2015–2016 RC review. Of a total 23 citations for procedural experience, 19 were resolved. One citation was extended, and 3 were newly issued.
In looking at the subset of previous citations for pass rates on the board certification examination during the 2015–2016 annual review, we found that of the 212 prior citations across 199 programs, the RCs resolved 71 (33.5%) and extended 141 (66.5%). The RCs also issued 115 new citations in this category, bringing the total number of programs with citations for pass rates on the board certification examination to 244 (less than 3% of all programs) at the end of the 2015–2016 annual review.
The final distribution of all citations by status and specialty group is shown in table 5. The final accreditation status correlates with the resulting number of citations. Across all programs with continued accreditation, there was an average of 0.14 citations per program, consistent with our findings for 2014–2015. Among programs with citations, programs with a status of continued accreditation had 2 citations on average, programs with warning had an average of 4.4 citations, and programs with probation were issued 8.7 citations on average. The averages for programs on Continued Accreditation and programs on warning were consistent across the 3 specialty groups, while the distribution of citations among programs on probation varied among specialties.
Discussion
While surgical RCs issue citations to a higher percentage of programs, there is a consistent pattern across the 3 specialty groups in terms of percentage of citations declared resolved, and the average number of citations by accreditation status. The exception is the small number of programs on probation. This reflects both individual RC's emphasis on various aspects of the program (clinical experience of the residents, scholarly activity, board pass rates, etc) and overarching consistency in the accreditation process.
One explicit goal of the new system is for RCs to improve programs more quickly through the annual review of programs on probation or warning to determine whether these programs are rapidly improving, or whether problems persist. Programs that move from Continued Accreditation to probation or to withdrawal of accreditation are identified more quickly than prior to the transition to NAS, when programs frequently did not undergo a review for a period of 3 to 5 years. During the transition to the new system, the community expressed concern that the annual reviews may not offer sufficient time to adequately address the citations. The high percentage of programs on warning and probation that moved to a status of Continued Accreditation, and the more than 70% of citations that are resolved from year to year, suggests that the annual review cycle provides adequate time for programs to make improvements in most areas. Some critical areas, such as addressing a lower pass rate on certification examinations, may require longer periods to demonstrate improvement; yet even in this area the data showed a clear tendency toward improvement.
Limitations of this article include the exclusive use of ACGME accreditation data, which may not capture all program deficiencies, and this study's focus on common patterns, which may not have captured all areas of interest to a specialty community. Further study is needed to assess patterns of improvement in response to citations. Beyond citations, this could include RC-identified areas for improvement that are communicated to programs. Moving forward, the findings will enhance the graduate medical education community's understanding of patterns and trends in RC identified deficiencies, and program responses to this element of the accreditation process.
Conclusion
Data on change in the accreditation status of programs and on citations patterns for the 2015–2016 annual review show that the majority of citations are resolved in a single annual review period. This provides evidence for the effectiveness of the new accreditation system in identifying areas of deficiency in accredited programs, and facilitating timely improvement in the majority of these areas.
References
Author notes
Editor's Note: The ACGME News and Views section of JGME includes data reports, updates, and perspectives from the ACGME and its review committees. The decision to publish the article is made by the ACGME.
