Implementing the Next Accreditation System: Results of the 2014–2015 Annual Data Review Open Access

On July 1, 2013, the Accreditation Council for Graduate Medical Education (ACGME) began the phased initiation of the Next Accreditation System (NAS). The goals of this new model of accreditation were to reduce the burden on accredited programs and transition to an accreditation process that could respond in a more timely and focused way to potential problems in the educational environment.¹  Seven specialties entered the NAS in 2013, and the remaining specialties followed 1 year later on July 1, 2014.

By moving away from episodic program site visits and reviews, and implementing a data-driven, annual review of performance, low-performing programs are promptly identified. Review Committees can then use requests for clarifying information and site visits, if necessary, to gather additional information and to provide targeted feedback to help these programs improve. Performance problems that would previously not have been identified until the next full review of a program can now be captured and addressed in a much more timely fashion. Concurrently, high-functioning programs are granted relief from adhering to all the detailed program requirements; the annual review of data for all programs allows the ACGME to extend the period between scheduled site visits to 10 years, reducing the burden of site visit preparation.

In the new model, the ACGME uses extensive data collected throughout the academic year to assess program performance against a set of standard performance measures. Data reviewed annually include information about the program characteristics, the clinical learning environment, program and institutional leadership, the environment of scholarship, the opinions of faculty and residents through ACGME surveys, resident clinical experience data, and first-time pass rate data provided to the ACGME by the American Board of Medical Specialties member boards. Review Committees assess the overall performance, looking at trends and instances of potential noncompliance with the accreditation standards, to identify underperforming programs. These programs then undergo an in-depth review to determine which ones are not meeting the core and outcome standards. In the new accreditation system, Review Committees have the ability to make accreditation decisions in a more proactive manner: identifying and addressing downward trends before true problems arise, as well as moving programs out of a “Probationary Accreditation” or “Continued Accreditation With Warning” status when they see improvements.

The annual review cycle begins in the fall of each academic year and officially ends once all programs have been issued an accreditation status decision, the majority of which are issued in the spring of the same academic year. In 2014–2015, the NAS was fully phased in and all Review Committees participated in the annual review cycle, the result being that for the first time in a single year, every residency program accredited by the ACGME (beyond initial accreditation) received an accreditation status decision. Programs applying for accreditation and programs having a status of “Initial Accreditation” or “Initial Accreditation With Warning” were fully reviewed and assessed for compliance to the standards, unchanged with the implementation of the NAS.

In this article, we compare the accreditation decisions and citation distribution among programs before the NAS implementation (pre-NAS) to the decisions and citations issued to programs following its full implementation in 2014–2015 (in-NAS).

We identified pre-NAS accreditation status decisions for all programs as the most recent decision made by the Review Committee before the first annual review cycle. To transition to the new accreditation system, the status of all programs with a 1- or 2-year cycle with “Continued Accreditation” was converted to “Continued Accreditation With Warning.” The annual review process was phased in over the course of 2 years, with Phase 1 Review Committees conducting their first annual review between October 1, 2013, and September 30, 2014, and Phase 2 Committees conducting their first annual review between October 1, 2014, and August 31, 2015. Our analysis used the decisions made throughout the 2014–2015 annual review cycle (the second cycle for Phase I programs and the first cycle for programs in Phase 2) to represent the full implementation of the new accreditation system. We assessed programs' status transitions from their pre-NAS status to the 2014–2015 annual review. We begin by comparing the distribution of pre-NAS and NAS accreditation statuses of all programs to identify changes in the overall population of accredited programs. Analysis was then limited to programs beyond a status of initial accreditation that underwent an annual data review. For this subset of programs, we compared the overall and by Review Committee pre-NAS and NAS percentage of programs on warning and/or probation. To assess whether programs were identified as improving and moving to “Continued Accreditation,” we examined the status change of individual programs. We also compared the average number of citations by accreditation status that Review Committees issued in the pre-NAS system to those in the new system. Finally, to assess whether the new approach resulted in less burden for programs, we examined the number of programs previously issued 0 citations and the number of programs issued 0 citations in the new system.

The number of ACGME-accredited programs before the implementation of NAS was 9472, and the number had grown to 9793 in 2014–2015, a 3.4% increase, while the number of programs with “Continued Accreditation With Warning” decreased from 304 programs pre-NAS to 155 programs (a 49.0% decrease), and the number of programs with “Probationary Accreditation” increased from 24 to 34 programs (an increase of 41.7%; table 1). For programs beyond initial accreditation, the proportions with “Continued Accreditation,” “Continued Accreditation With Warning,” and “Probationary Accreditation” changed slightly in the new accreditation system, going from 96.2% to 97.9%, 3.5% to 1.7%, and 0.3% to 0.4%, respectively (table 2). This distribution of programs by accreditation status varies across Review Committees. The specialties with the greatest increase in the absolute number of programs on probation were anesthesiology, neurology, and obstetrics and gynecology, each having an increase of 3 or more programs on probation in the new accreditation system. The specialties with the highest percentage of programs on probation were emergency medicine, family medicine, obstetrics and gynecology, ophthalmology, and thoracic surgery. The specialties with the greatest percentage decrease (more than 5%) of programs on warning were neurological surgery, obstetrics and gynecology, preventive medicine, and thoracic surgery. Family medicine, internal medicine, and pathology had the largest absolute decrease in the number of programs on warning.

When we assessed the transition of programs in and out of the “Continued Accreditation With Warning” and “Probationary Accreditation” status during the 2014–2015 review period, we found that just 34 of 328 programs (10.4%) remained on warning or probation at the end of the annual review cycle (table 3). Within that group, 3 programs on warning were identified as getting worse and were issued “Probationary Accreditation” by their Review Committees, while another 3 programs moved up from probation to accreditation with warning. Only a small subset of programs were withdrawn (either voluntarily or by the Review Committee) and the remaining programs, 264 of 304 programs on warning (86.8%) and 19 of 24 on probation (79.2%), moved to continued accreditation after the Review Committee conducted a full review of the program-level data. Additionally, of the 189 programs that were issued “Continued Accreditation With Warning” or “Probationary Accreditation” status in the 2014–2015 review cycle, 155 (82.0%) previously had a status of “Continued Accreditation.” These programs were newly identified in the annual data review as having areas of potential noncompliance and through an in-depth review, these performance deficiencies were confirmed by the Review Committees, justifying a new status of probation or accreditation with warning.

The analysis of the average number of citations for programs beyond initial accreditation showed a decrease for programs with “Continued Accreditation” and “Continued Accreditation With Warning” statuses (from 2.6 to 0.2 and 7.1 to 6.0, respectively; figure). For programs on probation, the average number of citations increased from 7.8 to 11.0. Before the implementation of the NAS, just 19.8% of all programs had reviews with 0 citations. In the new accreditation model, 89.9% of all programs received 0 citations during the most recent annual review (table 4). There was some variation among Review Committees, with programs in neurological surgery and colon and rectal surgery receiving citations at a slightly lower rate than pre-NAS, but at a much higher rate compared to other Review Committees. The Review Committees for medical genetics, obstetrics and gynecology, and diagnostic radiology issued the fewest citations during the 2014–2015 annual review cycle.

Our analysis has limitations, including making direct comparisons between the site visit–based approach for all programs in the prior system and the new approach to annual data review in the NAS. We will continue to measure the effectiveness and monitor the trends of the NAS over time, and will report the results of these analyses to add to the evidence of the effectiveness of the ACGME's new approach.

After more than a decade of planning, the ACGME's new accreditation system is fully operational. This novel, data- and outcomes-based approach to accreditation is focused less on process details, and more on the outcomes of that education and improvement to the learning environment. The new system allows for a high rate of turnover in the programs on warning and probation as the Review Committees can note more rapidly an improvement or decline in key areas of performance. The new model of accreditation has also resulted in vast reduction in the number of citations issued. While adapting to these changes in the short term may place some added burden on programs and Review Committees, the new approach has reduced the need for episodic site visits, has decreased the number of citations issued, and has created a space for improvement and innovation that did not exist in the past.