Facilitating Institutional Oversight and Program Improvement Through Educational Competency Committees

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to engage in annual program evaluation and improvement by a designated team¹  and in some cases, stipulates follow-up.²  While there are no requirements for sponsoring institutions to review the annual program evaluations, several programs^3–8  and institutions^9–11  reported benefits from internal reviews, which ACGME previously required for institutional oversight. Beginning in 2012, Duke University graduate medical education (GME) leadership revised an annual program evaluation template that it required programs to complete¹²  and adapted beneficial elements from its prior internal review process.¹³  We reconceptualized our previous internal review teams as educational competency committees (ECCs), which we modeled after clinical competency committees (CCCs). The ECCs are multi-person teams charged to judge program performance, just as CCCs regularly review and judge residents' performance.

This study seeks to answer 2 primary questions: (1) Does this new process add value? (2) Are the costs of the process acceptable? We believed 1 important metric of success was to preemptively identify programs before ACGME raised concerns, regardless of whether these would result in citations, require progress reports, or lead to data-prompted site visits. The concerns result in measureable additional time and effort for programs and institutions.

Duke University Hospital is the sponsoring institution for more than 80 ACGME-accredited and 70 non–ACGME accredited programs. Beginning in 2005, Duke's GME Committee required programs to submit their annual program evaluations for review by the section head of its Program Oversight Committee, who reports to the designated institutional official. In May 2012, in preparation for the next accreditation system, we streamlined our template to focus on the 5 required components of the annual program evaluation. We used the GME management system (MedHub) for submission and review. In 2014, ACGME resident and faculty surveys and institutional duty hour data were provided to ECCs (table 1).

Educational competency committees used the color codes from the hospital's clinical scorecards to judge each component: not compliant (red), minimum compliance (yellow), substantial compliance (green), and exceeds compliance (blue). The ECC composition mimics our previous internal review team structure and includes a program director, a program coordinator, a resident, and a member of Duke's Office of GME. That position rotates among a senior administrator, a doctoral trained educator, and a prior program director.

Like CCCs, ECCs utilize the wisdom of a team to reach consensus on performance. Each ECC member received 5 to 9 annual program evaluations electronically to review in advance of the meeting, using a checklist of performance criteria. At the meeting, the ECC developed a consensus that was entered directly into the GME management system. The Program Oversight Committee Section Head reviews the results, applies the special review (SR) criteria, and identifies programs that were required to have an SR. Programs received results through the GME management system. Deidentified annual program evaluation summaries were aggregated into an institutional dashboard and trended by year (provided as online supplemental material). Vertical rows indicate successes and opportunities for program improvement and horizontal rows indicate successes and opportunities for institutional improvement.

Educational competency committee members were surveyed annually, and their feedback was used to improve the subsequent year's process. The survey asked questions, such as the amount of time spent on pre-review and whether the experience enhanced the participant's professional development and benefited their own program. The survey questions were developed by the authors (K.M.A., A.N.) without any testing.

The study was determined exempt by the Duke Medicine Institutional Review Board.

We considered the decision to require an SR as a “screening test.” We analyzed if those programs that underwent SRs were more likely to have ACGME citations, required progress reports, or data-prompted site visits. We constructed 2 × 2 tables of SR + (SR required) or SR – (SR not required), and then correlated these with ACGME accreditation decisions (table 2).

We calculated review costs using the educational relative value unit assigned by our medical school for the educational effort of faculty and administrative effort by coordinators. The section head received a small stipend for overseeing this process, as previously provided for overseeing internal reviews. Resident time was calculated by using the reimbursement model for internal moonlighting.

Table 2 summarizes our experience from 2012 through 2014, the 3-year period of the next accreditation system's implementation. A total of 237 annual program evaluations were reviewed over this period, and more than 100 program directors, coordinators, and residents participated. The time ECC members utilized to review materials before the meeting increased from 3 to 5 hours, while the meeting time decreased from 4 to 3 hours.

Educational competency committee members reported via the annual survey that their participation enhanced their professional development (95% in 2012, 86% in 2013, and 94% in 2014); and the majority reported that participation benefited their program (95% in 2012, 60% in 2013, and 88% in 2014).

We created specific thresholds for noncompliance, for example, a difference in 0.1 on the resident or faculty survey, board performance below the national rate, or failure to evaluate progress on the prior year's action plan. Based on their annual program evaluation, 6 programs (8%) in 2012, 2 (2%) in 2013, and 8 (10%) in 2014 received an overall rating of noncompliant. In 2012, we did not conduct special reviews as criteria were in development. A total of 8 SRs (10%) were conducted in 2013, and 13 (16%) in 2014. Thirteen programs (17%) were judged to have exceeded compliance in 2012, compared with 12 in 2013 (15%) and 1 (1%) in 2014.

In 2013, the total cost of the review was calculated at $23,081.33 ($285 per review) and $22,624.90 ($276 per review) in 2014 (table 3), or approximately $280 per program over the 2-year period. The only additional funding required was $95 per program to support resident participation.

In 2013 and 2014, the sensitivity of the special review was 50% and 83%, the specificity was 92% and 89%, and the negative predictive value was 97% and 99% (table 4). The aggregated scorecard from 2012–2014 is provided as online supplemental material.

We successfully modified the institution's prior internal review and oversight of the annual program evaluation processes to create ECCs. Educational competency committee members perceived value from participating and predicted future benefit to their programs. The activity itself served as professional development. One participant noted that “it provided . . . an opportunity to see behind the curtain.”¹⁴ 

We believed we captured the essential components of ACGME's published program evaluation template.¹⁵  Although there is no validity evidence for our ECC assessments, use of 9 to 12 assessments are known to increase validity in evaluating residents' performance.¹⁶  Our ECCs use at least 9 different assessments to evaluate program performance.

The ACGME has posted a resource for aggregating and monitoring action items from successive annual program evaluations.¹⁵  Our color-coded scorecards were valuable and familiar in our setting, as they are used widely for our health system's performance. They facilitate rapid visual identification of program and institutional successes, opportunities for improvements, and trends.

Experts in medical education suggest suboptimally performing residents fail to accurately assess their own performance, and their inability in accurately perceiving their performance impedes their ability to improve it.¹⁷  This also seemed true of our programs. Programs that reflected conscientiously on their performance submitted more robust action plans.

The number of SRs increased from 2012 through 2014, largely due to the inclusion of the resident survey and our selection of the threshold of concern as any deviation of ≥ 0.10 from the mean. We may have set the bar too high, in part because specialty-specific resident survey averages were not available. We continue to refine our calibration, minimize administrative burden on programs, and identify those that benefit from the deeper dive of an SR.

We found it reassuring that our ECCs require 1 to 2 hours less than the 10 to 12 hours required for the prior internal reviews, and that programs spend far less time in preparation. We believe we can further optimize efficiency by assigning the GME associate director to review all programs' ACGME Accreditation Data System information and duty hour documentation, and report only concerns to the ECCs.

We have attempted to determine the “educational sensitivity and specificity” of our decision to require an SR. The educational sensitivity of the SR increased after we added resident and faculty surveys to the review. The specificity remained approximately 90%, and the negative predictive value was ≥ 97%. In 2014, the ACGME conducted an early site visit on 1 program that we did not identify as needing an SR. Other programs that subsequently underwent an ACGME site visit, or were asked to submit a progress report, benefited from the preparation facilitated through the SR. The positive predictive value rose slightly but was still low. We required 8 programs to have SRs that subsequently had no issue identified by the ACGME, but we prefer to set our standards somewhat higher.

There are limitations to our study. Our experience reflects only a single institution. Costs will vary across institutions based on different faculty and administrative compensation models. Our survey lacked validity evidence and ECC members may not have interpreted the questions as we intended.

In the coming year we will incorporate core faculty into ECCs, individually “debrief” ECC findings with all program directors as a mentoring tool, and add questions about program activities related to the Clinical Learning Environment Review to our template.

We merged elements of our prior internal review and annual program evaluation processes to create ECCs, which provided peer-review professional development and identification of innovation and best practices. Sensitivity, specificity, and predictive value were acceptable.