Background

The Accreditation Council for Graduate Medical Education (ACGME) mandates that sponsoring institutions conduct internal reviews. In 1998, the ACGME Institutional Review Committee gave Duke University Hospital a citation for an inadequate internal review (IR) process. Since then, we have instituted several iterative changes. We describe the evolution of Duke University Hospital's current internal review process.

Intervention

We implemented a new review team composition, template report, use of the program information form, and centralization of documentation to improve our internal review process. In 2007, a more formal evaluation of the outcome and impact of these changes was instituted. This included a yearly survey of all participants and review team members, a review of programs, and a tracking process for the decisions of our Graduate Medical Education Committee (GMEC) on the status of reviewed programs.

Results

Participants from both the program under review and the review team evaluated the process favorably. Review teams reported they learned from the best practices of the program being reviewed. Program directors from the reviewed programs reported the process improved their documentation. Both groups reported the process better prepared them for their next ACGME Review Committee site visit. The GMEC has recommended “probationary sponsorship” for fewer programs since the IR process changes have been implemented. The IR process was recognized as a best practice in Duke University Hospital's 2004 ACGME institutional review.

Conclusion

We believe our IR process, review-team composition, template report, program information form, and centralized documentation now fully meets accreditation standards. Participants are reasonably satisfied and report value from the process. More programs are judged to be within substantial compliance by the GMEC.

Editor's Note: The online version of this article includes additional supplemental materials.

The Accreditation Council for Graduate Medical Education (ACGME) mandates that sponsoring institutions conduct internal reviews (IRs) to systematically and comprehensively review their programs. There is, however, limited research available about IRs. The literature indicates that a robust IR process is important to meeting ACGME requirements.1 Another article shared the design of a simplified IR that offers meaningful feedback and allows for continuous quality improvement,2 and a third article describes a system of peer review to meet the ACGME internal evaluation requirements and an administrative structure to support the process and its outcomes.3 

The ACGME Review Committee citations can be avoided by addressing poor board performance, lack of rotation objectives, missing patient types or procedures, inadequate scholarship, limited ambulatory experiences, and inadequate teaching and supervision.4 The IR provides a systematic opportunity to identify these types of issues precisely. When identified by the IR, staff has 1 year or more to develop and implement action plans to proactively bring programs into substantial compliance, thereby avoiding subsequent ACGME Review Committee citations.

However, the IR provides additional opportunities. It facilitates program and institutional self-assessment, aggregates educational outcomes, and models 1 of the ACGME 6 core competencies—practice-based learning and improvement. Benefits of IRs include providing an opportunity to (1) meet ACGME requirements for programs and institutions, both by correcting any lack of substantial compliance and by clarifying expectations; (2) serve as a “dress rehearsal” for the ACGME site visit; (3) facilitate continuous institutional education quality improvement; (4) identify common challenges for which institutional resources may be identified and deployed; (5) collect information on best practices that can be celebrated and/or shared; (6) publicly recognize program excellence; (7) promote collegiality and demonstrate transparency of program evaluation; (8) adjust to changing ACGME program and institutional standards; and (9) facilitate professional development for program staff and review team members, including residents.

Duke University Hospital (DUH) is the sponsoring institution for 73 ACGME-accredited programs, 1 non-ACGME accredited program, and approximately 50 institutionally sponsored programs with more than 950 residents in 13 clinical departments. In 1998, DUH received an institutional citation for an “inadequate internal review process.” The Institutional Review Committee (IRC) cited the lack of an adequate written protocol, inconsistent summaries, and frequently, an incorrectly constituted review team. This provided DUH with its first opportunity for a thorough redesign of the process. In 2001, DUH received a second institutional citation related to its internal reviews. To some extent, this was because of insufficient time for implemented changes to be reflected in subsequent IRs. Individual programs also received citations from their Review Committees for the inadequate institutional process. We anticipated that programs and review participants would benefit from change and believed the selection of our review team was an important improvement opportunity. This article outlines the strategies DUH employed to meet the ACGME institutional requirements for the IR and discusses the impact at our institution.

Internal Review-Team Composition

In 2000, DUH's designated institutional official (DIO) reorganized the Graduate Medical Education Committee (GMEC) into 4 subcommittees, 1 of which, the Program Oversight Section, assumed responsibility for IRs. The DIO selected a seasoned former program director as section head. The DIO, the Office of Graduate Medical Education (OGME), a newly recruited associate director of graduate medical education, and the section head created a formal protocol that standardized the composition of the team, centralized data collection and organization, and developed a template to be used for the written report.

The new approach was ratified by the GMEC. The IR team composition was changed to include a program director from another program as chair, an OGME representative, and a resident. An IR specialist in the OGME selects a program director from a different department on a rotating basis. Program directors can expect to be reviewed, as well as chair an IR team, during their own program's accreditation cycle. The schedule is published 1 year in advance.

Each team includes an OGME representative (the educator, associate director for graduate medical education, or administrator), whose presence on the team provides consistency because each of them participates in one-third of the reviews. They also provide ongoing support and resources for the program under review. In January 2007, a program coordinator was added to the team. Commonly, this individual is the coordinator of the same program as the program director chair.

In July 2007, OGME began compensating residents who served on IR teams as internal moonlighters. Program directors and coordinators already receive some institutional support for their roles, so no additional compensation was provided to them.

Document Revision

The OGME revised a number of documents in January 2007 to organize and guide the new IR process. These include a template for the report regarding the items that must be reviewed and documented (the reference) and a notebook for IR team participants (supplemental online appendix 1).

Programs reviewed are required to submit a completed, updated, program information form (PIF). Before the review, the IR Team is provided with the PIF, along with the other materials listed in supplemental appendix 1.

In addition, the OGME began tracking and classifying the citations from the prior year's IRs according to the classification system ACGME uses in the institutional review document. This chart, along with a companion chart of ACGME citations for the year, is shared with all DUH's graduate medical education programs annually and is reviewed by the IR team. This format allows challenging areas across programs to be more readily identified and OGME to prioritize its support.

Internal Review Flow Chart

Internal Review Format

 The IR process is shown in the figure. Each IR is scheduled for 3½ to 4 hours. The review team meets for the first 15 to 30 minutes to discuss the major issues they discovered, based on their review of the documentation provided, and to strategize how to proceed. The next hour is spent interviewing residents. The second hour is spent with the program director, coordinator, and key faculty. During the last hour, the team reviews trainee files and collaboratively begins to write the report.

The OGME IR specialist develops and distributes a schedule of IRs for the year. The schedule is included in the GMEC minutes and assigns individuals to the review team by date and program. The IRs are scheduled Wednesdays from 8:30 am to 12 pm, and IR team members have ample notice to schedule the review around their other responsibilities. The OGME IR specialist prepares and distributes the documentation to the team at least 1 week in advance.

IR Team Notebook Contents

 In March 2007, the DUH IR template was revised to include an option for the team to make a “recommendation for sponsorship” to the GMEC. Analogous to ACGME's decisions of continued accreditation and probationary accreditation, the IR Team recommends continued sponsorship or probationary sponsorship. The team may also recommend withdrawal of sponsorship. Withdrawal of sponsorship means the program will no longer receive institutional support and sponsorship, leading to the closing of the program. Continued sponsorship means the program is in substantial compliance. There may be minor revisions or changes to make before the next internal review (usually required as follow-up 6 to 12 months after the report is presented to the GMEC). Probationary sponsorship implies the program is not in substantial compliance. This status is recommended for programs that fail to remediate prior ACGME citations or items identified during the prior IR. It also is considered if a program lacks foundational educational processes, such as inadequate goals, objectives, or documented feedback or has duty-hour violations or other significant concerns in its learning environment. The report (supplemental appendix 1) includes a representative list of required action plan items that the program reviewed may be required to submit, usually in 6 to 12 months. The terminology recognizes ACGME “accredits” and sponsoring institutions “sponsor” programs, but we have attempted to use similar language to communicate what it hoped to be a similar level of concern. The review team can propose required follow-up for programs that receive continued sponsorship (depending on the extent and seriousness of citations). Follow-up is required for any program that receives probationary sponsorship.

IR Follow-up

The IR follow-up may include a complete internal rereview. Alternatively, the team may accept a written report, a meeting with the program director and coordinator, a meeting with residents, review of resident files, or any combination of the above. Once the written IR report is completed, it is sent to the head of the GMEC program oversight section. If the head concurs with the report, it is sent to the program director of the program under review before it is discussed and voted on by the GMEC.

Before late 2009, the GMEC voted through a show of hands. Presently, voting members use an audience response system to record their vote, allowing for anonymity. A summary of the discussion and the results of the voting become part of the GMEC's minutes. As a standing committee of DUH, the Executive Committee of the Organized Medical Staff reviews the minutes monthly. This committee includes all department chairs and the chief executive officer. The DIO and an elected resident representative also attend and can address any questions. This adds an additional element of accountability to the process.

Beginning in 2007, OGME surveyed all IR participants on an annual basis. An electronic evaluation developed in SurveyMonkey (Palo Alto, CA; www.surveymonkey.com) is sent by e-mail to the program director and coordinator of the program under review and the members of the review team. Excluding the OGME members, responses are reported only in the aggregate. The DUH Institutional Review Board exempted this initiative. The results (table) are presented as counts and percentages without formal statistical testing. Because relatively few individuals are surveyed each year, an examination of response trends over years was not conducted, and data were instead compiled over all years. The table shows a comparison of review team members perceptions and those of the reviewed programs about the usefulness of the process. The usefulness responses were considered ordinal categorical in the 3 levels (not at all useful, somewhat useful, and very useful). They were evaluated using the Cochran-Mantel-Haenszel row means score test with 1 df.

Sixty-one internal reviews, representing 84% of DUH's ACGME-accredited programs, were conducted between January 2007 and June 2009. Since the category of probationary sponsorship was implemented in March 2007, the percentage of programs recommended for this status has decreased each year (41%, 20%, 4%, respectively).

Review teams members were also asked to assess the usefulness of specific components of the process. Team members reported that it was somewhat or very useful to have the materials provided ahead of time, to interview trainees and program directors, to review trainee files, and to discuss the findings with other members of the review team.

The table displays the distribution of the 3-level ordinal responses for the usefulness items by the review team versus reviewed program. Most review team members and participants from the reviewed program judged the experience somewhat useful or very useful. A greater percentage of review team respondents than program reviewed responders, reported the IR process was very useful for 5 of 9 areas queried (P < .05). Differences were found in the following areas: (1) helping gain a better understanding of ACGME program requirements; (2) helping identify ways to teach core competencies; (3) helping identify and implement ways to evaluate trainees, faculty, and programs; (4) learning about best practices; and (5) learning about new resources.

Representative responses from open-ended questions on the evaluation are displayed in supplemental online appendix 2. Comments from the reviewed program and the review team are similar. All respondents were able to identify 1 thing they had done differently in their program as a result of their participation in the IR.

We feel we have greatly improved the quality of the IR process so that it not only meets ACGME institutional requirements but also, and just as important, is viewed as useful and educational for all participants, the program under review, and the IR team.

Interestingly, it is the IR team that reports the most benefit. Although the goal of an IR is to evaluate the status of the reviewed program, our team members perceive the process as useful to them as they develop a greater appreciation for program requirements and documentation expectations and resources they can apply to their own programs.

Although many institutions use a separate, discrete group of faculty to conduct all the IRs, we believe the use of program directors, coordinators, and residents serves as an important development opportunity. During the 3 years of our study, more programs were perceived to be of better quality by the GMEC. We believe this is a result of programs taking the IR process more seriously, additional efforts by programs, and greater collaboration with OGME, such as through program director and coordinator development.

Each year, responses to the question, “How could the internal review process be improved?” are analyzed by the program oversight section. Improvements are implemented the following year, when possible. In 2009, the timeliness of the final report was addressed based on participant feedback. The review team was required to submit its final report for presentation to the GMEC within 1 month of the actual review. In 2004, the IRC identified our IR process as a best practice, and no programs have received an ACGME Review Committee citation based on the IR since that time.

We continue to face time challenges as we depend on the generosity and professionalism of our program directors, coordinators, and residents. Some of the strategies to improve this process include scheduling well in advance, providing organized documentation for review, providing a template to facilitate the written report, and making participation on an IR team an expectation of all program directors and coordinators at least once every 4 to 5 years. We estimate 25% of a dedicated OGME team member (the GMEC specialist) is necessary to provide organization, reminders, scheduling of rooms, and administrative support to the reviews and subsequent follow-up.

The composition of the review team has enhanced the process and benefited participants. Since making this a moonlighting opportunity, there has been no difficulty in recruiting a willing resident member. Indeed, many have become skillful, knowledgeable participants, adept at the process, and more cognizant of educational and accreditation issues. We look forward to “growing” future program directors from this cohort. Adding a coordinator has also proven extremely beneficial. Coordinators are often the most thorough reviewers of program files and documentation and can share ideas from their own experience with the program being reviewed.

Selecting another program director to serve as chair of the review team had a dual benefit. It makes the process a true peer-reviewed activity, and it helps them stay in tune with evolving accreditation and institutional requirements. Requiring the program under review to complete a full PIF has constituted a great deal of work, but overall, it has been assessed as beneficial. Programs report that working through a PIF helps identify areas for improvement and provides a head start on completing the PIF for the next ACGME site visit. The template the review team must use for their final report, although long and detailed, provides explicit guidance for the team and a structured, standardized evaluation of the program. These requirements, in addition to the regular schedule for the actual review, bring consistency, minimize anxiety, and made the process more user friendly.

The greatest challenge has been in the designation of probationary sponsorship. The emotion around the word probation has been significant. Some program directors have embraced probationary status, believing it facilitates their negotiations with chairs or division chiefs in obtaining needed resources. Frequently, a probationary status is assigned because of educational concerns that are beyond the control of a single program director. Unfortunately, some program directors have taken the designation probationary sponsorship personally, convinced that it would affect resident recruitment and program morale. The GMEC members are committed educators who respect their colleagues' work, which sometimes makes it difficult to offer constructive feedback. This is similar to the challenges faced when attempting to give instructive feedback to residents or faculty colleagues in our program who have one or more areas of suboptimal performance. Employing an audience response system has helped because individuals can remain anonymous. There continue to be a small number of concerned GMEC members who are part of an ongoing dialogue about program quality and the fairness of the process. We continue to struggle with objective criteria that would make the decisions feel less “pejorative” at the same time as differentiating high-performing programs from those which require improvement.

Our study has several limitations. We cannot be certain the perception of the process benefits, which occurred over this same time period, were a direct result of the changes implemented. Simultaneous with our intervention, DUH's OGME enhanced other faculty and coordinator development efforts, and national organizations and program director societies offered development opportunities.

Institutional citations regarding our suboptimal IR process motivated us to comprehensively redesign the process, documents, and review team. Through numerous iterations, the process now not only meets ACGME institutional requirements but also is believed to be beneficial to both the review team and the program being reviewed. Participants were able to identify at least 1 thing they will do differently based on their participation, and most reported a positive change in education, assessment, and or educational climate. The more rigorous IR process has helped programs to be in a state of perpetual readiness, keeping their PIF up to date, enhancing their current documentation, and accepting a process of ongoing improvement. The number of ACGME and IR citations has decreased.

Unfortunately the label of probationary sponsorship continues to stir emotions. We continue to work at providing truthful, constructive feedback, believing it is equally respectful to tell our colleagues when we believe they fall short as when they excel. Our process continues to improve, reflecting participant feedback, changes in program requirements, and evaluation of program quality outcomes. Ongoing challenges include ensuring timely reports to the GMEC despite scheduling the meetings a year in advance and providing central coordination. Future research could compare the outcomes of the subsequent site visits and ACGME accreditation decision to assess the effect of the IRs.

1
Collins
,
J.
Internal program review.
Acad Radiol
2003
.
10
(
1
):
S44
S47
.
2
Hartman
,
D. S.
The internal review: simplifying a tedious task.
Acad Radiol
1997
.
4
:
67
68
.
3
Levine
,
H.
,
E.
Vanek
,
G.
Lefferts
,
W.
Michener
, and
G.
Weiker
.
A peer review process to assess the quality of graduate medical education.
J Med Educ
1988
.
63
(
4
):
288
293
.
4
Beasley
,
B. W.
,
D. R.
Scrase
, and
H. J.
Schultz
.
Determining the predictors of internal medicine residency accreditation: what they do (not what they say).
Acad Med
2002
.
77
(
3
):
238
246
.

Author notes

All authors are at Duke University Hospital, Office of Graduate Medical Education. Kathryn M. Andolsek, MD, MPH, is Associate Director; Alisa Nagler, JD, EdD, is Assistant Dean, Graduate Medical Education; Leslie Dodd, MD, is Program Oversight Section Head; and John L. Weinerth, MD, is designated institutional official.

The authors would like to thank Leslie Johnson for serving as OGME specialist in coordinating the Internal Reviews. The authors would like to acknowledge Rick Sloane for his thoughtful questions regarding the purpose and outcomes and his statistical analysis of the quantitative data. The authors would also like to thank Joanne Schlueter for assisting with data collection and analysis.

The study was exempted by the Duke University Institutional Review Board.

The methods and preliminary results of this study were presented in poster format at national conferences before this submission.

Supplementary data