Practice-Based Learning and Improvement for Institutions: A Case Report

The University of Virginia (UVA) Medical Center includes a 577-bed hospital, a Level I trauma Center, and multiple ambulatory clinics and imaging centers, which serve Virginia, West Virginia, Tennessee, and North Carolina. Currently, UVA sponsors 775 graduate medical trainees in 67 Accreditation Council for Graduate Medical Education (ACGME)–accredited programs, 31 additional fellowships (either nonaccredited or accredited by other groups than the ACGME), 1 American Dental Association–accredited dentistry program, and 5 paramedical programs in chaplaincy, clinical laboratory medicine, clinical psychology, pharmacy, and radiation physics. The UVA Health System is centered in the university town of Charlottesville, with 14 participating clinical sites.

In July 2005, UVA underwent its regularly scheduled institutional ACGME review. The Institutional Review Committee (IRC) cited 6 areas of deficiency and proposed the status of probation. Crucially, the designated institution official (DIO) and the graduate medical education committee (GMEC) had not performed the required number of program internal reviews (IRs), which had also been a previous citation. Although the DIO and GMEC were given the opportunity to rebut the citations and proposed status change, the IRC upheld 4 of the citations, and the accreditation was changed to probation in May 2006.

In addition to delinquent IRs, the institution was noted to be deficient in several other key areas (table). The IRC cited an unfavorable balance between service and education, manifested in both noncompliance with duty hour standards, and the lack of a safe and efficient work environment for trainees. Additionally, master affiliation agreements were unsigned or out of date, and there was concern that trainees were not provided the opportunity to participate either in scholarly activity or on institutional committees. All of the citations pointed to a critical lack of structured oversight by both the DIO and GMEC.1 

The consequences of an institution being placed on probation by the ACGME are serious. All current GME trainees, as well applicants who have been granted an interview, must be notified of its probationary status. In addition to the potential adverse effect on recruitment, there is a risk of loss of accreditation of not only the institution, but also all of its accredited programs if corrective action is not taken.

Most of our citations arose from insufficient oversight of operational activities of the GME office and poor communication among the GME office, administrators, faculty, and trainees. We underwent rapid self-assessment and implemented necessary corrective actions in order to have the status of accreditation restored as quickly as possible. In essence, the institution endeavored to become competent in practice-based learning and improvement (PBLI).

The ACGME's definition of PBLI is multifaceted.2 By replacing “patient” or “patient care” with “GME trainee” or “graduate medical education,” the central core of this competency becomes relevant to the actions of the DIO and GMEC. We define PBLI for GME as thus: “Constituents of Graduate Medical Education must demonstrate the ability to investigate and evaluate their education of GME trainees, to appraise and assimilate scientific evidence, and to continuously improve Graduate Medical Education based on constant self-evaluation and life-long learning. Those involved with Graduate Medical Education are expected to develop skills and habits in order to meet well-defined goals that will ensure that the Institutional Requirements set forth by the ACGME are met.” This competency became our guiding principle in developing corrective plans of action and undertaking the following PBLI activities:

1. Identifying strengths, deficiencies, and limits in one's knowledge and expertise. A Performance Improvement team was created to evaluate and appraise all administrative, financial, and legal responsibilities necessary for ACGME compliance. There were 6 regular team members, including the newly appointed DIO and members of the Medical Center's Performance Improvement, Quality, and Finance offices. The Performance Improvement team met weekly for 3 hours until the time of the next IRC visit. In addition, we commissioned an independent audit by the university. Key findings from both teams were reported to the DIO, GMEC, and senior leadership of the Medical Center and School of Medicine.

2. Setting learning and improvement goals. We fully engaged the Housestaff Council to organize monthly town hall–like meetings for the trainees, which were attended by the DIO, chief executive officer, and dean. Trainees felt that issues raised during the 2005 site visit arose from the frustration that service burdens detracted from their education. They offered multiple specific solutions to help the Medical Center operate more efficiently. For example, 2 issues prevented their timely discharge of patients: (1) the lack of a discharge planner on every unit and (2) laboratory results unavailable early in the morning. Goals were set to implement necessary changes in staffing and laboratory workflow. Three town meetings in total were held. During the final 2 meetings, the effectiveness of the enacted changes was reassessed. Trainees reported satisfaction with the results and stated that they had resulted in significant improvements.

3. Systematically analyzing practice using quality improvement methods, and implementing changes with the goal of practice improvement. The Performance Improvement team investigated deficiencies and worked with departments around the Medical Center to execute rapid modifications. The GME office and a fitness room for residents and fellows were refurbished, and GME office staff was increased by 2.5 full-time employees to allow all functions to be fully supported. In addition, affiliation agreements were updated, locks on call room doors were replaced, new computers were installed, scheduling of magnetic resonance imaging became more flexible, additional ultrasound technicians were hired for the Emergency Department, and reporting of radiology and pathology results were streamlined and made readily accessible by all trainees.

Most importantly, the GMEC underwent significant restructuring in order to focus close attention on its IR process and oversight of duty hours. Our IR process was completely overhauled, and a full-time employee was hired to maintain the calendar, form the teams, schedule the reviews, compile the materials, and write the reports for the committee. Our template was reformatted to be compliant with ACGME requirements. Action plans were defined, and deadlines for completion were marked on future GMEC agendas. In addition, the DIO and GME office staff and Subcommittee on Duty Hour Monitoring began to review trainee duty hour logs and send alerts to programs whose residents were in danger of committing a violation. Duty hour violations reported during an IR, Residency Review Committee letter of notification, annual exit survey, or the ACGME anonymous survey were then brought to the program director, who was required to report to the GMEC with a corrective action plan and deadline for its implementation.

4. Identifying and performing appropriate learning activities. We attempted to embody a commitment to GME, in some instances by fortifying programs that were already in existence. For example, we sponsored the GME Innovative Grant Program, which awards 2-year grants of $50 000 to faculty members to develop and apply innovative educational techniques in graduate medical education. New programs were also created to fill gaps, including the development of an institutional curriculum to organize and present topics relevant to trainees in all disciplines (physician wellness, fatigue, ethics, career development, quality, and health care reform). We also initiated a certificate program in GME that provides an opportunity for all interested trainees to earn 12 university credits toward 1 of 3 tracks: public health, clinical research, or global health.

These improvements came at substantial cost to the institution, including approximately $200 000 in additional salary support for GME office personnel and ultrasound technicians, $100 000 to refurbish the GME office and the fitness room, and $5000 in computers and new door locks. The Medical Center bore these expenses and viewed them not as an expansion of GME at UVA, but as necessary for the correction of our deficits.

The summative effect of our PBLI activities was to not only remove the deficits that led to probation but to have full accreditation restored in October 2006, 6 months after the IRC's action to sustain its original decision. No new citations were received, and all previous citations were removed except for (1) off-cycle IRs and (2) duty hour violations that had been noted in IRs conducted before institutional corrective action had been taken. The IRC acknowledged this by stating, “the effectiveness of the new monitoring system has yet to be proven although it is clear that the Institution has focused resources in this area.” Moreover, our actions substantially enhanced the environment for GME at our institution. Residents have continued to state satisfaction with the operations of the GME office, our average cycle length has increased from 4.0 to 4.2 years, and the 2007 match was the institution's most successful ever, with all programs but one (preliminary surgery) filling on match day with high-quality, geographically diverse applicants.

Our process of self-assessment led us to the following conclusion: most of our citations may have been prevented by regularly practicing PBLI at the institutional level. This would have included scheduled assessments of the operational and administrative activities of the GME office and GMEC, and implementation of corrective action when necessary. In essence, we recognized the need for IRs for our institution.

Although it is mandatory that each institution report annually to its operating medical staff, there is currently no requirement that a systematic review examining all key aspects of an institution's GME program be performed at timed intervals. As the latest reporting of common citations from the ACGME3 revealed that many institutions were cited with deficiencies similar to those that had led to UVA's probation, we surmise that each could significantly benefit from conducting a similar systematic review geared toward discovering and correcting institutional issues.

There are 3 potential methods for conducting an institutional IR. The first would be to develop a team from within the institution. However, most of those with significant knowledge or expertise regarding the ACGME's institutional requirements (the DIO, GME office administrative personnel, or senior program directors) would inherently possess a conflict of interest that would potentially prevent the type of rigorous and unbiased audit needed. An alternative would be to have a team composed of members from within the institution but outside the GME community, such as the compliance office or Audit Department. Although this method would generate little additional expense, these offices are generally not charged with oversight of GME and may lack the necessary knowledge and perspective to conduct an insightful review.

Professional consulting groups also offer the service of reviewing GME programs. They have both the expertise and objectivity to conduct thorough reviews for a broad range of institutions, from single programs to large academic medical centers. The major disadvantage of this approach is the expense. In preparation for our IRC visit, we used a consultant whose fees totaled $12 000. Some of our residency programs have also used external consults with costs ranging between $10 000 and $20 000. This sum may be prohibitive for GME offices with constrained budgets.

A third possibility, using another institution's DIO (or chair of its GMEC) to conduct the review, may be optimal. External DIOs could volunteer on the basis of geographic proximity or by similarity of size or structure of the academic medical center. The visiting DIO would be well positioned to objectively critique another institution, and could also share the knowledge of “lessons learned” and best practices from his or her home institution. The additional advantage of such a program would be the strengthening of a network of institutional GME programs, which would facilitate sharing assistance for matters routine in nature as well as provide crucial help in times of crisis (eg, catastrophic events, such as Hurricane Katrina). Although this would require a time commitment of 2 to 3 days by the visiting DIO, he or she would eventually benefit from the efforts of another DIO at the time of his or her own institutional IR. We know of several instances where this approach has been used, including one of us (S.E.K.) conducting institutional IRs for several large academic medical centers. In the one instance to date where an IRC decision has been rendered after using this method, the institution received a 5-year accreditation cycle with only commendations and no citations.

Although each institution may want to tailor its approach, at a minimum the institutional IR should include on-site interviews with the DIO and GME office staff, members of the GMEC, senior leaders of the institution (chief executive officer and/or dean), and a group of peer-selected residents. It should also contain a review of the following:

• Institutional review document, updated from the last ACGME site visit

• Response to institutional citations

• Affiliation agreements

• Institutional statement of commitment to GME

• Institutional organizational chart

• Program-specific citation and response summary

• Relevant institutional GME policies, including disaster, vendor, and grievance

• Review of resident contract

• Resident membership on committee and councils

• GMEC membership and minutes

• IR protocol, schedule, and reports

• Program accreditation letters

• GMEC procedure for reviewing duty hours and ensuring compliance

• Institutional educational programs (minimally physician wellness and fatigue)

• Institutional programs for faculty development

• Communications, including annual report, with the Organized Medical Staff

The written report should model the one required for program IRs and include the date of the review, the names and titles of those interviewed, documents reviewed, a description of corrective action taken since the last ACGME visit, and, critically, a list of areas of concern that must be addressed before the next visit. The report should be shared with the institution's GMEC and senior leadership (DIO, chief executive officer, and/or dean).

It should be noted that the ACGME recently released proposed changes to the Common Program Requirements that mandate new duty hour limits, enhanced supervision of residents, and attention to patient safety issues.4 Preliminary communication stated, “Recognition of the need for enhanced measures to promote compliance has led to a new program of annual site visits to sponsoring institutions, focusing on duty hour compliance, supervision, and provision of a safe and effective environment for care and learning.”5 It remains to be announced whether the annual visits will be separate from, and in addition to, the current institutional site visits with maximum 5-year cycles, or whether all elements of institutional oversight will be incorporated into the 2-day annual visits. Regardless, the need for regular IRs for institutions will become more important, and will likely need partnership from the institution's quality and/or patient safety programs to ensure adequate monitoring of the new requirements of resident supervision, transition of care, and assessment of fatigue.

After UVA's GME program was placed on probation, we realized that we had lacked the rigorous discipline required to remain in compliance with ACGME standards. This crisis may have been avoided if the institution had actively and regularly sought institutional competence in PBLI. We strongly encourage institutions to consider creating a key PBLI activity for GME sponsors, the institutional IR, as one means to achieve not only the minimal institutional requirements set forth by the ACGME, but excellence in providing an environment in which to train our future physicians.