A Multidisciplinary Approach for Teaching Systems-Based Practice to Internal Medicine Residents

In 2002, the Accreditation Council for Graduate Medical Education instructed residency training programs to evaluate residents using the 6 core competencies, leading to a greater emphasis on teaching and assessment of competency in systems-based practice.1–5 However, defining teaching and evaluation of systems-based practice in residency programs has proven to be challenging.6–8 As such, the graduating internist may not be fully prepared to integrate into an ever-more-complex health care system that demands of the practitioner not only excellent medical knowledge and patient care skills but also the ability to effectively manage quality improvement and patient safety–related endeavors. To address such issues, the Residency Review Committee for Internal Medicine created the Educational Innovations Project (EIP) to encourage select residency training programs to develop new strategies and methods to meet changing graduate medical educational demands. At the time our residency joined the EIP, in July 2006, we developed a 3-year, longitudinal curriculum (the administrative internship [AI]) to teach our residents systems-based practice through a series of hands-on quality and patient safety–centered rotations among our hospital administrators and nonphysician staff (and our own physician-faculty). Our aim was to provide an experience that would afford residents a systems-oriented view of their training environment, would foster a patient-safety and quality-improvement mindset, and would engender a positive attitude toward multidisciplinary collaboration and leadership in quality-improvement initiatives. We describe the process by which our AI curriculum was developed (based on the 6-step method of Kern et al9) and our experience with the program to date.

The curriculum for the AI began with the realization that a growing divide had emerged between what medical residency programs were teaching in systems-based practice and what the health care system was demanding. After reviewing the educational literature10–17 to identify relevant systems-based practice teaching and assessment strategies, our program formed a curriculum committee that conducted a needs assessment through a review of our end-of-rotation and end-of-year resident surveys and comments, suggestions box comments, faculty suggestions and anecdotal feedback, review of in-service and board examination scores, patient satisfaction surveys, and hospital adverse-events reports. A series of discussion sessions between the program director and the curriculum committee led to a tentative set of goals and objectives and the identification of potential educators thought to be most capable of addressing such goals and objectives. We approached potential faculty, enlisted their support, and began to work in cooperation to identify the components of an early curriculum and to lay out specific plans for a set of rotations. Each educator provided the curriculum committee with a set of didactic materials, goals and objectives, and a plan for a postrotation evaluation. In addition, each educator submitted 10 questions to the committee. From this pool, 20 questions would be selected for a “health care systems examination” to be administered to residents before the start and at the completion of the entire AI. Inconsistencies between the goals and objectives initially identified by the curriculum committee and those developed by the AI educators were reconciled at a meeting of the program director, the curriculum committee, and individual educators. Thereafter, a written curriculum was produced by the committee, along with an end-of-rotation global assessment form and a postrotation feedback form. These were disseminated to all faculty and residents just before the implementation of the program on July 1, 2006. The curriculum is shown in the box.

The AI consists of a series of rotations (ranging in length from 4 hours to 20 hours) that take place as a subcomponent of the annual month-long ambulatory-care block. Each involves close interaction with the (largely) nonphysician staff whose training and expertise not only help our facility to function as a cohesive unit but represent a valuable educational resource. All categorical residents proceed through the AI according to the sequence displayed in the figure, and a detailed description follows.

The resident spends one-half day reading assigned materials and then meets with a nutritionist for 2 full-day sessions, learning to identify patients at nutritional risk, to seek consultation appropriately, to choose appropriate therapeutic diets for high-risk patients, to review the institution's enteral formulary and indications for its use, to initiate and maintain patients on parenteral support safely, and to effectively use clinical nutrition staff and resources.

This full-day rotation includes a 2-hour meeting with the vice president of the Department of Clinical Care Management, who provides didactic materials and an introduction to managed care systems, health care financing, and case management. Working directly with case managers, the resident gains insight into the importance of early and thorough discharge planning; the roles of case management staff; the positive effect of collaborative efforts on patient outcomes; the need to tailor discharge plans to the patient's unique financial, personal, and social needs; and how financial issues influence postacute care placement, including home services and access to durable medical equipment and medications. The discharge appeal process is reviewed; at the end of the academic year, the resident completes an oral examination with the vice president of the Department of Clinical Care Management.

During this rotation, a pharmacist at the hospital with a Doctor of Pharmacy degree shows the resident basic pharmacy operations and medication error tracking, provides a historical perspective of the US Food and Drug Administration and drug legislation, and reviews important drug interactions, commonly employed monitoring tests for medications, and important drug-induced diseases. When possible, the resident attends the Pharmacy and Therapeutics Committee meeting and explores a root cause analysis of a medication error. Residents complete a 2-day pharmacy AI experience during each of their first 2 years of training.

Each year, residents devote a minimum of 5 half-day sessions to outpatient continuous quality improvement, working on projects of escalating complexity that begin with monitoring Joint Commission quality indicators during the intern year, progress to monitoring quality measures related to specific chronic conditions identified by the resident during the second training year, and culminate in the third training year with projects focusing on patient motivation and education. During the continuous quality improvement experience residents learn the importance of continuous quality improvement initiatives and appropriate documentation, learn the Plan-Do-Study-Act process and its application to various clinical settings, and develop hands-on experience in designing and implementing quality improvement projects.

In July 2009, based on recent graduate survey data, we also created a 2-week inpatient research rotation during which PGY-2 and PGY-3 residents have additional time to plan and collect data on a quality improvement topic—with an eye toward presenting their findings at a scientific meeting or following the project through to publication. Written summaries of the quality improvement projects become part of the residents' portfolio on the Residency Management Suite (Residency Management Suite, Inc, Uniontown, OH) system.

The resident meets with a physician who is versed in financial matters related to medical practice and is shown common coding and billing procedures and is introduced to practice management issues. Health Insurance Portability and Accountability Act guidelines are reviewed with the resident and Current Procedural Terminology codes and International Classification of Disease, 9th edition, coding and related billing methods are discussed in the context of the residency, hospital and outpatient business/care models.

The resident meets with the director of the Department of Hospital Quality Management and with key staff, who provide a series of three 2-hour lectures on the institution's quality-monitoring systems, ongoing initiatives (both resident initiated bottom-up and institutionally driven top-down types), and key participants in those initiatives. The resident then spends approximately 14 hours with nurses charged with monitoring quality indicators (National Hospital Quality Measures), learning the significance of and the reporting mechanisms for these quality measures. The resident is shown the types of incidents that are reported in the New York Patient Occurrence Reporting and Tracking System, the National Quality Forum, and the Sentinel Events Reporting System and how those events are detected and handled within the institution. The role of the Joint Commission's National Patient Safety Goals are reviewed and placed in the context of the quality structure.

The resident meets with the director of the Department of Risk Management during 5 one-half day sessions. At the first session, the director provides reading materials and introduces the resident to various legal concepts, such as the elements of a malpractice case and the basic features of the litigation process. The roles of appropriate documentation and informed consent are discussed in relation to quality, patient safety, and legal matters. The director relates these concepts to ongoing legal cases and the hospital's medical error and adverse events reporting systems. If either a root-cause-analysis or a court case is in progress, the resident observes the proceedings with the director of risk management.

Residents are introduced to the rotation by the director of the Department of Occupational Health, who provides didactic materials regarding current occupational and environmental medical practice. The resident then shadows the director as patients are seen in the clinic, but after becoming comfortable with the situation (typically after 1 to 2 hours), the resident begins to examine patients with the director acting as supervisor. This allows residents to become familiar with the presentation and management of common occupational disorders and to become more adept at delineating functional work limitations and at defining when employees may safely return to work. The resident is also shown the appropriate process for handling occupational exposures to blood and body fluids.

During 3 one-half day sessions, the resident works directly with a nutritionist in the outpatient setting performing nutritional assessments and counseling for all patients referred from the medical subspecialty (cardiology, renal, human immunodeficiency virus and AIDS, etc), obstetrics and gynecology, and surgery clinics. After becoming comfortable in the clinic, the resident performs assessments and counsels patients while being directly observed and guided by the nutritionist. Through this approach, the resident develops an appreciation for the medical conditions for which nutritional evaluations are critical, advances his or her counseling skills, better understands the sources of important nutrients, and appreciates the benefit of a multidisciplinary approach to nutritional management.

The rotations and overall educational experience have remained relatively constant since AI inception—the principal exceptions being that 2008 graduates completed only 2 years of the 3-year curriculum, and 2007 graduates participated in only 1 year of the curriculum. Minor scheduling adjustments and some turnover of educators have also occurred. In addition, the process of evaluating the AI and its participants has evolved during the course of the program. For that reason, this report is limited to an overview of the program's early impact and a description of the past and current evaluation process. A presentation of specific outcome measures will be reserved for a future publication, at which time, a larger number of participants will have been evaluated in a uniform fashion.

The AI participant performance has been monitored through a combination of verbal feedback, end-of-rotation written assessments, rotation-specific written and oral examinations (pharmacy and case management, respectively), a written systems-based practice test (20 questions, pre-AI and post-AI), and program ratings of quality improvement portfolios. During this academic year, we placed greater emphasis on direct observation by implementing a health care system clinical evaluation exercise, which is conducted by an associate program director using chart-stimulated recall of a patient with complex medicine service, following each PGY-3 resident's completion of the AI. A less-conventional tool by which we provide systems-related feedback is the resident report card, which consists of a letter grade computed by our performance department for interns and residents during months spent on floor duty. The score reflects the resident's case-mix–weighted mean length of stay and the efficiency with which discharge orders were entered, as well as an adjustment factor for against-medical-advice departures. The data allow us to remind trainees (during formative feedback sessions) that physician labors are becoming ever more closely tied to performance measures and financial reimbursement. This year, we modified the report card to account for National Hospital Quality Measures core-measures adherence and patient-satisfaction scores. Since the outset of the AI, residents have completed a brief postrotation survey regarding their AI experience (2006 to present). Since 2009, residents also complete a more comprehensive survey after concluding their entire AI experience, and are also surveyed regarding their perceptions of their AI experience in light of their newly acquired practice experience. Faculty educators also are surveyed annually to determine their perceptions of the AI and needed changes.

Although it is difficult to quantify the institutional-level influence that the AI has had at our facility, several developments bear mention. Following a 2009 presentation of the AI program to our Graduate Medical Education Committee and the hospital board's quality council, the institution expanded the AI to permit participation by residents from other departments. Institutional financial support of AI-related projects has significantly increased annually, allowing us to enhance our sign-out software and to create a Center for EIP Quality Research, which now facilitates collaborative projects between residents, faculty, and quality improvement personnel—including our ongoing efforts to identify best hand-off practices18 and to improve physician near-miss and adverse-events reporting through our handheld platform. Through end-of-rotation evaluations, our educators have noted that residents' insights have spurred important systems changes: near-miss and adverse-events reporting leading to improvements in how narcotics and insulin are ordered and how medications are dispensed; simulation training to better prepare residents to act as “code team” leaders; use of sign-out software instant-messaging capability to facilitate discharge planning, to permit correct provider-relaying of critical imaging and laboratory test results and to facilitate concurrent monitoring of National Hospital Quality Measures (core measures) by members of the quality staff. Without the AI to foster a patient safety and quality mindset, to promote collaboration between residents and hospital personnel, and to enlist the support of our institution, we do not believe our facility would have benefitted from such a depth of quality improvement activities. This suggests that institutions that strongly support systems-based practice educational programs may derive significant returns from their investments.

All residents now participate in both inpatient and outpatient AI quality improvement projects, reporting their results at quality improvement committee meetings and noon conferences. Many also have presented their findings at local and national meetings and/or published their work. The table displays a selection of recent AI-related projects. The materials from AI projects are also uploaded into the New Innovations Portfolio module and are reviewed during summative evaluations by the program director. The resident with the best portfolio receives the annual Center for EIP Quality Research award. Given that many projects started during AI rotations are voluntarily carried through to publication or presentation, the AI curriculum appears to promote leadership among our trainees.

Although our experience in establishing the AI has been positive, there have been a number of challenges encountered along the way. First, because the project involved the creation of an institution-wide educational process, logistical problems arose. For example, although standardized evaluation forms were created along with the new curriculum, use by faculty educators was inconsistent. This limited to some degree (despite other methods of evaluation) our ability to rate the extent to which AI participants meet the educational aims on particular rotations. However, as the AI is now well established and a computerized evaluation system is in place, we have found the end-of-month evaluation process to be more manageable and useful. Other challenges have related to availability of educators. As teaching on several AI rotations depended heavily on particular educators, we were not able to easily compensate for their absence during vacations and other times. This resulted in some loss of consistency in the educational experience. Perhaps the greatest limitation of our AI program has related to the finite amount of time that can be allocated. Nearly all of our educators have lamented the fact that residents cannot spend more time on each AI rotation and have expressed a belief that the educational experience would be enriched by additional training. A final item worth noting is that our institution has been supportive of our EIP-related educational activities. Absent such support, developing and implementing a similar systems-based practice curriculum elsewhere might prove to be challenging.

We believe our study confirms earlier reports that implementing a comprehensive systems-based practice teaching curriculum within the structure of an internal medicine training residency is both possible and beneficial.19 As we have focused more intensively on teaching systems-based practice, our residents have become leaders in quality and patient safety projects that now help to shape how we care for patients. We believe that our results are important in that they may provide guidance to other residency programs that seek to implement a similar curriculum and may encourage institutional support for such initiatives. In the near future, our challenge will be to show that our educational initiatives can, in fact, result in quantifiable quality improvements.