Competence in Patient Safety: A Multifaceted Experiential Educational Intervention for Resident Physicians

Editor's Note: The online version of this article contains a table showing examples of patient safety improvement projects and a figure depicting the levels of a high-risk organization.

In 1999 the Institute of Medicine's (IOM's) report that medical errors result in 44 000 to 98 000 annual deaths brought patient safety to the forefront of public concern.1–3 Concurrently, the Accreditation Council for Graduate Medical Education defined a competency-based framework for physicians, specifying the domains of systems-based practice and practice-based learning and improvement as relevant to patient safety and quality improvement.4–6 A decade later, many health care organizations have not yet completely addressed the IOM's recommendations.7–9 As front-line providers, residents are in an excellent position to analyze adverse events and help prevent their recurrence,4,10–15 and many of the patient safety goals specified by accrediting and national safety organizations focus on clinical problems largely handled by residents.5,7,16 Residency programs in surgery, emergency medicine, and family medicine have developed educational programs using small-group didactics, morbidity and mortality conferences, patient chart reviews, case presentations, and observed structured clinical examinations to enhance resident knowledge and skills in safety and quality improvement.10,16–25 Yet, implementation of safety education programs has been variable and many residents and faculty remain with little or no formal training in these concepts.7,9,23,26,27 

In 2006, the University of Michigan implemented a Patient Safety Learning Program (PSLP) in its internal medicine and medicine-pediatrics residency training programs. Recognizing we had few qualified faculty members, 23,28–30 few “formal” learning experiences,7,8,23,24 and limited precedent for resident-driven safety and quality projects, we defined 3 primary goals for our PSLP: (1) to teach residents key patient safety concepts; (2) to transform our cultural environment; and (3) to provide experiential opportunities for residents to apply the patient safety and quality improvement concepts. We describe our PSLP and how our experiences reshaped the program to enhance our residents' learning experiences.

We began by defining key learning objectives that residents would achieve on completion of the PSLP (table 1). Beyond knowledge, we felt the program should improve resident communication, reflection, problem solving, and teamwork, and we designed 3 curricular elements and an assessment program that mapped to our learning objectives.

First, postgraduate year–1 (PGY-1) residents participated in 2 small-group seminars that presented core concepts of patient safety (PS)/quality improvement. The aim was to (1) help residents understand and identify the etiology of adverse events2,3,31–33 and (2) provide a hierarchy of the effectiveness of solutions to reduce them (table 2).34 The models were developed locally by work groups with expertise in the areas of quality and safety, human factors engineering, medical sociology, and educational assessment.

Second, residents (PGY-1 through PGY-4) participated in self-guided reflection exercised on adverse events, to develop their reflection, analysis, and problem-solving skills. Expert faculty provided feedback on the self-reflections.

Third, residents (PGY-1 through PGY-4) participated on Patient Safety Improvement Project (PSIP) teams that pursued team-based projects to address important patient safety issues. To enhance the PSIP teams' success, we developed (1) an academy of faculty mentors to guide resident teams through the projects; (2) a plan to facilitate longitudinal resident participation; and (3) criteria and processes for project selection, implementation, and dissemination.

Faculty was recruited from different disciplines within internal medicine and had demonstrated excellence in teaching, mentorship, clinical care, research, as well as administrative experience and a commitment to patient safety and quality improvement. Participating faculty members had 7% of their time protected and funded. The 9 academy members participated in a 5-session program led by a nationally recognized patient safety expert (J.W.G.). Sessions focused on the content covered during the resident patient safety seminars, project management, and team facilitation. Throughout the year, periodic academy meetings were held to continue faculty development and serve as a forum for discussion.

Postgraduate year–2 (PGY-2) and PGY-3 residents in our residency programs were assigned to 1 of 9 PSIP teams. Each team consisted of 10 to 11 residents and 1 academy mentor. A total of 141 residents participated during a 2-year period. Each PSIP team developed projects, and a committee composed of 4 faculty members familiar with the institution's safety and quality infrastructure provided feedback on the projects, highlighting their importance, feasibility, and multidisciplinary potential. Teams worked on their projects longitudinally; one half day was provided for each resident during every elective/ambulatory month, such that each resident received between 3 and 6 half days of protected time during the year. The protected half days were uniform across all PSIP teams (eg, the second Thursday afternoon of each month) to facilitate team collaboration. In addition, we created access to a centralized web-based archive to foster information sharing in recognition that it would be necessary for the teams to work asynchronously and between the scheduled and protected PSIP meetings to advance their projects. Additional resources, such as institutional collaborators, were identified to enhance the project's likelihood of success. At the end of the academic year, teams presented their project at a Patient Safety Symposium, which included residents, faculty, and leaders across the institution, to promote dissemination.

We used multiple methods to assess the impact of the PSLP. Each method mapped directly to the relevant learning objectives (table 1). Approval for the assessment program was obtained from the University of Michigan Institutional Review Board.

As an initial step, we developed a comprehensive survey to explore resident and faculty perceptions of patient safety, medical error, and attitudes toward recognizing, reporting, and discussing medical errors in our institution. Content was adapted from the Agency for Healthcare Research and Quality's Hospital Survey on Patient Safety Culture,35 the Leapfrog Hospital Survey,36 the HIMSS (Healthcare Information and Management Systems) Leadership Survey,37 and our institutional patient safety survey. Content validity was promoted through review by faculty, program leadership, and chief medical residents. The survey was administered in 2007 and 2008 to all internal medicine and medicine-pediatrics residents, academy faculty, and internal medicine faculty hospitalists. Comparisons between survey responses were analyzed by Fisher exact test.

Academy faculty evaluated each resident's performance annually by using a standard evaluation form that assessed systems-based practice, practice-based learning and improvement, communication (effective information exchange), and professionalism (carrying out professional responsibilities). Scores ranged from 1 to 9 (unsatisfactory, 1–3; satisfactory, 4–6; and superior, 7–9). A score of 5 was considered expected level of performance. This assessment is analogous to our “standard” rotation evaluation and has not been formally validated. Faculty development focuses on standard interpretation of the anchors, but we have not further measured reliability or validity.

Assessment of the PSLP included evaluation of the residents and faculty and assessment of the impact of the safety projects managed by the PSIP teams. In 2008 and 2009, all PGY-3 and PGY-4 residents completed anonymous surveys to assess the perceived educational value of their learning experiences, including those in the PSLP. Curricular experiences were rated relative to each other. Academy faculty feedback was collected in twice yearly meetings, grouped by theme and analyzed.

Three to 6 months following each symposium, the PSLP leadership followed up with the PSIP teams to assess how the projects impacted patient care processes and outcomes. The results are provided as online supplemental material. In addition, the leadership team collected data on characteristics previously demonstrated to correlate with successful project outcomes.30 

The results of the initial and follow-up surveys is summarized in table 3. We did not find any changes in resident or faculty perceptions of our culture after implementation of the PSLP, with the exception of a slight increase in resident-reported comfort with discussing errors with peers/colleagues, and a slight reduction in reported concern that discussing errors may negatively impact their careers, though neither reached statistical significance. Similarly, faculty attitudes and behaviors did not change, except for a slight increase in reported comfort with discussing medical errors with peers/colleagues.

Academy faculty completed 126 evaluations of assigned residents (90%). Although direct comparisons across learning experiences are difficult, evaluations of residents were lower for the PSIP rotation than for other rotations (3.18% versus 0.18% rated as “unsatisfactory,” P < .001; mean rotation score, 6.90 versus 7.34 of 10, P < .001). Qualitatively, “unsatisfactory” evaluations generally focused on residents' poor performance in the domains of professionalism and communication, and noted lack of teamwork, project participation, and/or responsiveness to faculty communication. In spite of lower mean scores for the PSIP rotation, more than 50% of residents were rated “superior” in professionalism, communication, practice-based learning and improvement, and systems-based practice.

On the 2008 and 2009 graduation surveys, residents rated the PSIP in the lowest of 5 quintiles of their educational experiences (2008 mean quintile, 1.55; range, 1–4; and 2009 mean quintile, 1.63; range, 1–4). Qualitative evaluation of the PSLP, specifically the team-based projects, revealed logistic difficulties. First, most academy faculty and residents felt that longitudinal resident participation on PSIP teams was unsustainable owing to scheduling problems and competing responsibilities. Second, faculty and residents questioned the engagement of some of the other resident team members and their accountability to the team. Third, academy faculty noted that the PSIP teams were too large, making communication and coordination difficult. Finally, perceived lack of progress in some of the projects was attributed to the unrealistic scope of the proposals. Despite these logistic issues, academy faculty and many residents indicated the PSLP was important and worth maintaining, but suggested a need for significant program restructuring.

In spite of the challenges highlighted by residents and faculty, the PSIP teams' projects addressed a number of patient safety problems (provided as online supplemental material). Commonly explored domains included communication and transitions of care, device-related complications, and complications of hospitalization. PSIP teams had variable success implementing lasting interventions, but most were able to pilot small changes and analyze the impact on patient care processes and clinical outcomes.

For example, a team focusing on the rising incidence of Clostridium difficile infection identified delays in recognition and test turnaround time as key contributing factors. Working with nursing, infection control officers, and the Office of Clinical Affairs, the residents improved testing methods and turnaround times. The team also implemented a unit-based trial of preemptive isolation precautions, the results of which led to an institution-wide program.

We designed and implemented a comprehensive program to communicate the priority of patient safety, address preidentified barriers within our curriculum, and intentionally involve learners early in their training.7,28,38,39 This produced some improvements in patient safety conditions, but received a low evaluation from residents, and did not result in a change in resident or faculty perceptions of the local patient safety culture. These outcomes are not surprising. Achieving cultural change faces significant obstacles, including long-standing cultural norms on handling error,31,32,40 residents as a vulnerable learner population,13 and the need to develop programs de novo.41 

Examination of our safety projects revealed themes that predicted a project's likelihood of success: project scope, alignment with institutional goals, presence of an identified champion, and solutions that went beyond education and training. A focused, realistic project scope, developed at inception by the team, driven by a detailed up-front analysis of the problem, clearly improved a project's chance of success.6,38,42,43 Projects that were too broad in scope often needed to compromise their goals, which limited their effectiveness. Projects aligned with institutional goals (such as reducing the rate of STAT [emergency] laboratory orders) were more likely to gain traction, involve interdisciplinary partners, and receive resources.

Identifying a consistent champion outside of the PSIP team was critical to the lasting impact of a project.32,44 One team attempted to improve communication of abnormal findings between radiologists and primary care physicians. The radiologist “champion” departed the university mid project and no replacement was identified. As a result, the project stopped and the proposed improvements were not implemented. This underscores the reality present in many health care settings, one in which the expertise and motivation to lead safety interventions is concentrated in a small number of individuals.45–47 

Finally, our teams consistently faced the temptation to propose mostly educational and training interventions, consistent with their role as learners. While these offer opportunities for improving knowledge, our treatment hierarchy reflects their general ineffectiveness when implemented in isolation (table 2).6,7,40 

As we finished the second year of the PSLP, we needed to reconcile the conflicting results of our intervention. We had worked to create a culture of safety, had developed faculty PS mentors, and many residents had immersed themselves in patient safety and quality initiatives. Yet, program evaluation revealed inconsistent resident engagement in the safety teams and low faculty and learner satisfaction. Our steering committee undertook a comprehensive reexamination and redesign of the PSLP, which was implemented in July 2009. We retained certain key elements, including our conceptual model, resident team-based project selection/participation, and open discussion in public forums.

To balance the PSLP with concurrent educational responsibilities, we consolidated the PSIP project experience into a more intensive experience within a 1-month block. This increased the protected time for residents to work on their projects, placed the PSLP on equal footing with other rotations, and facilitated team interactions. This structural change also addressed a significant obstacle in our earlier model by decreasing the amount of asynchronous and additional work that team members needed to complete outside of time specifically protected to advance their project. In addition, team size was reduced to 4 to 5 residents to enhance individual accountability. At the end of the rotation, teams present their work at a monthly departmental PS conference.

To date, preliminary analysis shows much higher resident and faculty satisfaction, and improved resident engagement and observed teamwork behavior. Challenges remain, including the shorter project duration, which reduces the likelihood that residents can see a project through to implementation.16,19,30 As the new model evolves, we anticipate teams may build on work completed by other teams, through piloting previously proposed solutions, for example. We do not anticipate a reduction in project scope or impact of projects and, thus far, this prediction has held true. Further research is needed to refine the PSLP, to develop more proactive resident learners in the area of quality and safety, and to further our program and institutional culture of safety.

We have laid the foundation to create a culture of patient safety in our residency programs and will continue our assessment of the magnitude of this change over time. Although many of our results are thematic and qualitative in nature, we feel the conclusions, drawn from 2 years of experience involving nearly 150 residents, can serve as guidance for others who attempt to tackle this important issue. The lack of a positive impact on safety culture findings highlights the difficulty educators face when attempting to change culture and speaks to the inherent methodologic challenges of quantifying attitudinal and behavioral change. The trend of increased comfort with reporting errors and patient safety incidents is encouraging, and we believe our intervention created an environment that enabled our residents to understand the importance of patient safety and provided opportunities to effect change.