Institutional Attributes Associated With Innovation and Improvement: Results of a Multisite Study Open Access

Despite a seeming preoccupation with innovation and improvement in health care,1 and research that demonstrated the positive effects of culture and specific approaches such as total quality management,2,3 we know of no studies that have explored why some teaching hospitals and health systems appear to offer a more fertile environment for innovation and improvement in the learning environment for residents and fellows. As a consequence, little is known about the role of organizational structure, strategy, culture, and senior leaders and other individuals in providing favorable conditions for innovation and improvement in resident and fellow education. These factors have been studied extensively in the general literature on organizations. Adding to our understanding of potential common attributes of these settings, and the process by which some programs and institutions achieve innovation and improvement in their learning environment, could provide broad benefits. Among them are improved learning environment across residency programs, encouragement of innovation focused on residency education and the potential to identify improved accreditation standards.

In 2004, the Accreditation Council for Graduate Medical Education (ACGME) formed a Committee on Innovation in the Learning Environment and charged it with exploring innovation and improvement in teaching settings in the learning environment. One element of the charge related to the conditions for innovation and improvement at the institutional level. This prompted the development of the Learning Innovation and Improvement Project (LIIP) in 2006. The aim was to study a small group of institutions that appeared particularly successful in innovation in their residency and fellowship programs, and seek to validate the findings in a second larger group. A small project advisory group provided oversight.

Study goals entailed (1) gathering ground-level observations on processes and common attributes of institutions and programs that succeed in innovation and improvement in their learning environment; (2) disseminating this information for adoption and adaptation by other teaching institutions; and (3) exploring how learning about attributes and conditions of high performance could inform accreditation and could overcome barriers for institution- and program-level innovation that could be attributed to the constraints posed by the accreditation standards and processes.

The first phase of the project was initiated in December 2006. We identified a small group of institutions that appeared to be particularly successful in innovation and improvement in the learning environment, using the number of original articles in peer-reviewed journals and other relevant sources that described local innovation and improvement initiatives at the program and institutional level. This identified 17 candidate institutions, which had published extensively on innovation and improvement in peer-reviewed journals, and added sources such as posters and abstracts presented at conferences. Publications that were considered focused on innovations in residency and fellowship education and related matters. A review of ACGME accreditation performance screened out institutions with accreditation deficiencies at the program or institutional level, as well as suboptimal accreditation status with other applicable accrediting bodies such as the Joint Commission. Review of accreditation performance reduced the pool to 14 candidates.

We selected 4 institutions for the alpha phase of the project, which entailed comprehensive interviews and site visits. Selection considered geographic and institutional diversity. Two participants were university-based teaching hospitals (University of Rochester and Dartmouth-Hitchcock Medical Center); 1 was a large academic group practice with a medical school (Mayo Clinic Rochester); and 1 was an independent academic medical center and health system (Henry Ford Hospital/Health System). All 4 institutions agreed to participate.

Staff and the advisory committee collectively designed a semistructured interview framework that offered flexibility through open-ended questions and the ability for follow-up questioning tailored to the given response. The framework combined a deductive approach, through the use of validated performance excellence criteria,4,5 and an inductive approach that could be sensitive to local strengths and emergent properties of institutions that have sought to improve and innovate in the learning environment.6 An open-ended, qualitative approach was used to (1) capture and describe program processes that are dynamic and evolving; (2) allow documentation of variations in implementation and ongoing evolution at different sites; and (3) ensure that the questions were sensitive to the fact that local initiatives may seek different outcomes. Questions were designed and asked to explore institutional attributes in 4 areas: general questions regarding education, questions on the topic of change and innovation, questions regarding learning practices at the institution, and questions that explore areas in which an organization can serve as a role model for others. The interviews also identified the attributes participants thought most innovative and beneficial to improvement in the particular local and institutional context. A copy of the interview questions is shown in  appendix A.

Staff and members of the project advisory committee conducted multiple telephone interviews with each institution and each received a site visit. Institutional representatives who participated in the interviews and site visits included (1) the designated institutional official and his or her staff; (2) 6 to 12 graduate medical education (GME) leaders involved in innovation and improvement and individuals with key roles in program level innovation and/or improvement; (3) senior institutional leaders (to assess their commitment to and their role in innovation in the learning environment); and (4) residents and fellows. ACGME staff and members of the Project Advisory Group spent approximately 2 days onsite, and each site visit concluded with an exit interview with the designated institutional official and GME leaders to discuss conditions that fostered and those that presented barriers to innovation.

The interviews were tape-recorded and transcribed, and extensive field notes were taken during the site visits. Structured transcripts were shared with participants who were asked to correct errors, provide feedback on the utility of the process, and suggest additions to explore concepts thought important to innovation and improvement at their institution.

Given the small sample of 4 alpha participants, in late fall 2007, we contacted the 10 other candidate institutions to request their participation in a second round of the LIIP project, using telephone interviews and questions that had been refined during the first phase of the study. Five institutions (shown in  appendix B) agreed to participate. The aim of the telephone interviews was to validate the concepts identified during the alpha phase and to offer support for the generalizability of the factors identified, their relevance to a broader set of institutions with innovation and improvement in their learning environment, and their utility for adoption and adaptation by other institutions wanting to offer a better learning environment. The second phase of the project entailed 2 to 3 telephone interviews with the DIO and a small group of senior educational leaders lasting approximately 60 minutes each and the information was recorded and transcribed.

In addition, a limited amount of ACGME and public data were collected and analyzed for 4 other groups of institutions: (1) institutions in good accreditation standing that did not meet the criterion of innovation; (2) institutions that met criterion for innovation but had suboptimal accreditation performance; and (3) fellowship programs that were not accredited by the ACGME or a member board of the American Board of Medical Specialties. The last group was included to explore whether unaccredited programs may be in a better position to innovate, owing to the absence of potential constraints posed by the accreditation standards. Data collection for this group, which entailed 2 focus groups with program directors at institutions with a large number of unaccredited subspecialty programs, was conducted in October 2007, with additional individual interviews between November 2007 and January 2008.

Because of the voluntary nature of participation and the fact that participants were institutional units, not individuals, Institutional Review Board approval was not sought.

After completion of the telephone and on-site interviews, the transcribed responses were imported into QSR NVivo (QSR, Melbourne, Australia). NVivo has been used extensively in social science research to analyze qualitative information.7 The underlying approach is grounded theory, which facilitates the interpretation and reinterpretation of data as the theory becomes more refined in successive reviews of the data.8 

The coding strategy established for this project was to use NVivo's options for open, axial, and selective coding to aggregate information into categories and allow for the exploration of similarities and differences.9 Each interview was analyzed to identify the sections of the text conveying main thoughts, with these sections ranging from short sequences of words to a few sentences in length. Axial coding was used to explore the relationships among categories. The aim was to produce a linked, contextual interpretation of the coded text.9 The final step in the coding process was selective coding to identify common elements present in the institutions under study.

Three themes emerged from the interviews and site visits of the participants in the alpha phase: (1) an organizational structure and culture that promote integration and inclusion; (2) a recognition of the value of resident education to the institution; and (3) a learning organization that promoted both use of data and ongoing change, improvement, and innovation. A summary is shown in the box, and the following sections elaborate on each of the themes.

An integrated vision of education, research, and patient care was a common thread for all 4 alpha sites, facilitated by an organizational structure that closely linked the 3 domains. Another attribute was an emphasis on faculty and leadership development and the positive engagement of a broad group of individuals. The institutions felt that their GME programs served as a means for collaboration and for bringing the organization together in a focus on curricula, learning models, and evaluation and assessment.

Two institutions, the Mayo Clinic and Henry Ford Health Systems, noted that the integration of patient care, research, and teaching stemmed from the vision of their founders and is embedded in the organization's culture and expressed in the expectations for all faculty, staff, and learners.

The idea of inclusiveness and involvement across organizational units and levels was a key concept in the comments from all 4 sites. It was noted on several occasions that an integrated structure and operations allow for cohesiveness among the units, with physical proximity and frequent regular meetings noted as an important attribute in 2 sites (“our offices are right next to each other”).

Finally, all 4 institutions reported that their residency program coordinators are vital to the success of the GME enterprise and included them in the information loop related to the residency and fellowship programs, as well as gave them input into decisions.

For the participants in the alpha phase, the missions of education, patient care, and research were not segregated but combined, with the institutions actively exploring and exploiting the synergies among them, in fulfillment of a larger institutional vision. A key concept was involvement of learners in multiple ways and multiple levels, going beyond patient care to contributing to improvement efforts and suggesting ideas for innovation. Faculty reported that this provided added incentives for life-long learning. One interviewee noted, “Having residents ask questions, be intrigued, and experience the thrill of discovery, challenges our faculty to be better at what they do, to read more, to do more research and facilitate clinical improvement. So the impact on the patient is that we provide better quality.” The institutions also offered a range or financial and other incentives to encourage faculty engagement in education and innovation in this area.

The 4 institutions shared a belief that patient care is improved because of graduate medical education but that the educational mission also serves as a convening element that brings different parts of the institution together. This is evident in the comments of the designated institutional official from one of the other alpha phase participants.

“I think the other thing education does for us is that it acts like the glue that holds the group and the various departments and divisions together. Even research doesn't bring us together like education. Research can be carried out in a silo fashion, at least some parts, and medical education really cannot be done that way. I participate in many other residency programs, and they participate in my program. This has allowed us to build many collaborative relationships.”

The institutions had a strong focus on innovation and improvement, and supported this through ongoing efforts to try new things in all areas related to their multiple missions, and leaders who treated successes and failures as opportunities for learning. Participants reported that the interest in improving education and patient care contributes to a positive organizational climate not only for learners but also for faculty and staff as well. Individuals are comfortable with change, including change in the educational programs.

“As we prepare for a site visit, if we do not see a lot of items mentioned in the program information form (PIF) that asks for ‘changes since the last site visit’ we worry something may be wrong with the program.”

The institutions used data extensively for innovation and improvement and to promote accountability. Data are used in the learning process for resident physicians (improvement education) and are also aggregated to assess organizational performance through metrics and dashboards that facilitate ongoing improvement and strategic planning. These efforts create measurable improvement that contributes to a greater effectiveness in patient care and/or education. Collecting data on outcomes allows for accountability and for aligning organizational resources with mission-driven goals, with participants with fairly diverse access to resources reporting that their system's accountability made them comfortable with the amount of resources devoted to education. One attribute related to accountability is that data on performance is also shared with the public. This is an area of the common attributes that each alpha site was committed to do, but where each has an individual approach for achieving it. The most well developed system for sharing outcome information was found at Dartmouth-Hitchcock Hospital, which publicly releases outcome measures for its clinical and educational enterprise, often with benchmarking comparisons to the highest-performing institutions.10 

A final attribute of these learning organizations is the presence of scouting and dissemination related to innovation, found at all levels of the organization. “Scouting” in this context refers to a using participation in a national forum to collect information on new ideas in the field, with a focus on their suitability for adoption and adaptation in the local setting. The Mayo Clinic has put into effect ways to disseminate innovation among its units through a social network that connects its 3 locations to facilitate dissemination of innovative ideas.

During the interviews with institutional and education leaders at the conclusion of each site visit that sought to explore barriers to innovation for the 4 alpha pilot participants, none of the interviewees suggested that the accreditation standard or process presented barriers to innovation. Instead, barriers to innovation for institutions already engaged in system-wide innovation efforts may be related to the time and opportunity costs of more comprehensive innovation and improvement, once the institution has taken advantage of the “low-hanging fruit.” The information also suggests that limits to further expansion of innovation may be related to the time and opportunity cost for program- and unit-level champions of innovation, who may become exhausted, or to the lack of a succession plan for these individuals, with innovation and improvement efforts stagnating or being terminated if these individuals leave or assume other roles within the organization.

The interviews with the 5 participants in the beta phase confirmed the presence of 2 of 3 attributes of the alpha study institutions in the second group: recognition of the value of resident education to the institution, and the organizational attribute of being a learning organization that promoted use of data and ongoing change and improvement. Beta phase participants commented on collaboration among organizational units but did not share the structural cohesion, nor did they report having a culture that promoted integration and inclusion to the same degree as the alpha participants.

Review of accreditation and other data and brief interviews with the 3 institutions that innovated but did not perform well in accreditation suggested that their suboptimal performance was unrelated to their innovation efforts. Interviews with 4 additional institutions that had smaller numbers of publications on innovation than the institutions selected for the alpha and beta phase suggested that for some, facilitating local improvement was more important than reporting their innovations in publications. It is thus possible that institutions not selected for our study are equally effective in facilitating improvement and innovation in the learning environment. Our analysis was not able to assess whether these institutions shared organizational attributes with their counterparts that were included in the LIIP study.

Follow-up interviews for residency programs in specialties not accredited by the ACGME found that unaccredited subspecialties frequently are very small and operate in niche specialties, where the care itself is often innovative, but the programs tend to have only a single fellow, close contact with faculty, and minimal infrastructure. Innovation in the design of the educational program was not an attribute of small programs. In addition, institutional sponsors often apply the ACGME institutional and Common Program Requirements to these unaccredited programs, particularly if the subspecialty is under consideration for accreditation, suggesting that they may operate under similar environmental constraints as those faced by accredited programs.

While a number of studies have explored interinstitutional collaborations to improve care,11–13 there has been little research to identify the attributes that contribute to innovation and improvement in learning, or the role of intrainstitutional collaboration in contributing to improvement at the interface of clinical care and learning. There is benefit in better understanding common attributes of these settings and the process by which some programs and institutions achieve innovation and improvement in the learning environment. A strength of our approach is that it used interviews that combined deductive and inductive approaches that enabled us to identify the attributes most relevant to the particular respondent and the defining characteristics or environmental conditions for the given exemplary learning site.

Some of the concepts identified in the study of 9 teaching institutions (4 alpha-phase and 5 beta-phase participants) could be replicated by other institutions, with the goal of enhancing innovation and improvement in the learning environment. Concepts ranging from use of data to enhance care, education and accountability, to involving residents in improvement, and including and involving program coordinators more broadly in carrying out the GME mission, are relevant and are suitable for adoption or adaptation by other institutions that wish to create a better learning environment. In contrast, the structural factors, particularly the organization and compensation of faculty in a way that does not create a disincentive for teaching, may pose a challenge for many institutions, given the focus on clinical productivity and the relatively static nature of physician organization and governance at many institutions, particularly those with university affiliations.

Our study has several limitations. One relates to the small sample size for the alpha and beta phase of the study, which may limit generalizability. A second is that some innovative institutions chose to focus on local improvement in lieu of national dissemination, as discussed above. A more significant potential limitation is that, unbeknownst to the researchers at the time the 4 alpha participants were selected, 2 of these institutions shared structural design elements with the Mayo Clinic. In the case of the Henry Ford Health system this is due to its founder having sought the advice of the Mayo Clinic leadership on the design of the institution that bears his name. For Dartmouth-Hitchcock Medical Center, the link was that its faculty practice plan was in large part based on the organization of the medical staff at the Mayo Clinic. Finally, given that the interviews were conducted by ACGME staff, it is possible that institutional participants may have been reluctant to discuss whether accreditation standards or process posed barriers to innovation.

Future research to assess the factors and conductions that promote innovation and improvement is needed. This should seek to overcome some of the limitations of this study through a larger and more diverse sample of teaching institutions, and an approach that explores the contribution of a broad range of factors that may foster or hamper innovation and improvement in the learning environment.

• 1. What is your vision of the role of education in your organization? (Please expand on that thought.)

• 2. How do education and clinical care interact and affect one another in your settings?

• 3. How do you, your office or the programs in your institution foster interest in innovation and improvement in education and patient care?

• 4. Do you share lessons learned among your programs with other institutions?

• 5. What formal or informal mechanism currently stimulates change and improvement in your program? (Is there another event or mechanism that would stimulate change/improvement in your program(s)? Are you currently making any changes?)

• 6. Have you had some multi-disciplinary initiatives? (What are advantages of this approach? How does the multidisciplinary nature of the project add to their complexity?)

• 7. What else can you tell us from your experiences about implementing innovations/improvements in graduate medical education? (What were barriers or advantages in those situations?)

• 8. Do you collect and use data on performance (educational and/or clinical)?

• 9. What role does the data support in organizational learning?

• 10. What are areas/aspects of your institution that could serve as a role model for others?

Henry Ford Hospital and Health System, Detroit, Michigan

Mayo Clinic and Mayo School of Graduate Medical Education, Rochester, Minnesota

Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire

University of Rochester, Rochester, New York

University of California at San Francisco Medical Center, San Francisco, California

University of Missouri-Columbia School of Medicine, Columbia, Missouri

Lehigh Valley Hospital, Allentown, Philadelphia

University of Pennsylvania, Philadelphia, Pennsylvania

Virginia Mason Medical Center, Seattle, Washington