Using Appreciative Inquiry to Inform Program Evaluation in Graduate Medical Education

Graduate medical education (GME) programs are required to conduct an annual program evaluation to promote goal achievement and identify specific steps toward improvement, as required by the Accreditation Council for Graduate Medical Education.¹  Philibert²  and Philibert and Nasca³  called for GME programs to exceed accreditation standards and develop innovative evaluation approaches that accelerate program improvements.

Innovative yet feasible approaches to program improvement are needed, particularly for small programs that may lack the number of trainees necessary to draw useful conclusions from traditional, anonymous program evaluation surveys. Appreciative Inquiry is a process that appreciates, envisions, and builds on the best of “what” is to create a better future for individuals in a group.^4–6  When applied to program evaluation, Appreciative Inquiry is a strength-based approach that focuses on what is going well. It is used to identify successful moments, effective processes, and program strengths, and it can also identify problems by reframing deficiency-focused language into wishes or desires.^4–6  Recognizing that several models exist to describe Appreciative Inquiry, we used Preskill and Catsambas's 4-I model of appreciating the best of what is (Inquire), envisioning what could be (Imagine), setting new strategic directions (Innovate), and navigating change (Implement).⁵  In medicine, authors have described applying Appreciative Inquiry to professionalism, quality improvement, organizational change, and finding meaning in work.^7–11  Its use in GME program evaluation has not been described.

In this article, we describe a novel application of Appreciative Inquiry and demonstrate how it led to meaningful improvements in a fellowship program. To offer evidence of the value of Appreciative Inquiry as an approach to program evaluation, we organize the results around 3 program evaluation standards: (1) easy to collect evaluation information (feasibility), (2) fair and accurate representations of stakeholders' perspectives (propriety), and (3) useful for informing decisions (utility).¹² 

The Pediatric Pulmonology Fellowship at Baylor College of Medicine/Texas Children's Hospital (BCM/TCH) has 9 fellows in a 3-year program, with the primary goal of preparing trainees for independent practice. Prior to Appreciative Inquiry, the program was evaluated annually using anonymous surveys.

We interviewed alumni who had graduated from 2012 to 2016 in person or over the telephone, using an Appreciative Inquiry approach. Recognizing stakeholder involvement as a dimension of effective program evaluation,¹³  we considered alumni to be key stakeholders because they are uniquely positioned to evaluate the program and discuss how it prepared them for practice.

Working with a skilled program evaluator and qualitative researcher experienced with Appreciative Inquiry (D.F.B.), the lead author (J.A.R.) developed a semistructured interview guide that addressed the first 2 Appreciative Inquiry phases, Imagine and Inquire (table) to elicit alumni's peak experiences during fellowship, the values underlying those experiences, and wishes for how the program might be improved.⁵  Recognizing the potential for alumni to respond in a socially desirable manner, a faculty member from another division (C.F.) interviewed alumni currently working at BCM/TCH, and the lead author interviewed alumni working at other institutions. Interviews were conducted between December 2016 and March 2017 and were audiotaped and transcribed. They lasted an average of 45 minutes. There was no financial incentive for alumni to participate. The Institutional Review Board at BCM/TCH approved this study.

We took an inductive approach to analyzing qualitative evaluation data,¹⁴  looking for recurrent concepts. We used Appreciative Inquiry's 4-I model to categorize concepts and organized illustrative comments into each category to summarize and represent the program's successes, strengths, and areas for improvement (ie, wishes).

As part of the Innovate phase, the lead author shared the analysis with the Program Evaluation Committee (PEC). Through an iterative exchange of ideas and rich discussion that lasted 2 hours, the PEC developed an action plan and targeted several areas for improvement. The subsequent Implement phase occurred over the course of the academic year.

Eleven of 14 (79%) alumni (6 BCM/TCH and 5 non-BCM/TCH physicians) responded to the invitation and discussed life after fellowship. All alumni described peak experiences of fellowship training, despite having completed their fellowships up to 5 years prior to the interview and without advance knowledge of the interview questions.

Prior to conducting interviews, 2 authors (J.A.R. and C.F.) engaged in several hours of self-directed learning about Appreciative Inquiry. One author (D.F.B.) had experience in program evaluation and formal training in Appreciative Inquiry. All authors were familiar with qualitative data analysis. Analyses required 1 to 2 hours per transcript, and costs for transcription were less than $1,500. Administrative staff were minimally involved in interviewee recruitment.

Alumni served as key stakeholders and provided an essential perspective, having transitioned from fellowship to the postfellowship world. In this regard, alumni informed the program's goal of preparing fellows for independent practice in ways that prior program evaluation had not.

During the Inquire phase, alumni talked about program strengths, such as supportive relationships with faculty, broad exposure to patient care, and diverse experiences with faculty. During the Imagine phase, alumni commented on ways the program could help future fellows better prepare for independent practice. For example, alumni wished that they had learned specific skills related to difficult conversations and palliative care and that they had spent more time in outpatient clinics.

Guided by information from the Inquire and Imagine phases, the lead author and the PEC partnered in the Innovate phase to build on program strengths and develop an action plan for program improvement. For example, the PEC brainstormed ways to help fellows learn how to have difficult conversations. They also discussed ways to integrate more learning experiences in the fellows' ambulatory clinic.

The Implement phase lasted into the next academic year as the lead author and the PEC revisited the action plan. For example, a session was added with palliative care faculty in which fellows role-played end-of-life discussions. In addition, the PEC partnered with faculty and clinic administration to develop a clinic rotation at satellite centers. Based on current fellows' positive regard for learning in a community model of practice, the committee made this rotation a requirement for all fellows.

Appreciative Inquiry can be used in an innovative approach to program evaluation that has value in GME, demonstrated by its feasibility, propriety, and utility in a pediatric pulmonology program. By describing our use of Appreciative Inquiry to connect with and learn from alumni, we add to the literature on general models and ideas for program evaluation.^13,15–18 

We acknowledge that the relatively time-intensive nature of data analyses is a limitation that likely makes Appreciative Inquiry more feasible for small programs. However, larger programs could choose to purposefully sample selected graduates or use a smaller number of Appreciative Inquiry questions to tailor the approach to the size of program and available resources.

Appreciative Inquiry is an innovative and feasible approach for GME program evaluation that can facilitate actionable improvement. We believe its value lies in its ability to generate relevant improvement information from key stakeholders. We plan to use Appreciative Inquiry interviews with current fellows and expect it will continue to move our program closer to attaining a program goal of preparation for independent practice.