Getting the Evaluation Focus Clear: A Shared Understanding of What Is Being Evaluated

Your program director (PD) recently sent you the resident evaluations of the rotation you direct and the end-of-year residency program evaluation. She asked you to report on how you will address resident concerns about the balance of service and learning at the next program leader meeting. Based on your informal conversations about this service-learning issue, faculty perceive that the rotation develops residents' skills in managing large numbers of complex patients. In contrast, residents believe that they are caring for far too many patients that require intensive management. What do you do next?

Evaluation can inform judgments about the “value” of an educational activity (eg, rotation, journal club) or prescribe improvements for that activity. However, evaluation is a complex undertaking: perspectives on intended outcomes of educational activities differ; patient care needs change; and accreditation standards evolve. Without a clear evaluation focus, its findings often lack direction and have little to no utility.

Focusing an evaluation entails making judgments about which evaluation question(s) will best facilitate decision-making.¹  Evaluation models help focus the questions by clarifying outcomes and processes that lead to an effective educational activity. Each model typically describes a range of possible outcomes. Strong evaluations focus on a limited set of outcomes guided by evaluation standards.²  For example, the New World Kirkpatrick Model for program evaluation³  emphasizes the outcomes of educational programs at 4 levels (reaction, learning, behavior, results) and draws attention to critical intervening factors that connect educational activities with outcomes.³  Kirkpatrick's model is intended to improve learning, enhance the application of learning to achieve organizational goals, and justify training. Each of these could be the focus of an evaluation.³  The Table is an example of focusing on the service-learning challenge.

1. Form an evaluation workgroup. It can be informal but must include representatives from the participants involved—residents, faculty, nurse leaders, and a program leader (PD) or designee.

2. Define the focus of your evaluation. Achieving a shared understanding of the intended outcomes of an educational activity is essential to determine how to alter the activity to reach those outcomes. How will your findings change an activity? Asking “how” and “why” allows you to probe how your evaluation will impact the decisions made by your program leaders.

3. Ask for advice and seek data. Other programs have likely faced similar challenges. Ask your PD to post your question to your specialty listserv. Search the literature specific to this challenge.

4. Reframe your evaluation focus using a model. Kirkpatrick is only one of the available evaluation models that help to clarify the problem. Review other evaluation models as needed.⁴ 

5. Level up. Identify the Kirkpatrick level associated with your challenge. Explore factors within a level and consider going up a level. You know your residents are dissatisfied with your rotation (Level 1). How involved are they in daily decision-making (Level 1–Engagement)? Explore the issue further by examining whether they are achieving the stated objectives for the rotation (Level 2).

Evaluation standards²  provide a check on the evaluation focus and guide your long-term evaluation plan.

1. Accuracy: Share your focus area with your participants outside your workgroup to ensure you have accurately targeted critical elements and to gain their buy in.

2. Integrity: During the process of developing an evaluation focus, often the most important participants are those directly involved: the teaching faculty, residents, and PDs. Remember that nurse leaders and other members of the health care team may contribute valuable perspectives.

3. Utility: Develop a timeline for the evaluation. Know when you must decide about the rotation and the resources that you have available for conducting the evaluation. What do you need to do urgently? What do you need to do to sustain long-term improvements?

4. Feasibility: Anticipate the resources you need to sustain the program improvement effort. Faculty development activities may be needed to improve faculty teaching effectiveness. Refined learning objectives for the rotation may communicate the value of the rotation for residents and faculty.