Blueprinting Program Evaluation Evidence Through the Lens of Key Stakeholders

Residency and fellowship program leaders regularly seek information to determine the value or effectiveness of their programs' sponsored activities and processes to drive continuous improvement. This information often includes individuals' judgements and perceptions about what is or is not working. These judgements and perceptions may be influenced by individuals' backgrounds or values, as well as limited data about the program. Thus, key stakeholders in any program evaluation must be queried about evaluation evidence prior to gathering any new information.

A stakeholder is an individual or group that has an interest in any decision or activity in a residency or fellowship program. Stakeholders can be internal to the program (residents/fellows, faculty, program coordinator) or the sponsoring organization (Graduate Medical Education Committee, designated institutional official, C-suite leaders). These stakeholders often seek evidence about training experiences and outcomes. Stakeholders can also be external to the program (accreditation bodies, medical school leaders) and may have additional questions that require different kinds of evidence. Patients and community leaders as external stakeholders can provide insights into alignment between the program goals and community interests. The selection of the program's evaluation focus¹  will determine the key stakeholders to include in the evaluation and their level of involvement in the process. For example, consider who is responsible or accountable for making decisions about the program, who should be consulted or who just needs to be informed (RACI).²  Spending time up-front to query stakeholders regarding the evidence they need to inform their decisions about the program will optimize their engagement and the evaluation's utility,³  increasing capacity for change.⁴ 

1. Identify key stakeholders. Consider those whose decisions and actions about the program or activity under review will have a primary effect on the evaluation approach, and thus, success of the endeavor.

2. Obtain internal stakeholders' perspectives. Ask stakeholders what evidence matters. What information would advise their decision or change their perspective on what is or is not working? There are multiple ways to obtain this information, ranging from surveys to brief phone calls. To determine the best data collection approach, use available resources to optimize interactivity within given constraints (eg, time, virtual dialogue), particularly with internal and sponsoring organization stakeholders.

3. Ask external stakeholders for guidance. Carefully examine accreditors' required processes and data sources and add those recommendations to your data blueprint.⁵  Consider perspectives from affiliated medical school deans who are involved with students' residency applications, organizations that hire your graduates, and community partners.

4. Visualize/diagram the data. Organize stakeholder perceptions about evidence that matters into an evaluation blueprint. Identify any evidence that cuts across key stakeholders and unique data that matters to high-stakes decision-makers. See shaded rows in the blueprint example (figure).

5. Assess alignment. Consider if the key evidence you want to obtain is consistent with your mission and aims. Will the evaluation process and use of the evidence meet program evaluation standards?³ 

1. Identify existing sources of evidence. Start by tapping into data you already have, consistent with the stakeholder-identified evidence. Data that can sufficiently provide insights for stakeholders are often already available. The Accreditation Council for Graduate Medical Education faculty and resident surveys provide comparison data to other programs (eg, recommend program, patient safety). Unobtrusive data such as online or e-learning participation rates, number of uncompleted chart notes, or program-required individual learning plans can also be used as evidence.

2. Determine the scope of the evaluation focus. Prior to collecting new data, check that the existing evidence is specific to evaluation focus. Eliminate what may be intriguing but not matched to this evaluation.

3. Take a 360-degree perspective. Consider triangulating data sources to obtain a robust understanding of the evaluation focus. For example, board pass rates may be a proxy for quality of graduates; however, richer insights can be gained by examining critical incidents and safety events, malpractice claims, patient experience scores, or community preceptors' evaluations of residents.

4. Continue to communicate with key stakeholders. Sustained stakeholder engagement in the evaluation and ultimately in the change process is optimized with brief, periodic updates about the evaluation progress and findings. Highlight and share evidence emerging from the stakeholder analysis. Provide periodic updates electronically, in physical or virtual face-to-face meetings, and seek stakeholder input and insights.

5. Seek evaluation expertise. Evaluation is a profession with its own scholarship. Engaging a formally trained individual with experience in evaluation models, principles, and methods can pay off in terms of improved accuracy, integrity, feasibility, and utility. An evaluator will pose questions from different perspectives and paradigms.