Multisource Feedback for Residents: How High Must the Stakes Be?

Clinical competence has been defined as “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” This powerful definition3 delineates the multifaceted nature of competence, as well as its context-dependent nature.4 “Doing well” here and now may be different from doing well in other contexts, and doing well in the eyes of peers may differ from doing well in the eyes of patients, managers, or other colleagues. Hence, doing well cannot be captured in a standardized single measure, or evaluated from one perspective or by one person. MSF, or 360° evaluation, is the combined evaluation of a person by multiple individuals that have different working relationships with the person, through use of questionnaires and a compiled feedback report. For trainees, reviewers generally include peers, supervisors, other health professionals, and patients.

Building a balanced image of a medical trainee, including strengths and weaknesses in an array of competencies, requires a broader view than most traditional methods of assessment. Competence is multifaceted and includes domains that are difficult to capture by traditional methods, among them professionalism, health advocacy, practice-based learning ability, scholarly approach, teamwork ability, and interpersonal skills.5–7 Many of these aspects cannot be evaluated by a single focused observation or by standardized assessment in a simulated environment. Miller's pyramid's highest level, “does,” is not an assessment setting. It is like the light in the refrigerator. We want to trust that it works—that is, that the light turns off when we close the door—but we cannot observe that property. We need observers from inside to inform us. MSF has the potential to inform a resident and a program director about observed behavior over a longer period of time under natural circumstances and by “insiders,” staff not otherwise responsible for formal judgments about trainees. Studies have shown that MSF can contribute particularly to the development of behavior in more intangible domains of competence, such as communication, timeliness of completing tasks, taking responsibility,8 and professionalism.9 

MSF was originally designed to be formative in purpose—to provide feedback to individuals to increase their awareness of their performance, and to guide learning and improvement. Used in this manner, concerns for the degree of rigor of the reliability and validity of the process are not as high as for MSF employed for summative purposes.

During the past decade, a more high-stakes version of MSF has evolved in medical education. Researchers have investigated psychometric and other properties of MSF that would enable its use in high-stakes summative decisions. This has included identifying the domains being assessed, developing items and scales, identifying and preparing appropriate reviewers, making decisions regarding inclusion of narrative comments and providing feedback, and maintaining confidentiality.10 This is laudable work. However, collecting valid information from multiple sources and feeding this information back in a structured manner that meets the rigorous standards required for high-stakes summative feedback pose a number of challenges. First, a certain number of reviewers are required to provide reliable and anonymous feedback: generally about 8 to 20, depending on the degree of variability in the reviewer group, the number of groups, and the number of questionnaire items. Logistically, there are multiple challenges in managing the recruitment of reviewers, data collection, report generation, and feedback in a confidential and organized manner. If this is done via paper, there is a lot of paper involved that must be collected and organized without mistakes. In addition, there are a substantial number of privacy issues. Personal information for feedback purposes is confidential and must be treated this way. All participants must be informed and understand their role in the procedure. Observers must be trained to be a credible and reliable source of information about residents.

The paper by Richmond et al11 in this issue reports challenges in the implementation of the Assessment of Professional Behaviors, an MSF program of the US National Board of Medical Examiners. Programs withdrew from this pilot study when they realized the extent of what was required to implement the MSF assessments. This is not a new finding. In a surgical residency program, Weigelt et al12 found that the burden of work to carry out MSF did not match the benefits because average scores did not substantially add to information from existing assessment tools. To use MSF for high-stakes evaluations much preparation is needed, including the training of many evaluators. However, if the emphasis is not on summative decisions and average scores, but rather on formative narrative feedback for growth, a more informal MSF approach may yield added rich information without a heavy burden of obligations on clinical staff.

If the MSF goal is primarily enhancing trainees' self-awareness of their own performance to enable improvement, the emphasis can be placed on the kind of feedback information that best facilitates these activities. Credibility and specificity of feedback are critical ingredients.13 Although most, if not all, MSF programs use numeric rating scales, useful for establishing psychometric qualities and making comparisons with others, evidence indicates that scores may be less helpful for feedback. In contrast, narrative comments are invaluable in raising self-awareness and guiding improvement.14 We believe MSF in medical education has its predominant strength when used for formative feedback and not for summative purposes. Also, facilitation by an engaged supervisor appears to be beneficial in helping the trainee to understand the feedback and make improvements.15 

In 2008 Utrecht University made available an easy-to-use, web-based MSF instrument for residents,16 targeted to Dutch residency programs (www.multisourcefeedback.nl). The instrument is designed to support the preparation for residents' annual progress interviews with their program director. It is recommended by a Dutch national authority for graduate medical education (the Central Bureau for Education in Healthcare), which also funded its development.

The Utrecht instrument is simple to access and use. Interested program directors registered with the Royal Dutch Medical Society can apply for an account. Next, a personal website is created that provides space to register residents, their e-mail addresses, and the date for closure of the feedback process, usually 1 to 3 months later, when a report will be generated. The program director starts the procedure by listing all residents who must receive MSF. The residents receive an e-mail that requests the e-mail addresses of multiple observers in 3 categories: medical colleagues (6 or more), other health care colleagues (6 or more), and patients (10 or more). The first two groups will receive an e-mail with a link to a confidential MSF subsite that contains a questionnaire with space for narrative comments: “tops” to stress positive observations, and “tips” for improvement. Patients are asked to participate following a clinical encounter, either after hospital or ambulatory care. Residents can observe the response process online and send automatic reminders, but they cannot identify or access individual responses. The procedure stops at the preset closure date; a report is generated and automatically sent to the resident and program director. All quantitative data are summarized in a small table, categorized according to the CanMEDS framework, and followed by a long list of “tips” and “tops.” Whereas the source of comments remains confidential to the resident, the program director may, for a limited time, identify respondents if desired. The program director and the resident can discuss the report in the resident's progress interview.

An evaluation of this use of MSF is currently underway and will include assessment of its psychometric properties. Since its launch in November 2008, the service has been used by more than 110 program directors to evaluate more than 700 residents in 38 hospitals and 33 disciplines, with more than 11 000 respondents. Participants receive training only via instructions provided by text on the website and supplemented in some cases by an oral explanation upon request. Informal feedback has been positive. The most common observations are that the procedure is user friendly, nonbureaucratic, and information rich, particularly because of the included “tips” and “tops” comments.

We propose that the primary goal of MSF should be learner improvement. Focusing on the logistic and personnel burdens of MSF for valid high-stakes decisions may distract from the many strengths of this approach for assessing and providing constructive feedback on clinical performance. It would be unfortunate if programs were to decide not to use MSF because of a logistical quagmire, whereas a simple formative approach is likely to yield invaluable feedback. We should continue to explore this promising method.