The Challenge

Multiple snapshots of a trainee's real-time performance are required for program directors and Clinical Competency Committees to provide a comprehensive performance assessment and deliver formative feedback.1  Multi-source feedback (MSF), or 360-degree evaluation, is a formal questionnaire-based assessment method that relies on workplace-based direct observation of performance by multiple raters from different perspectives.25  While decades of MSF studies in medical education support its feasibility and provide validity evidence,24  educators wishing to implement MSF programs may be intimidated by issues of time, labor intensity, and scope.1,5,6 

What Is Known

The MSF approach is best suited for direct observation of real-time trainee performance. Using diverse raters' perspectives over multiple observations yields a more comprehensive assessment of a learner's performance than any single-source approach.24  The MSF approach can be powerful for assessing behaviors in nontechnical or nonprocedural domains, such as professionalism, interpersonal and communication skills, and practice-based learning and improvement, that are difficult to capture by traditional methods.1,3,4  Educators planning to develop an MSF tool must adhere to a rigorous development process to optimize instrument feasibility, reliability, and validity as well as to avoid measurement errors. Steps in this process include the development of items or the use of existing items that will yield specific, meaningful feedback to trainees,1,6  the identification of the behaviors that each rater group is best suited to observe and judge, determining who the assessor(s) are, and the training of raters. Validity evidence can be demonstrated by correlations between MSF instruments (concurrent validity) and between MSF ratings and other assessment measures (predictive validity).3 

Rip Out Action Items

Program directors should:

  • 1. 

    Seek to implement assessment programs that incorporate data from multiple sources. These should be based on multiple snapshots of real-time in situ workplace observations.

  • 2. 

    Prioritize adopting, adapting, or creating multi-source feedback instruments to optimize validity evidence while ensuring feasible implementation (eg, allows the rater to assess the performances they are best suited to observe).

  • 3. 

    Continue to cultivate a feedback culture that is safe and supportive and that promotes frequent feedback conversations between learners and credible raters.

Sources of the Feedback: The Raters

The MSF raters can include self, peers, supervisors or preceptors, nonphysician coworkers (eg, nurses, social workers, pharmacists), and patients.3,6  Self-assessment plays an integral role in the MSF process (table). Trainees' self-assessment, when compared with other MSF ratings, may identify gaps needing further attention.24  Eight to 20 credible rater assessments (depending on the source) are needed to optimize validity. Residents perceive raters as credible when raters directly observe their performance and are capable of providing meaningful feedback about that performance.1,4  Thus, each rater must observe the trainee on the unique set of domains matched to the scope of their role and relationship with the trainee in order to provide feedback that will be perceived as credible.46  Trainees have identified patient feedback as a particularly powerful trigger for behavior change.4,6 

table

Multi-Source Feedback Sources by Feedback Domain

Multi-Source Feedback Sources by Feedback Domain
Multi-Source Feedback Sources by Feedback Domain

Delivering MSF: The Impetus to Change Behavior

The MSF approach may be more likely to lead to behavioral change when trainees buy in to the goals and value of MSF from credible raters.2  Facilitated feedback delivery by a mentor or coach may improve learner receptivity and incorporation of MSF.4  Adding narrative comments to numerical item scores makes the feedback more meaningful.4,6  Trainees report that guidance in identifying gaps between self-assessment and MSF results improves MSF acceptance and helps them generate an action plan to change behavior.4 

How You Can Start TODAY

  • 1. 

    Perform a local needs assessment: How is your program already obtaining real-time observation data? How can you expand MSF instruments and sources? Identify existing survey-based feedback of trainees and opportunities to expand.

  • 2. 

    Educate residents, faculty, and ancillary staff about the rationale behind MSF and the professional behaviors that are most amenable to MSF assessments.

  • 3. 

    Adopt or adapt available MSF tools, and pilot these in a simulated educational setting to increase learner and faculty familiarity with the tools. Sample items and instruments are available for adaptation.2,79 

  • 4. 

    Select and train raters that trainees will perceive as credible; raters must be able to directly observe the behaviors you intend to assess.

What You Can Do LONG TERM

  • 1. 

    Consider faculty development aimed at promoting a culture of feedback within your training program and institution that includes MSF.

  • 2. 

    Build a framework to support MSF use in Clinical Competence Committees for subcompetency assessment and to inform entrustment decisions.

  • 3. 

    Provide MSF to trainees as part of semiannual reviews to identify improvement targets with action plans.

  • 4. 

    Implement MSF to assess faculty teaching; incorporate this into the annual performance review with the department chair.

Resources

Resources
1.
ten Cate O, Sargeant J
.
Multisource feedback for residents: how high must the stakes be?
J Grad Med Educ
.
2011
;
3
(
4
):
453
455
.
2.
Lockyer
J.
Multisource feedback in the assessment of physician competencies
.
J Contin Educ Health Prof
.
2003
;
23
(
1
):
4
12
.
3.
Donnon
T,
Al Ansari
A,
Al Alawi
S,
et al.
The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review
.
Acad Med
.
2014
:
89
(
3
):
511
516
.
4.
Ferguson
J,
Wakeling
J,
Bowie
P.
Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review
.
BMC Med Educ
.
2014
;
14
:
76
.
5.
Berk
RA.
Using the 360 degrees multisource feedback model to evaluate teaching and professionalism
.
Med Teach
.
2009
;
31
(
12
):
1073
1080
.
6.
Sargeant
J,
Mann
K,
Sinclair
D,
et al.
Challenges in multisource feedback: intended and unintended consequences
.
Med Educ
.
2007
;
41
(
6
):
581
596
.
7.
MedEdPortal
.
Development and implementation of multi-source assessment tools for ACGME residents and fellows
. ,
2017
.
8.
MedEdPortal
.
How to develop, implement, and assess multi-source feedback (MSF) for trainees
. ,
2017
.
9.
The Royal College of Anaesthetists
.
Multi-source feedback (MSF) summary form
. ,
2017
.

Author notes

Editor's Note: In order to expand the resident assessment toolbox for programs and medical education researchers, JGME is presenting a series of Rip Outs on assessment instruments “beyond surveys.” This multi-source feedback Rip Out is the first in this series.