Feasibility of Implementing a Standardized Multisource Feedback Program in the Graduate Medical Education Environment

Multisource feedback (MSF) is an assessment approach that uses input from peers and colleagues to gather information about an individual's behavior in the workplace.1 This information is then aggregated and provided to the individual as feedback. Multisource feedback has been recommended by the Accreditation Council for Graduate Medical Education (ACGME) as a key method for assessing several of the competencies, including professionalism, and interpersonal and communication skills.2 Reliability, validity, and feasibility are central considerations in selecting assessment tools.3 

Feasibility in assessment considers costs, logistical considerations, and user acceptance.4,5 Despite its importance, feasibility is an underexamined construct that is given only superficial attention in many validity studies. There is a small base of literature supporting feasibility of MSF in the medical training environment.6,7 However, only a few studies8–14 address use of MSF in graduate medical education, including one of the early implementations of the MSF program described in this study.15 Authors note the time burden associated with survey completion8,11–14 and often low survey completion rates,15 participants' perceptions of program feasibility,14,15 or report that a given site or rater group was excluded on feasibility grounds without providing specific detail.9,12 

The organizational problems inherent in planning and carrying out an MSF program are only rarely addressed explicitly.1,15 Furthermore, few studies report on the feasibility issues of MSF implementations that span more than 1 site.16 If MSF is to achieve its potential in graduate medical education, it will be necessary to understand and find solutions to the implementation barriers.

Our study begins to address this gap in knowledge through an analysis of outcomes associated with a 2007–2008 multisite field trial of the Assessment of Professional Behaviors (APB). The APB is an MSF program developed by the National Board of Medical Examiners (NBME) to provide physicians, medical students, residents, and fellows with feedback on the professional behaviors that are essential to the safe, effective, and ethical practice of medicine. The APB program is built around a standardized MSF survey instrument developed with input from experts in medical education, medical professionalism, psychometrics, survey design, and industrial organizational psychology. The purpose of this study was to examine (1) program implementation, completion, and attrition; (2) the communication challenges associated with implementation; (3) reasons for implementation delays; and (4) benefits of the MSF program as perceived by participants.

A description of the MSF instrument used by all participating sites is provided in table 1. We chose behaviors for inclusion in the instrument based on surveys of potential respondents at the participating sites before program implementation; these surveys asked about the behavioral items' observability, clarity, and importance in the graduate medical education environment. Participating sites obtained local Institutional Review Board (IRB) approval for this study, and each site was responsible for obtaining consent from participants as required by their IRB.

The APB program includes a leadership orientation, participant orientation, feedback provider training, delivery of feedback reports, and feedback sessions. As part of the 2007–2008 field trial, a rater training experiment was also performed during the course of implementation. Each ABP milestone, along with the initial timeline of the APB field trial implementation, is described in table 2.

Residency and fellowship programs were eligible to participate in the field trial if they were located in the northeastern or midwestern United States and based at an institution where multiple programs were using 1 major residency management and evaluation system. The NBME contacted the designated institutional officials (DIOs) of institutions meeting these criteria to invite their participation in the field trial. Interested DIOs were asked to distribute field trial information to their program directors, including a request for statements of interest.

Programs were responsible for orienting all participants to the APB program. Potential raters included attending physicians, faculty, residents, fellows, nurses, and other staff. Subjects of the evaluation were primarily residents and fellows. As part of a randomized, controlled rater training experiment, all programs were asked to submit a list of individuals participating as raters. The NBME randomly assigned the raters to web-based training, workshop training, or no training. The NBME suggested that training be delivered approximately 2 weeks before participants were to be assigned the MSF instrument. The NBME worked with the programs and DIOs to schedule the workshop training sessions delivered by NBME staff and contracted training consultants. Instructions for accessing the web-based training were delivered locally through the program's evaluation system at about the same time as the live workshops.

Upon completion of the rater training, sites were responsible for assigning the MSF instrument. The instrument was loaded by the NBME into the evaluation software (E*Value, Advanced Informatics, Minneapolis, MN) and made available through the participating programs' accounts so that local staff could administer the MSF instrument on a locally defined schedule. Participating programs determined who would observe and rate whom and how often. After collecting MSF data for a period of time, the programs were to notify the NBME when they wanted to receive their feedback reports. Through a data exchange system between NBME and the evaluation software vendor, NBME generated and delivered reports to the individual observees (ie, those being observed and evaluated) through the evaluation system.

To convey the breadth of the field trial experience and the extent to which the reality of implementation departed from our expectations, we use mixed quantitative and qualitative methods17 to analyze both process and outcome data. Process indicator data included program attrition, timing of key implementation milestone completion, number of participants and number of MSF surveys assigned and completed, and adherence to the rater training experiment protocol.

To further understand the local implementation issues, interviews were conducted with stakeholders from the various sites, including those who completed an implementation cycle (defined as completion of at least 1 round of MSF data collection) and those who dropped out of the field trial before completing a full cycle. A small monetary incentive was offered to interviewees. A total of 28 stakeholders were interviewed, including 6 from sites that did not complete a full cycle. Stakeholders included program champions (those responsible for implementing APB at their site), program coordinators, faculty, residents and fellows, and designated institutional officials. A sample of the guiding questions asked of participants is provided as supplemental material.

Among the 27 programs at 5 institutions that submitted letters of interest, significant attrition from the field trial occurred almost immediately. One institution with 5 interested programs withdrew early in the process because the institution's implementation leaders had fundamentally misunderstood the purpose of the field trial, highlighting communication failure as one early barrier to implementation. An additional 10 programs withdrew before administering the MSF instrument, with the most common stated reasons for withdrawal being lack of faculty commitment and program resources for implementation.

During interviews, participants indicated that the time commitment required for rater training was a major obstacle to their participation. This concern was expressed among both programs that continued in the field trial and programs that dropped out. Program leaders also suggested that these delays reduced morale and diminished their motivation to participate in the field trial.

Of the 12 remaining training programs that started administering the instrument, 8 completed a full implementation cycle. table 3 shows that 303 unique observees participated and were rated with 2148 surveys. The average survey completion rate was 65.4%.

Postimplementation interviews with 22 key staff members at sites that completed an implementation cycle indicated that uncertainty regarding responsibilities remained at several programs. Communication challenges began during recruitment when the DIOs were identified as the initial point of contact regarding the APB field trial. As a result, many program champions did not immediately appreciate the resource commitment that would be needed. Additionally, while role expectations were communicated by NBME staff during the leadership orientation, staff members who were key to the implementation still were unclear about their responsibilities, in some cases owing to inability to attend the orientation. For example, some program coordinators indicated that they did not understand why they were involved, or did not anticipate the amount of work that would be required. Interview respondents reported that despite their sites' participation in these orientations, little information about the program made its way to local stakeholders before implementation. This is paralleled by the trainers' experience in conducting the workshop rater trainings. Many of the participants did not understand why they were at the workshops, or the purpose of the APB program, suggesting that not all participants had been oriented to the APB program.

Completion of every significant project milestone was delayed at some sites. Survey administration and feedback provider training were delayed by about 4 months at all sites. Structured feedback sessions were delayed by up to 8 months relative to the intended schedule.

Initial delays resulted from poor coordination of communication between the DIOs, the residency programs, and the NBME at several institutions. For example, 1 institution did not supply a list of participating residency programs until 4 months after leadership training was completed. This made it difficult to determine which programs would be participating in the field trial and thus delayed the scheduling of additional training workshops and the completion of subsequent implementation milestones.

A second reason for implementation delays was the rater training experiment. Sites took between 1 to 4 weeks to compile lists of participants for experimental assignment. Delays also occurred owing to the following:

• Negotiation with the sites over the scope of the experiment;

• Additional work required of the site administrators to schedule the participants into workshop sessions and assign web-based training in the evaluation software;

• An assignment process that was complicated by coordinators' varying familiarity with the software;

• Longer time required of the NBME to prepare the training modules than had been anticipated because of the need to train external consultants to deliver workshops, and because of information technology problems associated with the hosting of video content for the web-based rater training module;

• Workshop scheduling constraints due to the need to coordinate participant and trainer availability.

Despite the assignment of two-thirds of raters to a rater training intervention, uptake of training was minimal (table 4).

Furthermore, the assignment of MSF surveys was sufficiently delayed such that any impact of training on rater behavior would likely have been attenuated.

Despite the problems, field trial participants reported a number of benefits of participation. Generally, the problems with rater-based assessments were well known among the participants. Interview respondents indicated that they believed the APB program addressed some of those problems by providing appropriate structure and a clear focus on “professionalism” rather than being an add-on to existing assessments. The behavioral focus of the form was perceived as an improvement over current practice. Respondents also reported that the form was well structured and easy to use. Although participants expressed concern over rater fatigue, with some arguing that the 25-item form was too long, participants also recognized that a significantly shorter form would be too “vague” to be useful for giving feedback to learners.

Respondents recognized the difficulty of delivering feedback effectively and considered the APB to be helpful, and the feedback provider training program was also valued. However, dissemination of feedback provider training was subject to many of the same difficulties affecting dissemination of the other training modules.

Our research indentified several implementation challenges of the APB field trial, including communication gaps and difficulty scheduling onsite rater training. These challenges contributed to implementation delays, program attrition, and low training uptake. Participants identified several program changes that should enhance feasibility, including increasing communication and streamlining the training process. Despite the challenges, many participants still found value in implementing the APB program, as it helped to clarify performance expectations around professionalism and enhance the feedback process.

Our findings are consistent with those of previous research efforts that have identified the complexity and importance of communication in implementing an MSF program,1 and that have found MSF instruments to be of value in defining performance criteria about topics that may not otherwise be explicitly addressed.1,15 Our study, which includes analysis across several voluntary implementation sites in the United States, contributes to the limited literature on feasibility of multisite MSF implementation, which previously has been focused in the United Kingdom where MSF assessment is mandated.16 

An important limitation of this study is that it is based on a relatively small and likely nonrepresentative sample of training programs. As described in the analysis, program attrition from the field trial was significant and attributed often to lack of resources for implementation. Sample selection bias is therefore a potential problem and may have produced better outcomes in this study than should be expected in the general population. Additionally, dependence on 1 residency management software system may have yielded results that do not generalize to other systems. The impact of rater training, while potentially important, could not be measured in the current study.

Acceptance by participants is an essential element of MSF feasibility, and true acceptance cannot be established until participants are familiar with the components and requirements. Standardization and clear communication are therefore critical. Our experience underscores the importance of establishing the feasibility and acceptability of a well-defined intervention before attempting experimental research to demonstrate effectiveness.18 Our findings provide further evidence of the complexities involved in implementing an MSF program. Despite the problems encountered, participants in the APB field trial identified several benefits of the program. Streamlining the implementation process may increase the feasibility and standardize the use of the APB and other MSF interventions among a broad population of training programs. Our research points to some important ACGME program requirements, particularly frequent communication and reduction of barriers (eg, scheduling, technology) to fulfilling training requirements that should lead to better outcomes for any organization undertaking a new MSF program. Organizational learning seems to be a critical ingredient in the success of MSF.19 We anticipate that MSF implementation processes will improve over time as programs gain more familiarity with this assessment modality. Further research is warranted to better understand the resource requirements for a successful MSF implementation and to show how outcomes change as MSF gains acceptance by the broader graduate medical education community.