Providing Delayed, In-Person Collected Feedback From Residents to Teaching Faculty: Lessons Learned

Feedback is considered fundamental to performance improvement in medical education for trainees and faculty, especially as studies show self-assessment can be flawed.^1-4  The Accreditation Council for Graduate Medical Education requires residents have the opportunity to evaluate faculty, which is often called upward feedback.^5-7  The predominant goals of upward feedback are to improve clinical teaching and support faculty promotion.^6-9  While most faculty desire feedback from trainees,¹⁰  the literature is mixed on what type of feedback improves clinical teaching.^6,11  Numerical rating scales have not been shown to change teaching performance, while written comments have mixed results.^12-16 

An inherent issue in faculty feedback is the power differential; feedback to faculty is perceived as having greater consequences.⁸  Residents raise concerns that providing anything beyond praise could negatively impact relationships, given that clinical education occurs in small intimate teams.^7-9,17  If faculty react poorly, this could result in less positive evaluations or recommendations with potential downstream consequences for fellowships or jobs.^7,18,19  In addition, residents may feel uncomfortable providing constructive upward feedback.²⁰ 

Residency programs often make upward feedback anonymous to reduce resident concerns, yet resident discomfort persists.²¹  Small departments with fewer faculty find it difficult to maintain anonymity, and even in larger hospitals, faculty may attend only once or twice a year, making it possible to identify the resident.¹⁹  These concerns, along with resident workload and time pressures, often limit the amount of feedback to faculty resulting in vague generalized praise, rather than anything actionable. This violates the second tenant of the Feedback Intervention Theory, which posits that the effectiveness of feedback depends on a credible feedback source, specific and actionable content, and how the individual perceives the content.^8,13,22-24 

Ensuring anonymity also limits the ability to provide timely feedback, which is considered a cornerstone of feedback. Balancing timeliness and anonymity can pose challenges.⁸  It is unclear what time delay is needed to obtain anonymous and actionable feedback from residents.

With this question in mind, we developed a longitudinal approach to collecting resident feedback, promising anonymity by withholding release of their comments until they completed training. We sacrificed timely feedback to protect resident anonymity and anticipated residents would provide more constructive and candid comments.^7,19  We hypothesized that faculty would find this time-delayed feedback valuable and actionable.

This study was completed at Massachusetts General Hospital (MGH), an academic hospital in Boston, Massachusetts. Participants were inpatient teaching faculty and residents in internal medicine. During the time of the study, the residency program had more than 170 trainees. The general medicine inpatient teams consisted of a postgraduate year (PGY) 2 resident leading 2 to 4 interns (PGY-1) in the care of 12 to 24 patients, with 1 to 2 teaching faculty. Rotations were 2 weeks long and PGY-2 residents completed 4 to 6 rotations.

From 2011 to 2017, the inpatient associate program director (APD) (K.M.F.), held a brief 10-minute meeting with each PGY-2 resident shortly after their rotation to ask about their experience with the faculty. Residents had the opportunity to discuss any issues, but the meeting largely focused on feedback. Residents were told their feedback was confidential, would be combined with that of other residents, and released only after they finished training.

The residents were asked 4 questions, chosen based on review of the literature: (1) How were your attendings? (2) What was their teaching style? (3) Can you describe the level of autonomy? and (4) What were their strengths and areas for improvement? Residents’ responses were typed up in real time and saved on a hospital shared drive with access limited to study personnel. During this study period, the program expected residents to provide verbal feedback and complete anonymous written evaluations released yearly.

In 2019, individual feedback about each faculty was combined, rearranging the chronology to further reduce the risk of resident identification. Each resident’s comment was a distinct paragraph. To evaluate faculty’s response to this feedback, faculty completed a timed 3-part survey from January through March. Faculty were informed about the project via email and asked about their prior experience with resident feedback on inpatient services. Upon completion of survey 1, faculty were emailed their personal feedback, along with an explanation of the process and examples (online supplementary data Appendix A and B). Immediately after reading their feedback, they were asked to complete survey 2. Faculty were offered the opportunity to meet with the APD to discuss their feedback. Faculty who did not want to participate in this study were emailed their feedback separately. Since feedback often triggers emotions, faculty were sent the survey 3 months after receiving their feedback to assess if their views changed over time.²²  The secure web-based application Research Electronic Data Capture (REDCap)^25,26  was used to manage survey distribution and collect responses, and automated reminders were sent out at 1, 2, and 4 weeks. We obtained outside email addresses for faculty who had left MGH. Those completing survey 2 were entered into a lottery for a gift card, and all received a $5 coffee card for completing survey 3.

The 3 surveys were developed from a literature review and description on types of feedback.^11,27  The first survey drafts and research methodology were presented to an advisory panel of 20 medical education researchers for feedback. Two experts in survey design reviewed and edited the surveys, and 2 faculty not in the study pilot tested them prior to distribution (see online supplementary data for surveys). The surveys collected basic demographic information (age, gender, and years post-residency training). Items were rated on a 5-point Likert scale or had open text boxes. Based on a previously published model,^27,28  faculty were asked to estimate whether feedback they received contained praise (positive statements about you as a teacher), positive constructive feedback (affirming comments about specific behavior), negative constructive feedback (corrective comments about specific behaviors), and criticism (negative statements about you as a teacher). Identifying information was kept separate to ensure anonymity and confidentiality.

Descriptive statistics were used to summarize data, such as the characteristics of respondents. Inferential statistics were used to analyze the responses between surveys, where the surveys were analyzed as independent, rather than repeated data, which is a more conservative approach and allowed for more data points to be included in specific analyses. As some data were not parametric, nonparametric analyses were completed. To compare the percentages of 4 specific types of written feedback faculty received between baseline and the new approach, Mann-Whitney U tests were used. To analyze proportions of faculty agreement, faculty confidence, and faculty perception of quality, 2 proportion z-tests were used to sequentially compare responses from 2 surveys at a time. Stata 17 (Stata) and 2 online calculators (Social Science Statistics and Statology) were used for these analyses. Statistical significance was accepted at P=.05.

This study was deemed exempt by the Partners Human Research Committee.

During the 6 years, feedback was collected from 371 PGY-2 residents about 251 faculty. Faculty were removed if they had retired, died, or had feedback from only one resident, which would have made them easily identifiable, resulting in 189 eligible faculty. Survey 1 was completed by 157 faculty (83.1%), survey 2 by 151 (79.9%), and survey 3 by 140 (74.1%). Survey responses from the 140 faculty who completed all 3 surveys were included in the analysis. More respondents were men (59.3%), 11+ years out from training (55.7%), and served as attendings for 5+ years at MGH (62.1%; Table 1). This matched current MGH faculty demographics.

Based on prior experience, the majority of faculty reported receiving verbal feedback at least some of the time (37.1% [52 of 140] often/always, 41.4% [58 of 140] sometimes) while far fewer received written feedback (2.9% [4 of 140] often/always, 12.1% [17 of 140] sometimes; Table 1).

After reading their new feedback, 72.9% (102 of 140) of faculty agreed/strongly agreed it matched expectations. When asked to compare the new feedback to prior feedback, written or verbal, 81.0% (81 of 100) agreed/strongly agreed the feedback was higher quality, 82.4% (84 of 102) higher quantity, and 85.4% (88 of 103) more specific. When asked about the new feedback approach, 77.9% (109 of 140) agreed/strongly agreed it was structured well, 93.6% (131 of 140) said they planned to reflect on their feedback, and despite the time delay, 32.1% (45 of 140) believed they could identify some residents (Table 2).

When comparing the 4 specific types of feedback, the results highlighted differences between prior feedback and the new feedback in the percentage of praise and negative constructive feedback, identified by faculty. Descriptive results noted a decrease in praise (median: 50.0%, [IQR 35.0-75.0%] vs median: 35.0%, [IQR 20.0-50.0%], P<.001), and an increase in negative constructive feedback (median: 5.0%, [IQR 0.0-15.0%] vs median: 20.0%, [IQR 10.0-25.0%], P<.001; Table 3).

When asked about the feedback content compared to prior feedback, more faculty agreed/strongly agreed the new feedback focused on teaching (92.9% [130 of 140] vs 75.9% [101 of 133], P<.001), clinical knowledge (72.1% [101 of 140] vs 51.9% [69 of 133], P=.001), leadership skills (67.9% [95 of 140] vs 43.6% [58 of 133], P<.001), and time management (72.1% [101 of 140] vs 35.3% [47 of 133], P<.001). However, fewer responded that the new approach focused on interactions with patients (47.1% [66 of 140] vs 70.7% [94 of 133], P<.001). More faculty were confident the feedback provided via the new approach was truthful compared to prior (62.9% [88 of 140] vs 7.5% [10 of 133], P<.001).

Faculty agreed/strongly agreed the new feedback was more valuable than prior written feedback (94.3% [132 of 140] vs 67.1% [49 of 73], P<.001) and a higher proportion agreed the new feedback would likely lead to a change in their teaching behaviors (82.9% [116 of 140] vs 50.7% [37 of 73], P<.001; online supplementary data Table).

Nearly all faculty reported a belief that residents should provide feedback, which did not change with the new feedback (98.6% [138 of 140] vs 96.4% [135 of 140]). Over a third of faculty believe resident feedback is only valuable when given in a timely fashion, which did not change after receiving this time-delayed feedback (37.1% [52 of 140] vs 35.7% [50 of 140]). Faculty were split whether a time delay is needed to obtain constructive feedback and a sizable number (41.4%, 58 of 140) felt the time delay reduced the value of the feedback. There were no differences between prior and new feedback in how the feedback made them feel (online supplementary data Table).

Three months after receiving their feedback, most faculty still agreed the value of the feedback was high, but the percentage had decreased (94.3% [132 of 140] vs 84.3% [118 of 140], P=.01). There was a decrease in agreement they would change teaching behaviors (82.9% [116 of 140] vs 63.6% [89 of 140], P<.001). When asked how the feedback made them feel, there was a decrease in interest to improve their teaching (87.1% [122 of 140] vs 77.9% [109 of 140], P=.04) and an increase in frustration about their teaching role (12.9% [18 of 140] vs 23.6% [33 of 140], P=.02; Table 4).

We explored a longitudinal approach to providing feedback to teaching faculty from residents, sacrificing timeliness in exchange for anonymity, resulting in more constructive and specific feedback. A majority of faculty reported that this new feedback was more valuable, truthful, and higher quality than prior experience and would cause them to change their teaching behaviors. More than a third noted the time delay decreased the value of the feedback. They were divided as to whether a time delay was necessary to obtain the anonymous content, as a third of faculty believed they could still identify some residents.

This study echoes findings from others’ work that obtaining feedback from busy residents is challenging.^13,14  Factors that contribute to this include resident workload and time pressure, along with fear of repercussions for any feedback beyond praise.^7,8,17,19,29  The business literature advises caution in providing upward feedback to bosses, and higher education literature supports anonymity to protect learners.^21,29-31  There are limited studies in graduate medical education (GME) regarding the best methods for evaluating faculty, including the impact of anonymity.^8,11,13,18  One study evaluating non-anonymous feedback found some benefits, but noted that residents worried about consequences, and concluded anonymous feedback must be available.⁸  With small clinical teams, anonymity is difficult to achieve in GME. Even with a significant time delay, faculty in this study believed they could still identify some residents.

Feedback Intervention Theory (FIT) suggests feedback is more effective when it avoids praise and criticism and focuses on specific behaviors.^11,22-24,32  In our study, this method of collecting feedback shifted the balance from praise to constructive feedback, with faculty reporting that they would change their behavior. Another tenant of FIT is the trustworthiness of the source. In order to accept feedback, recipients need to believe feedback is honest and accurate.^4,21,29  In an earlier study on non-anonymous feedback, faculty were concerned that residents “moderated the message.”⁸  In our study, nearly 63% of faculty were confident residents were more truthful, making feedback potentially more actionable.^33,34  Finally, the third tenant of FIT is how the recipient perceives and reacts to the feedback, especially emotionally. Our study asked faculty to categorize their perceptions of the feedback initially and then later, to determine if processing the feedback had changed their views. With a time delay, faculty were less positive about the feedback, perhaps reducing its value, since feedback effectiveness is in large part determined by the recipient.

We observed that this method of collecting feedback required a considerable time investment, which may not be possible at other institutions. However, there were several additional benefits, including sending an unwritten message that feedback about faculty was important and enabling real-time awareness of concerns. All residents agreed to meet with the APD.

This study has multiple limitations, including being a single-center study with its own feedback culture and practice. The feedback provided spanned 6 years, resulting in a significant time delay, which likely contributed to a reduction in its value. Bias was likely introduced by notifying faculty with a letter explaining the new feedback process and contextualizing the content immediately before comparing this feedback to prior experiences. In addition, the time between prior feedback and reading the new feedback was highly variable for each faculty, which also may have affected faculty perception. Because the residents’ feedback was typed up verbatim by one individual, this may have introduced partiality to the process. Our survey design included agreement scales, which can introduce bias. While the survey response rate is greater than other studies of physicians, 17 faculty nonresponders may bias the results in unknown ways.³⁵ 

The optimal time delay for ensuring anonymity but still providing valuable upward feedback to faculty remains unclear.³⁶  Future research might investigate shorter time delays than those used in this study, assessing resident anonymity and comfort, feedback quality, and faculty perception of its value.

This approach to obtain resident feedback for teaching faculty resulted in more constructive feedback, which faculty perceived as valuable and planned to utilize. The anonymity achieved with a significant time delay likely resulted in improved content, but faculty were divided on whether the delay’s benefits justified the tradeoff.

The authors would like to thank the Massachusetts General Hospital internal medicine residency program and teaching faculty for their support and cooperation and the MGH Education Council for reviewing the methods and surveys.

The online supplementary data contains instructions for faculty, an example of resident comments about an attending, further data from the study, the surveys used in the study, and a visual abstract.

Funding: A small grant was obtained from the Massachusetts General Hospital Department of Medicine’s Center for Educational Innovation and Scholarship.

Conflict of interest: The authors declare they have no competing interests.