Improving Narrative Feedback for Resident-Led Academic Rounds: The Effects of Assessment Form Design Changes

Feedback is an integral part of learning and has been shown to change trainees' behavior.^1,2  Learners benefit more from narrative feedback than numerical rating^3,4  when it is timely, specific, and with a clear goal of where to take action.^3,5-7  However, narrative feedback is often lacking in both quality and quantity,^1,3,8,9  and medical educators face many barriers to providing feedback.^1,2,5,10 

Despite evidence that medical learners benefit from narrative feedback, increasing the quality of this feedback remains a challenge. A potential intervention intended to emphasize the importance of narrative feedback, and possibly reduce the barrier of time, is moving the narrative comment section to the top of an assessment form. Previous research has tested this formatting change: one found no change in the specificity of comments,¹¹  and another found improved narrative feedback when the format change was accompanied by prompts.¹²  In this study, we explored if a format change would result in more frequent and specific feedback. We evaluated the quality of the feedback according to 3 key elements (task, gap, and plan) of deliberate practice as described by Ericsson: description of the task as a well-defined learning goal (task), identification of a gap between observed performance and a superior standard (gap), and description of a learning or action plan (plan).¹⁰ 

This single-site study took place in the psychiatry residency program at the University of Manitoba in Canada. Postgraduate year (PGY)-3 core residents and PGY-5/PGY-6 child and adolescent subspecialty residents present at academic case presentations during their rotations. These presentations typically occurred biweekly, and the presenters were assessed by the round's attendees including psychiatrists, trainees, and other health professionals, using a paper assessment form distributed at the time of the event.

From January to June 2017, we collected the original academic rounds assessment form (Form 1, provided as online supplementary data) that included 10 Likert scale questions and a narrative comments section located at the bottom of the form. Starting in July 2017, a new assessment form (Form 2, provided as online supplementary data) was distributed at rounds and was identical to the previous form except that the narrative comments section was located at the top of the page. The change in form was not announced or explained to assessors, nor did any of the processes surrounding the use of assessment forms change, such as their distribution, collection, and reception by residents. S.C. recorded the word count and the presence or absence of any comment from each assessment form.

To determine the presence and extent of the three components that facilitate deliberate practice, all de-identified narrative comments were coded using a tool designed by Gauthier et al¹³  and further adapted by Abraham et al.¹⁴  C.R., a medical education researcher, and S.C., a psychiatry resident, independently evaluated all narrative comments using the adapted tool. The raters first familiarized themselves with the tool and scored a small portion (10%) of randomly generated assessment forms together and discussed any discrepancies. The raters then independently evaluated the remaining forms and assessed their interrater reliability using the Kappa statistic. They subsequently met to discuss any discrepancies and agree on a rating.

Using SPSS Statistics 25 (IBM Corp), we calculated the differences before and after the format change in the quantity and specificity of comments with the chi-square test. Differences before and after the format change in word counts were calculated using an independent samples t test.

The study was approved by the Research Ethics Board at the University of Manitoba (Protocol #HS21443).

Assessors completed 93 assessment forms with the comments section on the bottom and 133 forms with the comments section on top. The kappa correction coefficients between the 2 coders' ratings of the components of task, gap and action were 0.937, 0.902, 0.941 respectively, showing a high level of agreement.

Figure 1 summarizes the proportion of deliberate practice components in the assessment forms before and after the form change. There was a significant difference in the specificity of comments before and after the form change related to the task, or what was done well (X²(3)=20.12, P=<.0001). There were no significant differences in respect to the gap and action scores.

There was no significant difference in word count between the 2 assessment forms. The mean word counts when the comments section was located on the bottom of the form was 17.6 (SD=14.51) and 17.27 (SD=17.07) when the comments section was on top. There were significantly more comments present (ie, presence of any words within the comment sections as opposed to left empty) when the comments section appeared on top (X²(1)=6.54, P=.011). These results are illustrated in Figure 2.

This study explored the effect of moving a narrative comment section of an assessment form to a more prominent position to determine whether this improved feedback. Our analysis of the written comments suggest that having the comment section at the top is associated with a modest increase in the number of comment sections completed as opposed to left blank, and an increase in specificity in the task component, or what was done well. There was no significant increase in word count or in descriptions of the gap component (difference between observed and desired performance) or the action component (how improvement might be accomplishment).

Similar to the study by Dory et al, we found that the location of the comment section resulted in improvements to the quality of narrative feedback, which suggest that location alone contributes to the quality of narrative comments. Our findings are similar to previous studies; compared to the gap and action components of deliberate practice, task was more frequently identified.^11,12  One reason for this is that task may be the easiest component to identify by simply recording a detailed account of the task performed.¹¹  The gap and action component may also be implied by some assessors through the numerical measures, both because it may be seen as being less time intensive and may be felt to protect the learner as more specific written feedback could be experienced as critical if not worded thoughtfully.

Results of this study suggest that narrative feedback or comment boxes located at the top of assessment tools has the potential to improve feedback provided to learners. Simple formatting changes to assessment tools require very little time and money and have the potential to benefit learners, educators, and even administrators. Additionally, future research might explore the effect of adding simple prompts to the top placement of narrative comment boxes. Furthermore, efforts should be made to decrease focus on less valuable aspects of assessments such as reducing the number of Likert ratings.

There are limitations to the study. This is a site- and discipline-specific study, and we focused on only one form of feedback that residents receive. Results may differ with other forms of feedback in different settings. Additionally, 2 of the study's co-authors participated in both phases of the assessments and were aware of the study beginning in phase 2; consequently, this may have changed how they completed assessments forms in the second phase of the study.

We found that placing the narrative comments section more prominently at the top of the form increased the feedback residents received by increasing the number of assessments with comment sections completed and increasing the specificity of the task component of the deliberate practice model.

The authors would like to thank Mark Bigder, MD.