Improving Pediatric Fellows’ Feedback Skills and Confidence Through Objective Structured Examinations Open Access

The ability to provide feedback is an important communication skill for medical trainees who are educators for students and residents.^1,2  Without feedback, residents may continue incorrect practices, and good behaviors may not be reinforced.³  Most clinicians do not receive training in delivering feedback.^1,4,5  While many publications describe various feedback models,²  few have evaluated feedback training during fellowship with objective measures.^6-11  Our objective was to address this gap by providing training for fellows to improve their skills and confidence in providing feedback using simulated feedback situations^2,12,13  through an objective structured feedback examination (OSFE).

This study was conducted from 2019 to 2020 at a children’s hospital that trains medical students, residents, and pediatric subspecialty fellows. We solicited participation from 20 medical and surgical subspecialties (n=73 fellows). Fellowship program directors completed a needs assessment survey describing what type of feedback training, if any, their fellows received, and whether they would allow fellows to participate. Nineteen of 23 directors (83%) responded; the majority (18 of 19, 95%) expressed interest in feedback training. The survey confirmed lack of feedback curricula in most fellowships, and those that provided feedback training were excluded. A recruitment email was sent to the 60 eligible fellows. They provided verbal consent to participate and were informed that their voluntary participation would not affect their fellowship assessments.

We followed Kern’s 6 steps for curriculum development (online supplementary data Figure 1).¹⁴  We created goals and objectives for our training and identified educational strategies (online supplementary data Figure 2) and measured outcomes with an OSFE, similar in format to an objective structured clinical examination (OSCE).¹⁵  We created the training session, scenario, and assessments based on literature review.^1,2,4,16-18  The training included feedback goals, models, and strategies for giving and receiving feedback. The OSFE scenario involved a struggling intern (online supplementary data Figure 3), played by a standardized recipient; no script was provided to allow for variety. To facilitate consensus, standardized recipients and faculty were trained to score performances and practiced using the checklist with 3 test subjects.

Fellows scheduled a 1-hour session at the simulation center and completed pretraining self-assessments on prior exposure to feedback training; application (ie, how often they give feedback); and self-perceived knowledge of and confidence in ability to give feedback (online supplementary data Figures 4, 5, 6).

Fellows then participated in an OSFE using the common scenario and provided feedback to the standardized recipient. The faculty observed from a separate room using live-stream video to support objective assessment. One standardized feedback recipient and one faculty investigator assessed fellow performances using a checklist (online supplementary data Figure 7).¹⁰  Then fellows participated in the 20-minute interactive feedback training session. Afterwards, the observer and recipient gave fellows feedback on their performance. Fellows then participated in a second OSFE using the same scenario. To decrease “practice effect,” in which repeated evaluation results in improvement, the scenario had multiple problems that the standardized recipient exhibited. Before leaving the session, fellows completed another confidence self-assessment and a training evaluation.

Due to the COVID-19 pandemic, the 3-month post-assessment was adapted by offering 30-minute videoconferencing sessions. Fellows completed another confidence assessment and application assessment, then participated in a third OSFE using the same scenario. The faculty observer was hidden and muted. Performances were scored as before, and fellows were given feedback.

Our primary outcome measure was change in OSFE performance. Our secondary outcome measures were changes in self-reported confidence and application of feedback skills, which were measured on a 5-point Likert scale. To rate OSFE performance, we modified a published checklist, the FEEDME-Provider instrument, by removing questions that did not apply in simulation.¹⁰  OSFE performance was scored along several domains: self-assessment, providing corrective feedback, and facilitating bidirectional conversation (online supplementary data Figure 7). Fellows provided satisfaction ratings and suggestions for improvement at the end of the session.

Change in confidence and OSFE performance were assessed using signed rank sum test. Change in application of feedback skills were assessed using paired t test. All analyses were conducted using SAS v9.2. P value of <.05 was considered significant.

The protocol was exempted by the University of California San Diego Institutional Review Board.

We enrolled 19 of the 60 invited pediatric fellows (32%); 100% participated in all assessments. All fellows showed improvement in OSFE performance after feedback training and individualized feedback with a mean change of +0.89 (P<.001; Table 1) aside from 2 items already high at baseline. All fellows showed improvement in self-perceived confidence with a mean change of +2.07 (P<.001; Table 2). Three months later, improvement in skills (mean change +0.92, P<.001) and confidence (mean change +1.67, P<.001) were sustained.

All fellows reported prior exposure to the feedback sandwich model. Most (63%, 12 of 19) reported prior exposure to the ask-tell-ask feedback model. Few had prior exposure to just-in-time feedback (42%, 8 of 19). Most fellows reported prior exposure to skills such as how to deliver formal sit-down feedback (74%, 14 of 19), deliver feedback to a problem learner (58%, 11 of 19), receive feedback (84%, 16 of 19), and self-assess (79%, 15 of 19). Only 32% (6 of 19) reported exposure to how to direct a learner to self-assess. At baseline, only 11% (2 of 19) reported giving sit-down feedback at least once every 2 to 4 weeks; only 21% (4 of 19) reported giving just-in-time feedback. Three months after training, 21% (4 of 19) reported performing sit-down feedback at least once every 2 to 4 weeks (P=.06); 63% (12 of 19) reported giving just-in-time feedback (P=.01).

Fellows (N=18) evaluated feedback training after the initial session as positive (speakers 5/5 on 5-point Likert scale for organization, engagement, effective content delivery). All reported the session met its objectives, was relevant, logical, and clear (5/5), and rated audio/visual aids easy to comprehend, session useful, and recommended to colleagues as 4.9/5. Fellows noted they learned the ask-tell-ask feedback model, importance of learner self-assessment, bidirectional feedback, and scheduling feedback time. Suggestions included using less text in the training presentation.

Our feedback training program, using an OSFE and modified FEEDME-Provider tool¹⁰  for assessment, resulted in improved feedback skills in fellows immediately after intervention and 3 months later. Fellows had improved self-perceived confidence in feedback knowledge and skills.

Constructive feedback is vital to help trainees improve but challenging to deliver. Use of simulated standardized cases and feedback in a low-stakes environment may allow fellows to feel more comfortable applying these skills in the clinical setting; thus, simulation has been increasingly used in medical education.^2,12,13  Combining interactive communication scenarios with didactic teaching may be beneficial to allow for multimodal learning.^19,20 

As the feedback training program was well-received and showed sustained improvement in feedback performance and self-reported confidence and application of skills, these methods and assessments could be expanded to residents or faculty and utilized to address other topics in medical education.

Several features limited generalizability. This study was conducted at a single institution with low enrollment rate, possibly related to COVID-19 pandemic stresses. Participants may have been particularly motivated to learn feedback skills, leading to self-selection bias. Shifting to videoconferencing may have introduced variability in our results but was easier to schedule and familiar for participants.^21-23  The OSFE scenario was the same throughout; practice effects might have biased results, though interactions were unscripted. Finally, this intervention was time- and personnel-intensive. However, we have successfully adapted our intervention to larger groups using small-group peer feedback for teaching and assessment. Further studies should use different scenarios and assess longer-term retention and need for booster sessions.

Giving feedback is a fundamental skill for clinicians, but formal feedback training is rare. We showed that feedback training using an interactive workshop, standardized recipients, and individualized feedback improved feedback performance, confidence in feedback skills and knowledge, and application of skills in pediatric fellows.

The authors would like to thank the pediatric subspecialty fellows for participating in the educational intervention as well as the pediatric subspecialty fellowship directors for completing needs assessments regarding feedback training within their curricula and allowing their fellows to participate. Thank you also to the Association of Pediatric Program Directors Leadership in Educational Academic Development mentors and Cohort 8 for providing support and mentorship.

The online supplementary data contains resources used in the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

This work has been previously presented virtually at the Academic Pediatric Association Region 9 and 10 Virtual Meeting, January 30, 2021; Association of Pediatric Program Directors Annual Spring Meeting, March 23-26, 2021; Pediatric Academic Societies Meeting, April 30-May 4, 2021; Pediatric Hospital Medicine Conference, August 4-6, 2021; and UC San Diego Inaugural Symposium for Innovation in Medical Education, September 10, 2021.