myTIPreport and Training for Independent Practice: A Tool for Real-Time Workplace Feedback for Milestones and Procedural Skills

Direct observations of clinical performance and the resulting feedback are important for residents and fellows to successfully progress to independent practice.^1–4  Challenges to feedback include duty hour limitations, reduced faculty contact with learners, increased pressures for clinical efficiencies, faculty reluctance to provide negative feedback, and faculty perceiving they lack skills for giving feedback.^5–10  Clinical teachers may not have access to learners' clinical progress to date, and there is a lack of a centralized, accessible repository for formative feedback. We designed myTIPreport to address these multiple feedback barriers. “TIP” stands for “Training for Independent Practice,” and the tool was developed for workplace interactive feedback on both the Accreditation Council for Graduate Medical Education (ACGME) Milestones and procedural skills. Real-time workplace feedback was defined as feedback in the clinical setting immediately or shortly after direct observation of a learner's clinical performance.

The aim was to gather validity evidence for myTIPreport by applying the framework outlined in the Standards of Educational and Psychological Testing.^11,12  To do so, the tool was designed and implemented at multiple, geographically diverse, US institutions over 6 months. Tool development was supported by the Foundation for Excellence in Women's Health, a nonprofit medical education organization.

A convenience sample of 19 geographically diverse, ACGME-accredited obstetrics and gynecology (Ob-Gyn) and female pelvic medicine and reconstructive surgery (FPMRS) training programs agreed to pilot myTIPreport between November 2014 and May 2015.

Participating teachers and learners were e-mailed a 21-item preimplementation survey and consent form via Qualtrics software (Qualtrics LLC, Provo, UT) prior to using the educational tool. The survey was developed by the primary author without further testing (available from the corresponding author). Teachers and learners were queried on aspects of real-time feedback utilized in their programs, and asked to rate their satisfaction with current feedback frequency, quality, and timeliness on a 5-point Likert scale (1, extremely satisfied, to 5, not at all satisfied).

Software engineers designed the web-based myTIPreport platform with a secure, user-specific login and password. It allowed users to perform a variety of feedback functions, including generating new formative feedback and review of past formative and summative feedback. myTIPreport included the ACGME Milestones and procedures tracked through ACGME case logs. The tool was built to allow teachers and learners to view current and previously assessed procedural feedback and milestone subcompetency levels assigned to each learner. It overtly labels learner behaviors to be performed and assessed to build response process validity by labeling clearly for teachers and learners how learners should be rated and by providing checklists. myTIPreport uses direct data entry to ensure integrity of data stored and its interpretation.

All ACGME Milestone sets for Ob-Gyn residency (n = 28) and FPMRS fellowship (n = 23) were incorporated into myTIPreport, with ACGME approval for use of the copyrighted milestones.^13,14  For each level-specific milestone, teachers recorded responses of yes, observed, or not applicable. As the learner and teacher completed the feedback checklist, they were blinded to milestone level to reduce assessment bias. Narrative comments could be recorded by teachers and learners. After feedback is completed, myTIPreport then assigns a milestone level. The learner achieves a given level when all milestones at that level and below are demonstrated 50% or more of the time.

For procedural feedback, we developed assessments for 20 surgical procedures for Ob-Gyn and FPMRS trainees. Procedures included those tracked in ACGME case logs among others. Procedures were divided into approximately 10 key steps by 2 authors (A.M.C. and K.K.), based on textbook descriptions and the authors' surgical experience. Specialty and subspecialty experts (10 faculty members) provided input and developed consensus on the procedural steps. Teachers rated learners on each step using the 5-stage Dreyfus model of skill acquisition, with ratings ranging from “novice” to “expert,”¹⁵  and assessed overall surgical procedural ability (sample shown in figure 1a). Narrative comments were recorded by teachers and learners, and addressed what went well and what needed to be improved (figure 1b). Individual programs were allowed to add their own procedural assessments.

Participants included Ob-Gyn residents and faculty and fellow teachers, and FPMRS fellows and faculty teachers. Program directors registered residents and fellows and were encouraged to register all faculty, even if a teacher might only provide feedback on 1 occasion. The primary author met individually with program directors to train them on myTIPreport. Training addressed the importance of real-time feedback, use of rating scales, and technical instructions. Program directors used in-person workshops, didactics, and informational video instruction to educate residents, fellows, and faculty on the purpose and benefits of real-time feedback and the use of the myTIPreport.

We encouraged faculty, fellows, and residents to complete 10 milestone assessments and 5 procedural assessments per month, and encouraged residents and fellows to solicit workplace feedback. Teachers and learners were instructed to complete feedback using myTIPreport immediately after a surgical procedure or work-based encounter.

After programs used myTIPreport for 6 months, teachers and learners completed a 29-item survey (available from the corresponding author). The instrument was developed by the authors without assessment of validity evidence. The survey was similar to the preimplementation survey and asked about the status of feedback after implementation of myTIPreport.

Participating program directors were surveyed via e-mail regarding the implementation of, ease of use, and satisfaction with myTIPreport.

Participating institutions were granted exemption or approval status per respective Institutional Review Board decisions.

Statistical analysis was performed using SPSS version 19.0 (IBM Corp, Chicago, IL). Data are described with appropriate descriptive statistics. Preimplementation and postimplementation survey satisfaction data are presented with responses of extremely satisfied, very satisfied, and satisfied considered as satisfied, and somewhat satisfied and not satisfied considered as not satisfied.

Twelve Ob-Gyn residency and 7 FPMRS fellowship programs from 14 institutions participated (provided as online supplemental material). Of 980 registered users, 656 (67%) completed the preimplementation survey. This included 273 of 378 (72%) learners (254 Ob-Gyn and 19 FPMRS) and 383 of 602 (64%) teachers (355 Ob-Gyn and 28 FPMRS). The majority of participants responded “yes” to utilizing an individual process for workplace feedback (70%, 381 of 546 respondents), reporting the most common was informal discussion (79%, 340 of 430 respondents; provided as online supplemental material). When queried on satisfaction with feedback frequency, quality, and timeliness, Ob-Gyn residents reported satisfaction rates of 38%, 42%, and 37%, respectively (table 1). Ob-Gyn teacher satisfaction rates with these feedback domains were 50%, 54%, and 54%, respectively, while FPMRS teacher rates were 39%, 54%, and 39%, respectively (tables 1 and 2). When queried about particular characteristics of their processes for feedback, a minority of residents and teachers reported being satisfied that their processes helped them guide learning, helped with feedback interactions, or helped with giving and receiving feedback (table 3).

The web-based platform was successfully launched in all residency/fellowship programs. Necessary technical support was established and allowed for maintenance of secure, uninterrupted, consistent access during the study.

A total of 474 of 980 (48%) registered teachers and learners used myTIPreport. One-third (201 of 602) of registered teachers completed at least 1 feedback assessment, while more than two-thirds (273 of 378) of learners received feedback via the tool.

A total of 4311 feedback assessments were recorded. Feedback provided entailed assessments of procedural competence (n = 3367) nearly 3.5 times more often than milestone-based feedback (n = 944). Feedback was recorded for all surgical procedures and all milestone sets incorporated into the educational tool.

Twenty-four percent of learners (92 of 378) and 19% of teachers (115 of 602) completed the postsurvey. While satisfaction significantly improved for Ob-Gyn residents, Ob-Gyn teachers, and FPMRS teachers, and remained high for FPMRS fellows, response rates were low (tables 1 and 2).

Program directors at 74% (14 of 19) of programs completed the postimplementation surveys, with 79% (11 of 14) reporting learners were early adopters of the educational tool, and 86% (12 of 14) indicating implementation was successful. Program directors reported the following interventions were helpful with myTIPreport implementation: (1) orientating users; (2) tracking and publicly acknowledging high users; (3) providing incentives for use; and (4) ensuring leadership support. Challenges to myTIPreport use included time, culture change, and remembering login passwords.

Consistent with other reports,⁵  our preimplementation data indicated low satisfaction with feedback. Increased satisfaction with feedback frequency, quality, and timeliness was noted for Ob-Gyn residents, Ob-Gyn teachers, and FPMRS teachers in postimplementation surveys, though response rates were low.

A number of specialties have explored the potential benefits of web-based feedback and its documentation.^5,16–23  Real-time feedback via web-based platforms and mobile applications has been reported to facilitate feedback by properly and efficiently relaying constructive criticism to residents,^17,20  maintaining longitudinal records of resident progress based on milestones,²⁰  improving clinical practices such as handoffs or sign-outs,²⁰  tracking procedural skills checklists, and tracking other quality metrics.^17,20–23  The results of this study showed myTIPreport achieved some of these goals by facilitating and storing workplace feedback on milestones and procedural skills.

A systematic review reported that few published studies have examined feedback receptivity and feedback provision.²⁴  These aspects are directly addressed by the tool studied as it specifically lists actions to assess for both the milestones and procedural key steps and, in so doing, labels learner behaviors to be performed and assessed.

Limitations of this study include that the findings may not generalize beyond Ob-Gyn and FPMRS, and that we did not collect validity evidence for the survey tools. Additionally, we had low postimplementation survey response rates, which can potentially introduce response bias, as those in favor of the tool may have been more likely to respond.

Further work should examine longitudinal acquisition of procedural skills and milestone progress, examine barriers to use of the tool, address how myTIPreport will inform both faculty development efforts and Clinical Competency Committee deliberations, compare resident self-assessments to faculty assessments, and explore use in other ACGME-accredited specialties and subspecialties.

The myTIPreport formative assessment tool provides a method for Ob-Gyn and FPMRS milestones and procedures to be measured through real-time encounters between residents and fellows and their teachers. Learners and teachers appeared satisfied with how the implementation of this assessment tool affected feedback practices.