Background 

Few tools currently exist for effective, accessible delivery of real-time, workplace feedback in the clinical setting.

Objective 

We developed and implemented a real-time, web-based tool for performance-based feedback in the clinical environment.

Methods 

The tool (myTIPreport) was designed for performance-based feedback to learners on the Accreditation Council for Graduate Medical Education (ACGME) Milestones and procedural skills. “TIP” stands for “Training for Independent Practice.” We implemented myTIPreport in obstetrics and gynecology (Ob-Gyn) and female pelvic medicine and reconstructive surgery (FPMRS) programs between November 2014 and May 2015. Residents, fellows, teachers, and program directors completed preimplementation and postimplementation surveys on their perceptions of feedback.

Results 

Preimplementation surveys were completed by 656 participants of a total of 980 learners and teachers in 19 programs (12 Ob-Gyn and 7 FPMRS). This represented 72% (273 of 378) of learners and 64% (383 of 602) of teachers. Seventy percent of participants (381 of 546) reported having their own individual processes for real-time feedback; the majority (79%, 340 of 430) described these processes as informal discussions. Over 6 months, one-third of teachers and two-thirds of learners used the myTIPreport tool a total of 4311 times. Milestone feedback was recorded 944 times, and procedural feedback was recorded 3367 times. Feedback addressed all ACGME Milestones and procedures programmed into myTIPreport. Most program directors reported that tool implementation was successful.

Conclusions 

The majority of learners successfully received workplace feedback using myTIPreport. This web-based tool, incorporating procedures and ACGME Milestones, may be an important transition from other feedback formats.

Direct observations of clinical performance and the resulting feedback are important for residents and fellows to successfully progress to independent practice.14  Challenges to feedback include duty hour limitations, reduced faculty contact with learners, increased pressures for clinical efficiencies, faculty reluctance to provide negative feedback, and faculty perceiving they lack skills for giving feedback.510  Clinical teachers may not have access to learners' clinical progress to date, and there is a lack of a centralized, accessible repository for formative feedback. We designed myTIPreport to address these multiple feedback barriers. “TIP” stands for “Training for Independent Practice,” and the tool was developed for workplace interactive feedback on both the Accreditation Council for Graduate Medical Education (ACGME) Milestones and procedural skills. Real-time workplace feedback was defined as feedback in the clinical setting immediately or shortly after direct observation of a learner's clinical performance.

The aim was to gather validity evidence for myTIPreport by applying the framework outlined in the Standards of Educational and Psychological Testing.11,12  To do so, the tool was designed and implemented at multiple, geographically diverse, US institutions over 6 months. Tool development was supported by the Foundation for Excellence in Women's Health, a nonprofit medical education organization.

A convenience sample of 19 geographically diverse, ACGME-accredited obstetrics and gynecology (Ob-Gyn) and female pelvic medicine and reconstructive surgery (FPMRS) training programs agreed to pilot myTIPreport between November 2014 and May 2015.

Preimplementation Survey

Participating teachers and learners were e-mailed a 21-item preimplementation survey and consent form via Qualtrics software (Qualtrics LLC, Provo, UT) prior to using the educational tool. The survey was developed by the primary author without further testing (available from the corresponding author). Teachers and learners were queried on aspects of real-time feedback utilized in their programs, and asked to rate their satisfaction with current feedback frequency, quality, and timeliness on a 5-point Likert scale (1, extremely satisfied, to 5, not at all satisfied).

Development of the myTIPreport Tool

Software engineers designed the web-based myTIPreport platform with a secure, user-specific login and password. It allowed users to perform a variety of feedback functions, including generating new formative feedback and review of past formative and summative feedback. myTIPreport included the ACGME Milestones and procedures tracked through ACGME case logs. The tool was built to allow teachers and learners to view current and previously assessed procedural feedback and milestone subcompetency levels assigned to each learner. It overtly labels learner behaviors to be performed and assessed to build response process validity by labeling clearly for teachers and learners how learners should be rated and by providing checklists. myTIPreport uses direct data entry to ensure integrity of data stored and its interpretation.

Milestone Feedback

All ACGME Milestone sets for Ob-Gyn residency (n = 28) and FPMRS fellowship (n = 23) were incorporated into myTIPreport, with ACGME approval for use of the copyrighted milestones.13,14  For each level-specific milestone, teachers recorded responses of yes, observed, or not applicable. As the learner and teacher completed the feedback checklist, they were blinded to milestone level to reduce assessment bias. Narrative comments could be recorded by teachers and learners. After feedback is completed, myTIPreport then assigns a milestone level. The learner achieves a given level when all milestones at that level and below are demonstrated 50% or more of the time.

Procedural Feedback

For procedural feedback, we developed assessments for 20 surgical procedures for Ob-Gyn and FPMRS trainees. Procedures included those tracked in ACGME case logs among others. Procedures were divided into approximately 10 key steps by 2 authors (A.M.C. and K.K.), based on textbook descriptions and the authors' surgical experience. Specialty and subspecialty experts (10 faculty members) provided input and developed consensus on the procedural steps. Teachers rated learners on each step using the 5-stage Dreyfus model of skill acquisition, with ratings ranging from “novice” to “expert,”15  and assessed overall surgical procedural ability (sample shown in figure 1a). Narrative comments were recorded by teachers and learners, and addressed what went well and what needed to be improved (figure 1b). Individual programs were allowed to add their own procedural assessments.

Implementation and Use

Participants included Ob-Gyn residents and faculty and fellow teachers, and FPMRS fellows and faculty teachers. Program directors registered residents and fellows and were encouraged to register all faculty, even if a teacher might only provide feedback on 1 occasion. The primary author met individually with program directors to train them on myTIPreport. Training addressed the importance of real-time feedback, use of rating scales, and technical instructions. Program directors used in-person workshops, didactics, and informational video instruction to educate residents, fellows, and faculty on the purpose and benefits of real-time feedback and the use of the myTIPreport.

We encouraged faculty, fellows, and residents to complete 10 milestone assessments and 5 procedural assessments per month, and encouraged residents and fellows to solicit workplace feedback. Teachers and learners were instructed to complete feedback using myTIPreport immediately after a surgical procedure or work-based encounter.

Postimplementation Survey

After programs used myTIPreport for 6 months, teachers and learners completed a 29-item survey (available from the corresponding author). The instrument was developed by the authors without assessment of validity evidence. The survey was similar to the preimplementation survey and asked about the status of feedback after implementation of myTIPreport.

Program Director Perspectives

Participating program directors were surveyed via e-mail regarding the implementation of, ease of use, and satisfaction with myTIPreport.

Participating institutions were granted exemption or approval status per respective Institutional Review Board decisions.

Data Analysis

Statistical analysis was performed using SPSS version 19.0 (IBM Corp, Chicago, IL). Data are described with appropriate descriptive statistics. Preimplementation and postimplementation survey satisfaction data are presented with responses of extremely satisfied, very satisfied, and satisfied considered as satisfied, and somewhat satisfied and not satisfied considered as not satisfied.

Preimplementation Survey

Twelve Ob-Gyn residency and 7 FPMRS fellowship programs from 14 institutions participated (provided as online supplemental material). Of 980 registered users, 656 (67%) completed the preimplementation survey. This included 273 of 378 (72%) learners (254 Ob-Gyn and 19 FPMRS) and 383 of 602 (64%) teachers (355 Ob-Gyn and 28 FPMRS). The majority of participants responded “yes” to utilizing an individual process for workplace feedback (70%, 381 of 546 respondents), reporting the most common was informal discussion (79%, 340 of 430 respondents; provided as online supplemental material). When queried on satisfaction with feedback frequency, quality, and timeliness, Ob-Gyn residents reported satisfaction rates of 38%, 42%, and 37%, respectively (table 1). Ob-Gyn teacher satisfaction rates with these feedback domains were 50%, 54%, and 54%, respectively, while FPMRS teacher rates were 39%, 54%, and 39%, respectively (tables 1 and 2). When queried about particular characteristics of their processes for feedback, a minority of residents and teachers reported being satisfied that their processes helped them guide learning, helped with feedback interactions, or helped with giving and receiving feedback (table 3).

Implementation and Feasibility of myTIPreport

The web-based platform was successfully launched in all residency/fellowship programs. Necessary technical support was established and allowed for maintenance of secure, uninterrupted, consistent access during the study.

A total of 474 of 980 (48%) registered teachers and learners used myTIPreport. One-third (201 of 602) of registered teachers completed at least 1 feedback assessment, while more than two-thirds (273 of 378) of learners received feedback via the tool.

A total of 4311 feedback assessments were recorded. Feedback provided entailed assessments of procedural competence (n = 3367) nearly 3.5 times more often than milestone-based feedback (n = 944). Feedback was recorded for all surgical procedures and all milestone sets incorporated into the educational tool.

Postimplementation Survey

Twenty-four percent of learners (92 of 378) and 19% of teachers (115 of 602) completed the postsurvey. While satisfaction significantly improved for Ob-Gyn residents, Ob-Gyn teachers, and FPMRS teachers, and remained high for FPMRS fellows, response rates were low (tables 1 and 2).

Program Director Perspectives

Program directors at 74% (14 of 19) of programs completed the postimplementation surveys, with 79% (11 of 14) reporting learners were early adopters of the educational tool, and 86% (12 of 14) indicating implementation was successful. Program directors reported the following interventions were helpful with myTIPreport implementation: (1) orientating users; (2) tracking and publicly acknowledging high users; (3) providing incentives for use; and (4) ensuring leadership support. Challenges to myTIPreport use included time, culture change, and remembering login passwords.

Consistent with other reports,5  our preimplementation data indicated low satisfaction with feedback. Increased satisfaction with feedback frequency, quality, and timeliness was noted for Ob-Gyn residents, Ob-Gyn teachers, and FPMRS teachers in postimplementation surveys, though response rates were low.

A number of specialties have explored the potential benefits of web-based feedback and its documentation.5,1623  Real-time feedback via web-based platforms and mobile applications has been reported to facilitate feedback by properly and efficiently relaying constructive criticism to residents,17,20  maintaining longitudinal records of resident progress based on milestones,20  improving clinical practices such as handoffs or sign-outs,20  tracking procedural skills checklists, and tracking other quality metrics.17,2023  The results of this study showed myTIPreport achieved some of these goals by facilitating and storing workplace feedback on milestones and procedural skills.

A systematic review reported that few published studies have examined feedback receptivity and feedback provision.24  These aspects are directly addressed by the tool studied as it specifically lists actions to assess for both the milestones and procedural key steps and, in so doing, labels learner behaviors to be performed and assessed.

Limitations of this study include that the findings may not generalize beyond Ob-Gyn and FPMRS, and that we did not collect validity evidence for the survey tools. Additionally, we had low postimplementation survey response rates, which can potentially introduce response bias, as those in favor of the tool may have been more likely to respond.

Further work should examine longitudinal acquisition of procedural skills and milestone progress, examine barriers to use of the tool, address how myTIPreport will inform both faculty development efforts and Clinical Competency Committee deliberations, compare resident self-assessments to faculty assessments, and explore use in other ACGME-accredited specialties and subspecialties.

The myTIPreport formative assessment tool provides a method for Ob-Gyn and FPMRS milestones and procedures to be measured through real-time encounters between residents and fellows and their teachers. Learners and teachers appeared satisfied with how the implementation of this assessment tool affected feedback practices.

1
Ende
J.
Feedback in clinical medical education
.
JAMA
.
1983
;
250
(
6
):
777
781
.
2
Holmboe
ES.
Realizing the promise of competency-based medical education
.
Acad Med
.
2015
;
90
(
4
):
411
413
.
3
Ericsson
KA.
Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains
.
Acad Med
.
2004
;
79
(
suppl 10
):
70
81
.
4
Ericsson
KA.
An expert-performance perspective of research on medical expertise: the study of clinical performance
.
Med Educ
.
2007
;
41
(
12
):
1124
1130
.
5
Connolly
A,
Hansen
D,
Schuler
K,
et al.
Immediate surgical skills feedback in the operating room using “SurF” cards
.
J Grad Med Educ
.
2014
;
6
(
4
):
774
678
.
6
Bowen
JL,
Irby
DM.
Assessing quality and costs of education in the ambulatory setting: a review of the literature
.
Acad Med
.
2002
;
77
(
7
):
621
680
.
7
Veloski
J,
Boex
JR,
Grasberger
MJ,
et al.
Systematic review of the literature on assessment, feedback and physicians' clinical performance: BEME Guide No. 7
.
Med Teach
.
2006
;
28
(
2
):
117
128
.
8
Bosse
HM,
Mohr
J,
Buss
B,
et al.
The benefit of repetitive skills training and frequency of expert feedback in the early acquisition of procedural skills
.
BMC Med Educ
.
2015
;
15
:
22
.
9
McQueen
SA,
Petrisor
B,
Bhandari
M,
et al.
Examining the barriers to meaningful assessment and feedback in medical training
.
Am J Surg
.
2016
;
211
(
2
):
464
475
.
10
van de Ridder
JM,
Stokking
KM,
McGaghie
WC,
et al.
What is feedback in clinical education?
Med Educ
.
2008
;
42
(
2
):
189
197
.
11
American Educational Research Association, American Psychological Association, National Council on Measurement in Education
.
Standards for Educational and Psychological Testing
.
Washington, DC
:
American Educational Research Association;
1999
.
12
Downing
SM.
Validity: on the meaningful interpretation of assessment data
.
Med Educ
.
2003
;
37
(
9
):
830
837
.
13
Accreditation Council for Graduate Medical Education, American Board of Obstetrics and Gynecology, American College of Obstetrics and Gynecology
.
The Obstetrics and Gynecology Milestone Project
.
September
2015
. .
14
Accreditation Council for Graduate Medical Education, American Board of Obstetrics and Gynecology, American College of Obstetrics and Gynecology
.
The Female Pelvic Medicine and Reconstructive Surgery Milestone Project
.
July
2015
. .
15
Dreyfus
HL,
Dreyfus
SE.
Mind Over Machine
.
New York, NY
:
Free Press;
1986
.
16
Cooney
CM,
Cooney
DS,
Bello
RJ,
et al.
Comprehensive observations of resident evolution: a novel method for assessing procedure-based residency training
.
Plast Reconstr Surg
.
2016
;
137
(
2
):
673
678
.
17
Reynolds
K,
Barnhill
D,
Sias
J,
et al.
Use of the QR reader to provide real-time evaluation of residents' skills following surgical procedures
.
J Grad Med Educ
.
2014
;
6
(
4
):
738
741
.
18
Mittal
V,
Krieger
E,
Lee
BC,
et al.
Pediatrics residents' perspectives on family-centered rounds: a qualitative study at 2 children's hospitals
.
J Grad Med Educ
.
2013
;
5
(
1
):
81
87
.
19
Topps
D,
Evans
RJ,
Thistlethwaite
JE,
et al.
The one minute mentor: a pilot study assessing medical students' and residents' professional behaviours through recordings of clinical preceptors' immediate feedback
.
Educ Health (Abingdon)
.
2009
;
22
(
1
):
189
.
20
Moore
DL,
Ding
L,
Sadhasivam
S.
Novel real-time feedback and integrated simulation model for teaching and evaluating ultrasound guided regional anesthesia skills in pediatric anesthesia trainees
.
Paediatr Anaesth
.
2012
;
22
(
9
):
847
853
.
21
Yarris
LM,
Jones
D,
Kornegay
JG,
et al.
The milestones passport: a learner-centered application of the milestone framework to prompt real-time feedback in the emergency department
.
J Grad Med Educ
.
2014
;
6
(
3
):
555
560
.
22
Martin
SK,
Farnan
JM,
McConville
JF,
et al.
Piloting a structured practice audit to assess ACGME Milestones in written handoff communication in internal medicine
.
J Grad Med Educ
.
2015
;
7
(
2
):
238
241
.
23
Wagner
JP,
Chen
DC,
Donahue
TR,
et al.
Assessment of resident operative performance using a real-time mobile web system: preparing for the milestone age
.
J Surg Educ
.
2014
;
71
(
6
):
41
46
.
24
van de Ridder
JM,
McGaghie
WC,
Stokking
KM,
et al.
Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review
.
Med Educ
.
2015
;
49
(
7
):
658
673
.
What was known and gap

There is a dearth of tools for workplace-based feedback for residents and fellows.

What is new

myTIPreport is an online tool that provides milestone- and procedures-based feedback to obstetrics and gynecology residents and female pelvic medicine and reconstructive surgery fellows.

Limitations

Implementation surveys without validity evidence and sampling bias may reduce generalizability.

Bottom line

The majority of learners successfully received workplace feedback using myTIPreport over a 6-month period.

Funding: This research was fully funded by the Foundation for Excellence in Women's Health.

Conflict of interest: Dr Blanchard serves as a board member for the Foundation for Excellence in Women's Health. Dr Kenton reports grants and nonfinancial support from Boston Scientific Corp, outside the submitted work.

These results were presented as an oral presentation and poster at the CREOG/APGO Annual Meeting, New Orleans, Louisiana, March 2–5, 2016; and as an oral presentation and poster at AAGL 45th Global Congress, Orlando, Florida, November 14–18, 2016.

The authors would like to thank the following contributors to this work: Gabriella Gosman, MD; Abigail Litwiller, MD; Susan E. Gerber, MD; Gena Dunivan, MD; Thomas Gregory, MD; Kimberly Gesci, MD; Sylvia Botros, MD; Felicia Lane, MD; Robert Higgins, MD; Elizabeth Lutz, MD; Karen Adams, MD; Carol Major, MD; Gary N. Frishman, MD; Jessica Bienstock, MD; Leigh Cantrell, MD; Kristiina Parviainen, MD; Richard Uribe; Jacquia de la Cruz, MD; Stephanie Sullivan, MD; Sara Tinkham; and Nancy Chescheir, MD.

Editor's Note: The preimplementation and postimplementation survey tools and the program director survey tool are available on request from the corresponding author.

Supplementary data