Preoperative Practice Paired With Instructor Feedback May Not Improve Obstetrics-Gynecology Residents' Operative Performance

Although athletes and musicians commonly practice right before a game or performance, surgeons do not typically rehearse prior to surgery.

It has been well documented in the sports literature that practice before a game has the ability to enhance performance and reduce errors.^1,2  Practice or rehearsal also has a positive effect on cognitive skills³  and can reduce anxiety by increasing the perceived control of the situation by allowing the individual to anticipate and prepare for potential problems.^4,5 

Activities that require strenuous physical activity, strenuous mental activity, and the ability to perform both within required coordination and task performance restraints benefit from pre-performance practice.⁶  Despite the fact that laparoscopic surgery requires all of these, most surgeons typically do not practice prior to operating. However, recently, several studies^7–9  have shown a benefit of preoperative practice on trainees' surgical performance. This has the potential to improve educational experience.

Since this is a new area of study, the optimal method of preoperative practice is not clear. In surgical education, it is common for teachers to mentor or “coach” trainees through a task in order for them to learn or improve. There is evidence that trainees who receive instructor feedback learn more efficiently when performing a complex laparoscopic operational task in both simulated¹⁰  and real-life¹¹  activities. Therefore, preoperative practice with directed feedback from an expert surgeon may provide additional improvement in surgical technique compared to either approach used alone.

The primary aim of this study was to determine if preoperative practice by obstetrics-gynecology residents with instructor feedback (compared to either preoperative practice alone or feedback alone) improves the score on an objective structured assessment of laparoscopic salpingectomy.

The study participants consisted of postgraduate year 2 to 6 trainees in the Department of Obstetrics and Gynecology at the University of Toronto, Ontario, Canada, completing rotations at Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada, an academic tertiary care hospital, between 2013 and 2014. Trainees participated on a voluntary basis, and informed consent was obtained.

The study used a randomized controlled study design with 3 arms (figure). All participants completed a demographic questionnaire on study entry and baseline surgical proficiency testing, which was performance of a laparoscopic salpingectomy on a virtual reality surgical simulator scored of 0% to 100%, previously evaluated for construct validity (LapSim, Surgical Science, Göteborg, Sweden).¹²  The baseline testing was completed at least 1 day prior to the study intervention and outcome assessment. Using baseline scores, participants were stratified into 2 groups of skill level, low performer (< 70%) or high performer (≥ 70%). Stratified block randomization was then used to randomize trainees to either preoperative practice with feedback (PPF), preoperative practice alone (PP), or feedback alone (F). Central randomization was performed using computer sequence generation.

Those randomized to the PP arm completed a preoperative practice, those in F were given instructor feedback (based on their performance on the baseline testing), and those in PPF completed a preoperative practice (same exercise as PP) and also received feedback based on the preoperative practice. Preoperative practice and/or instructor feedback for each group took place within 1 hour prior to the operation. Preoperative practice consisted of 15 minutes of preparation on the laparoscopic salpingectomy module on the validated virtual reality surgical simulator. Instructor feedback was standardized and given in an evidence-based fashion shown to optimize effectiveness.

We chose to use terminal feedback, in which feedback was given at the completion of the preoperative practice, since there is some evidence that it is more effective than concurrent feedback in simulation education.¹³  The same 2 instructors gave feedback in equal distributions between the 2 groups randomized to receive feedback. The instructors gave each participant 3 constructive recommendations based on their performance so that each participant received an equal amount of feedback.

Trainees then completed a laparoscopic salpingectomy in the operating room with the assistance of a staff surgeon who was blinded to trainee group designation. The study participants did not participate in any other surgical cases, nor did they use a simulator outside of the study on the same day they carried out the laparoscopic salpingectomy, to attempt to avoid any other “practice” apart from the study. The staff surgeon only held the camera so that the trainee performed the procedure independently, and gave advice intraoperatively only if there were patient safety concerns, in which case this was noted. All procedures were video recorded. Procedures were restricted to normal anatomy, no adhesions, a body mass index of less than 30, and no treatment of ruptured ectopic pregnancy, to ensure consistent levels of difficulty between cases.

Trainees were evaluated using the objective structured assessment of laparoscopic salpingectomy, which has been previously evaluated for construct validity.¹⁴  Two blinded independent observers assessed the recorded operations, and the average score was used (maximum score of 45). The primary outcome was the average score on the objective structured assessment of laparoscopic salpingectomy.

Ethics board approval was obtained from Sunnybrook Health Sciences Centre.

Sample size was calculated using SAS version 9.2 (SAS Institute Inc, Cary, NC), with a minimal relevant difference of 4 points and standard deviation of 2.4 (based on the previous study of the objective structured assessment of laparoscopic salpingectomy¹⁴). With α = 0.05 (2-sided) and a power of 80%, we calculated that the study required at least 18 trainees (6 in each arm).

Statistical analysis was carried out using SAS version 9.2. Baseline characteristics of the 3 groups were compared using Fisher's exact test. The primary outcome was compared between the 3 groups (PP versus F versus PPF) using the Kruskal-Wallis test. A P value less than .05 was used to indicate statistical significance between groups. A nonparametric regression analysis was used to compare the primary outcome (the average score on the assessment tool) between groups, controlling for participant baseline score on the simulator. The reliability of the structured assessment between the 2 observers was determined by calculating the intraclass correlation coefficient.

Eighteen trainees completed the study, 6 in each arm. Table 1 includes the baseline characteristics of the participants based on the self-reported demographic questionnaire and the baseline scores after completion of a laparoscopic salpingectomy on the virtual reality surgical simulator. There were no statistically significant differences between groups.

The median score on the assessment tool for laparoscopic salpingectomy in the PPF group was the highest (32.75; range, 15–36) compared to the PP group (14.5; range, 10–34) and the F group (21.25; range, 10.5–32). However, this difference was not statistically significant using the Kruskal-Wallis test (P = .18; table 2). The interrater correlation between the video reviewers was 0.87 (95% confidence interval 0.70–0.95) using the intraclass correlation coefficient.

Using a nonparametric regression analysis, we compared the primary outcome (average score on the assessment tool) between groups adjusting for baseline score, and the results were still not significant (P = .07).

This randomized controlled trial study of obstetrics-gynecology trainees failed to demonstrate a significant difference in trainee performance when comparing preoperative practice on a simulator compared to instructor feedback or both combined.

Although there is some evidence that a preoperative practice improves operative performance, to our knowledge no previous studies have compared preoperative practice with or without feedback. One systematic review,¹⁵  including 6 randomized controlled trials and a total of 87 participants across all studies, found that warming up before an operative procedure improves trainees' technical, cognitive, and psychomotor performance. Primary studies included residents and fellows in general surgery, obstetrics-gynecology, and urology. Out of 6 studies in the review, 5 found a significant improvement in laparoscopic performance after preoperative warming up. The optimal duration and timing of preoperative practice still needs to be elicited, as does the modality. A recent Cochrane review¹⁶  concluded that virtual reality training appears to decrease operating time and improve the operative performance of surgical trainees with limited laparoscopic experience compared to no training or with box-trainer training. It is important to determine if one type of training method is superior to another for the purposes of preoperative practice. We chose to use terminal feedback in our study, based on some evidence that demonstrates it is superior to concurrent feedback.¹³  It is possible that, coupled with preoperative practice, another type of feedback may be better.

Although we performed a sample size calculation using previously validated and published data, the higher standard deviation in our study indicated more variance among participants. Therefore, we may have required a larger sample size to confirm that there is no difference in trainee performance after preoperative practice with feedback, compared to either one alone.

Alternatively, a certain level of baseline performance may be required prior to demonstrating an effect of preoperative practice with feedback compared to either one alone. In our study, the range of scores was quite large (10 to 36), and 4 participants had to be given intraoperative feedback. It may have been worthwhile to first practice with study participants until they reached a certain benchmark prior to study entry, since the low performers may have biased the results, and their performance may not have been reflective of the preoperative intervention to which they were randomized.

Limitations of this study include its single specialty, single institution nature, limiting generalizability. Also, despite advance calculations, our study may have been underpowered to detect smaller, but meaningful, differences.

At a time when resident work hours are decreasing, it is important to elicit new techniques and optimal ways to use them to enhance trainee education. The fact that no difference in operative performance was seen in our study may be a function of the small sample of our study. Alternatively, it may suggest that any form of preoperative practice, rehearsal, or feedback is enough to enhance trainee education in the operating room. Further research is needed. It will also be important to determine if the improvements seen with preoperative practice actually translate to clinically relevant differences, such as improved patient morbidity, operative time, and cost.

This study demonstrates that preoperative practice combined with instructor feedback may not improve the operative performance of surgical trainees compared to either a preoperative practice or feedback alone. Further research with larger samples and use of different approaches to giving feedback is needed to substantiate and elaborate on these results.