Background

Evidence suggests that simulation leads to improved operative skill, shorter operating room time, and better patient outcomes. Currently, no standardized laparoscopy curriculum exists for gynecology residents.

Objective

To design a structured laparoscopy curriculum for gynecology residents using Delphi consensus methodology.

Methods

This study began with Delphi methodology to determine expert consensus on the components of a gynecology laparoscopic skills curriculum. We generated a list of cognitive content, technical skills, and nontechnical skills for training in laparoscopic surgery, and asked 39 experts in gynecologic education to rate the items on a Likert scale (1–5) for inclusion in the curriculum. Consensus was predefined as Cronbach α of ≥ 0.80. We then conducted another Delphi survey with 9 experienced users of laparoscopic virtual reality simulators to delineate relevant curricular tasks. Finally, a cross-sectional design defined benchmark scores for all identified tasks, with 10 experienced gynecologic surgeons performing the identified tasks at basic, intermediate, and advanced levels.

Results

Consensus (Cronbach α = 0.85) was achieved in the first round of the curriculum Delphi, and after 2 rounds (Cronbach α = 0.80) in the virtual reality curriculum Delphi. Consensus was reached for cognitive, technical, and nontechnical skills as well as for 6 virtual reality tasks. Median time and economy of movement scores defined benchmarks for all tasks.

Conclusions

This study used Delphi consensus to develop a comprehensive curriculum for teaching gynecologic laparoscopy. The curriculum conforms to current educational standards of proficiency-based training, and is suggested as a standard in residency programs.

What was known and gap

Simulation improves operative skill, but no standardized laparoscopy curriculum exists for gynecology residents.

What is new

A structured laparoscopy curriculum was designed using experts and the Delphi consensus methodology.

Limitations

Potential for selection bias; lack of data to validate the curriculum.

Bottom line

The resulting curriculum conforms to current educational standards for proficiency-based training.

Editor's Note: The online version of this article contains a geographical representation of expert responses, a flow diagram detailing expert consensus, and a table of expert levels of proficiency for curricular tasks.

Laparoscopic surgery has become the standard of care for many gynecologic operations. Numerous obstacles exist for learning the principles and technical skills necessary for minimally invasive operations. This challenges the traditional preceptorship model of surgical education.16 

Simulation has been shown to be the most significant contributor to laparoscopic training of residents.7  Traditional box trainers and virtual reality (VR) simulators allow novice trainees to progress along the steep part of the learning curve before entering the operating room.1,4,8,9  Surgical bodies emphasize that training in laparoscopic surgery should take place in a simulated environment before the learner seeks experience in the operating room.1  The value of simulators may be enhanced through achievable proficiency benchmarks.3,4,1012 

Comprehensive curriculum development is lacking despite the established value of simulators in surgical education.2,3,7,9,13,14  Ideally, curriculum development should reflect practice patterns at diverse institutions,3  and encompass cognitive knowledge, technical skills, and nontechnical skills.3,1315  A North American surgical skills curriculum covering these areas was recently developed for surgical residents.16 

Our primary objective was to use Delphi methodology to support the development of a laparoscopy curriculum for gynecology residents. A secondary objective was to define proficiency benchmarks for the VR component of the curriculum.

Study Design

We used a modified Delphi method to achieve consensus on the components of a basic laparoscopic surgery curriculum for gynecology residents. The Delphi method—an iterative process in which responses are collated, analyzed, and presented anonymously back to participants—provides a means for obtaining opinions of experts in a systematic manner.15,17  The approach has been used to develop surgical curricula.3,18,19  An initial curriculum Delphi survey defined the curricular components (cognitive, technical, and nontechnical). This was followed by a VR Delphi survey that defined the technical VR component of the curriculum. Lastly, a cross-sectional design determined expert levels of proficiency for the defined VR curriculum.

Participant Selection for the Delphi Consensus

Sixty experts identified in a previous study20  volunteered to participate in the curriculum Delphi. Experts were either program directors or delegates from North American obstetrics and gynecology residency programs (54 US and 6 Canadian). Program directors were chosen because of their leadership role in academic obstetrics and gynecology, their institutional influence on surgical education, and their authority on departmental finances. The VR Delphi involved 10 experienced gynecologist users (9 US and 1 Canadian) of a laparoscopic VR trainer (LapSim, Surgical Science Inc). Membership in both expert panels was anonymous and concealed from participants.

Derivation of Survey Items

A traditional Delphi technique requires the participants to suggest survey items. Participants for the curriculum Delphi were supplied with a list of survey items based on literature review and local expert opinion.19,2124  This list was compiled within the context of a general obstetrics and gynecology residency program and included 12 questions (78 subheadings) pertaining to cognitive content, 10 questions (33 subheadings) pertaining to technical content, and 2 questions (13 subheadings) pertaining to nontechnical skills.

The LapSim 2013 edition was selected for the VR Delphi. LapSim has previously been evaluated for construct validity, learning curves, and skills transfer.4,8,10,2527  The role of the VR Delphi panel was to determine which of 13 basic and 4 gynecologic procedural tasks were relevant for junior gynecology residents.

Survey Administration

Online versions of the surveys were created using the web-based software SurveyMonkey. Prior to administration, 2 faculty members, 3 fellows, and 2 residents pretested the surveys. Individuals completed the survey and provided feedback to the authors to ensure question clarity. Participants were asked to rate the importance of including each item in the final curriculum using a Likert scale from 1 (strongly disagree) to 5 (strongly agree).

An e-mail was sent to volunteers who had agreed to participate in the curriculum Delphi.20  Five reminders were sent out at 1-week intervals. Data collection stopped 42 days after the original invitation.

The VR Delphi was distributed via e-mail. There were 2 iterations, with 2 reminders each. Data collection stopped after 35 days for the first iteration, and after 51 days for the second iteration.

Results of the surveys were returned to participants with group medians and interquartile ranges until expert consensus was reached. Expert consensus was predefined as a Cronbach α ≥ 0.80.3,28 

Final Structure of the Curriculum

After consensus was achieved, an outline of the final curriculum was created that included items that more than 80% of experts ranked as a 4 (agree) or 5 (strongly agree; curriculum Delphi and VR Delphi). Items were divided into cognitive component, technical skills component (box trainer and VR), and nontechnical skills component.

Final Structure of the VR Curriculum

The VR curriculum included the identified tasks from the VR Delphi at basic, intermediate, and advanced levels using the LapSim 2013 software. Difficulty increased progressively by involving increased repetitions, smaller objects, and increased blood vessel fragility.3,8 

Generation of Proficiency Levels for VR Curriculum

Ten experienced gynecologic laparoscopists were identified (5 Canadian and 5 Danish) in the authors' institutions to establish the benchmark levels. An experienced surgeon was defined as an individual who had completed more than 100 advanced minimally invasive procedures.3,4  Experts selected possessed advanced laparoscopic skills, were involved in laparoscopy training of residents, and had an interest in laparoscopy education.

A member of the study team (H.H. or F.B.) familiarized each surgeon with the simulator using instructional videos. Warm-up time on the simulator was not permitted. Each surgeon performed each task at each level in a predetermined sequence without assistance from members of the study team.

Benchmark scores were based on the automatic assessment parameters generated by the simulator. The parameters of interest were time and economy of motion (angular path length and path length), which are consistently measured on all LapSim tasks and demonstrate construct validity.3,8,10,24,26  Benchmark scores for time, path length, and angular path length were calculated for each curricular component at each level.

The hospital's Research Ethics Board approved the final version of the questionnaire and the dissemination plan.

Data Analysis

SPSS version 20 (IBM Corp) was used for data analysis. For the Delphi surveys, Cronbach α, medians, and interquartile ranges were calculated and reported back to the expert panelists. Median expert scores determined proficiency levels for the identified VR curricular tasks.3 

Comprehensive Curriculum Delphi

Of the 60 experts contacted,20  39 (65%) participated in the curriculum Delphi (provided as online supplemental material). The first iteration data were analyzed in 3 ways to assess consensus (provided as online supplemental material). Regardless of the method of analysis, consensus was reached after the first round, and further rounds were not required. Items that were included in the final curriculum (table 1) were rated as 4 or 5 by at least 80% of experts from a cognitive (74 of 78), technical (16 of 33), and nontechnical (13 of 13) perspective. See the figure for the proposed curriculum.

TABLE 1

Elements Included in the Final Curriculum

Elements Included in the Final Curriculum
Elements Included in the Final Curriculum
FIGURE

Flow Diagram Depicting the Proposed Curriculum

The proposed curriculum was developed by incorporating the elements from table 1 that met criteria for inclusion based on the Delphi consensus.

FIGURE

Flow Diagram Depicting the Proposed Curriculum

The proposed curriculum was developed by incorporating the elements from table 1 that met criteria for inclusion based on the Delphi consensus.

Close modal

VR Delphi

Few VR tasks met the criteria for inclusion in the curriculum Delphi. This obviated the requirement for a VR Delphi. Nine of 10 experts (8 US, 1 Canadian) participated. After the first iteration, Cronbach α was 0.52, indicating a lack of consensus. The 9 experts achieved consensus with the second round of the survey (Cronbach α = 0.80). LapSim tasks that at least 80% of experts rated as 4 or 5 were included in the final curriculum (table 2).

TABLE 2

LapSim Tasks Included in the Final Virtual Reality Curriculum

LapSim Tasks Included in the Final Virtual Reality Curriculum
LapSim Tasks Included in the Final Virtual Reality Curriculum

Expert Proficiency Benchmarks

Ten experienced surgeons completed the VR curriculum (6 tasks at 3 levels of difficulty), 5 from each participating institution. Half of the surgeons were men; all surgeons were right-handed, with the exception of 1 who was ambidextrous. Median scores were set as benchmarks for each task (provided as online supplemental material) because the data did not follow a normal distribution.

This study used modified Delphi consensus methodology to establish the content of a laparoscopy curriculum for gynecology residents, conforming to current standards of proficiency-based training.11  The curriculum includes cognitive, technical, and nontechnical skills elements, aligning with Zevin's principles of comprehensive curricula design.3,6,7,1315  North American obstetrics and gynecology programs lack a standardized laparoscopy curriculum for residents, despite multiple publications describing its necessity.7,9,19,21,22  The curricula in most programs may lack aspects of comprehensive curriculum design because they were developed at a local level.3,13,15  Comprehensive national curricula designed for junior gynecology residents were described in Denmark and the United Kingdom,9,18  yet neither included a nontechnical skills component. Previous work has demonstrated that basic task training using VR simulators translates into improved performance in the operating room.3,29 

In addition, the use of simulators for task training is more effective if the proficiency benchmarks are based on expert-derived performance goals for the chosen tasks on the available simulator.3,4,7,8 10,11,30  In the present study, benchmarks were formulated based on the simulator-generated scores using time and economy of motion, which are previously validated parameters.3,4,8,25 

Our efforts resulted in an international consensus statement on the elements of a structured laparoscopy curriculum for gynecology residents. Thirty-nine program directors or delegates (representing 15% of North American obstetrics and gynecology programs) participated, which enhanced the generalizability of the findings. The Delphi methodology offers the opportunity to recruit experts from a wide geographical distribution without assembling them in a central location,31  thereby decreasing costs and increasing feasibility.18,31  The anonymous nature of the Delphi methodology results in all respondents' opinions being weighted equally, thus preventing 1 person's opinions from swaying the group.31  The ability to achieve a Cronbach α > 0.80 in the first round reflects the inclusion of items that are well accepted by laparoscopic gynecologic educators.

The findings of this study and the elements selected for inclusion align with a recent UK study18  that determined a basic laparoscopic gynecology curriculum. Our contribution is unique in that it identifies a nontechnical skills component in addition to technical skills and cognitive components, similar to the national skills curriculum for surgical residents.

Our study has some limitations. A Delphi consensus relies on the inclusion of expert volunteers with knowledge and interest in a particular area,17  which may introduce selection bias. In addition, not all volunteers completed the curriculum Delphi; this may have been partly due to the long list of potential items. Regional differences in access to simulation technologies (ie, VR simulators) may have influenced the acceptance or rejection of an item. Finally, our proficiency scores may have been influenced by prior differences in participants' exposure to the VR simulator, resulting in higher scores that may be difficult for a novice laparoscopist to achieve.

We plan to use the information from this study to determine if a structured laparoscopy curriculum that includes the cognitive, technical skills, and nontechnical skills selected will translate to improved operating room performance by residents.

This study utilized Delphi methodology to reach consensus on the elements (cognitive, technical, and nontechnical skills) to include in a comprehensive laparoscopy curriculum for junior gynecology residents. Experienced gynecologic surgeons established benchmarks for the identified curricular VR tasks at basic, intermediate, and advanced levels. In addition, standardized curricula for selection, in-training, and certification of surgeons are lacking, and the proficiency benchmarks we identified may contribute to these applications. The curriculum conforms to current educational standards of proficiency-based training and can be offered as a standard in residency programs.

1
Aggarwal
R
,
Moorthy
K
,
Darzi
A.
Laparoscopic skills training and assessment
.
Br J Surg
.
2004
;
91
(
12
):
1549
1558
.
2
Goff
BA.
Changing the paradigm in surgical education
.
Obstet Gynecol
.
2008
;
112
(
2, pt 1
):
328
332
.
3
Palter
VN
,
Graafland
M
,
Schijven
MP
,
Grantcharov
TP.
Designing a proficiency-based, content validated virtual reality curriculum for laparoscopic colorectal surgery: a Delphi approach
.
Surgery
.
2012
;
151
(
3
):
391
397
.
4
Aggarwal
R
,
Grantcharov
TP
,
Eriksen
JR
,
Blirup
D
,
Kristiansen
VB
,
Funch-Jensen
P
,
et al
.
An evidence-based virtual reality training program for novice laparoscopic surgeons
.
Ann Surg
.
2006
;
244
(
2
):
310
314
.
5
Palter
VN.
Comprehensive training curricula for minimally invasive surgery
.
J Grad Med Educ
.
2011
;
3
(
3
):
293
298
.
6
Whitted
RW
,
Pietro
PA
,
Martin
G
,
Latchaw
G
,
Medina
C.
A retrospective study evaluating the impact of formal laparoscopic training on patient outcomes in a residency program
.
J Am Assoc Gynecol Laparosc
.
2003
;
10
(
4
):
484
488
.
7
Stefanidis
D
,
Heniford
BT.
The formula for a successful laparoscopic skills curriculum
.
Arch Surg
.
2009
;
144
(
1
):
77
82
.
8
van Dongen
KW
,
Ahlberg
G
,
Bonavina
L
,
Carter
FJ
,
Grantcharov
TP
,
Hyltander
A
,
et al
.
European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills
.
Surg Endosc
.
2011
;
25
(
1
):
166
171
.
9
Strandbygaard
J
,
Bjerrum
F
,
Maagaard
M
,
Rifbjerg Larsen
C
,
Ottesen
B
,
Sorensen
JL.
A structured four-step curriculum in basic laparoscopy: development and validation
.
Acta Obstet Gynecol Scand
.
2014
;
93
(
4
):
359
366
.
10
van Dongen
KW
,
Tournoij
E
,
van der Zee
DC
,
Schijven
MP
,
Broeders
IA.
Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish between novices and experts
?
Surg Endosc
.
2007
;
21
(
8
):
1413
1417
.
11
Korndorffer
JR
Jr
,
Dunne
JB
,
Sierra
R
,
Stefanidis
D
,
Touchard
CL
,
Scott
DJ.
Simulator training for laparoscopic suturing using performance goals translates to the operating room
.
J Am Coll Surg
.
2005
;
201
(
1
):
23
29
.
12
Strandbygaard
J
,
Bjerrum
F
,
Maagaard
M
,
Winkel
P
,
Larsen
CR
,
Ringsted
C
,
et al
.
Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial
.
Ann Surg
.
2013
;
257
(
5
):
839
844
.
13
Aggarwal
R
,
Grantcharov
TP
,
Darzi
A.
Framework for systematic training and assessment of technical skills
.
J Am Coll Surg
.
2007
;
204
(
4
):
697
705
.
14
Burden
C
,
Oestergaard
J
,
Larsen
CR.
Integration of laparoscopic virtual-reality simulation into gynaecology training
.
BJOG
.
2011
;
118
(
suppl 3
):
5
10
.
15
Zevin
B
,
Levy
JS
,
Satava
RM
,
Grantcharov
TP.
A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery
.
J Am Coll Surg
.
2012
;
215
(
4
):
580
586.e3
.
16
American College of Surgeons
,
Association of Program Directors in Surgery
. ACS/APDS Surgery Resident Skills Curriculum.
2010
. .
17
Graham
B
,
Regehr
G
,
Wright
JG.
Delphi as a method to establish consensus for diagnostic criteria
.
J Clin Epidemiol
.
2003
;
56
(
12
):
1150
1156
.
18
Burden
C
,
Fox
R
,
Lenguerrand
E
,
Hinshaw
K
,
Draycott
TJ
,
James
M.
Curriculum development for basic gynaecological laparoscopy with comparison of expert trainee opinions; prospective cross-sectional observational study
.
Eur J Obstet Gynecol Reprod Biol
.
2014
;
180
:
1
7
.
19
Singh
SS
,
Marcoux
V
,
Cheung
V
,
Martin
D
,
Ternamian
AM.
Core competencies for gynecologic endoscopy in residency training: a national consensus project
.
J Minim Inv Gynecol
.
2009
;
16
(
1
):
1
7
.
20
Shore
EM
,
Lefebvre
GG
,
Grantcharov
TP.
Gynecology resident laparoscopy training: present and future
.
Am J Obstet Gynecol
.
2015
;
212
(
3
):
298
301.e1
.
21
Morozov
V
,
Nezhat
C.
Proposal of a formal gynecologic endoscopy curriculum
.
J Minim Invasive Gynecol
.
2009
;
16
(
4
):
416
421
.
22
Brill
AI
,
Rogers
RM
Jr
.
A comprehensive program for resident training in operative laparoscopy
.
J Am Assoc Gynecol Laparosc
.
1998
;
5
(
3
):
223
228
.
23
Cundiff
GW.
Analysis of the effectiveness of an endoscopy education program in improving residents' laparoscopic skills
.
Obstet Gynecol
.
1997
;
90
(
5
):
854
859
.
24
Clevin
L
,
Grantcharov
TP.
Does box model training improve surgical dexterity and economy of movement during virtual reality laparoscopy? A randomised trial
.
Acta Obstet Gynecol Scan
.
2008
;
87
(
1
):
99
103
.
25
Larsen
CR
,
Grantcharov
T
,
Aggarwal
R
,
Tully
A
,
Sørensen
JL
,
Dalsgaard
T
,
et al
.
Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator
.
Surg Endosc
.
2006
;
20
(
9
):
1460
1466
.
26
Duffy
AJ
,
Hogle
NJ
,
McCarthy
H
,
Lew
JI
,
Egan
A
,
Christos
P
,
et al
.
Construct validity for the LAPSIM laparoscopic surgical simulator
.
Surg Endosc
.
2005
;
19
(
3
):
401
405
.
27
Eriksen
JR
,
Grantcharov
T.
Objective assessment of laparoscopic skills using a virtual reality stimulator
.
Surg Endosc
.
2005
;
19
(
9
):
1216
1219
.
28
Palter
VN
,
MacRae
HM
,
Grantcharov
TP.
Development of an objective evaluation tool to assess technical skill in laparoscopic colorectal surgery: a Delphi methodology
.
Am J Surg
.
2011
;
201
(
2
):
251
259
.
29
Larsen
CR
,
Oestergaard
J
,
Ottesen
BS
,
Soerensen
JL.
The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials
.
Acta Obstet Gynecol Scan
.
2012
;
91
(
9
):
1015
1028
.
30
Panait
L
,
Bell
RL
,
Roberts
KE
,
Duffy
AJ.
Designing and validating a customized virtual reality-based laparoscopic skills curriculum
.
J Surg Educ
.
2008
;
65
(
6
):
413
417
.
31
Zevin
B
,
Bonrath
EM
,
Aggarwal
R
,
Dedy
NJ
,
Ahmed
N
,
Grantcharov
TP
,
et al
.
Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery
.
J Am Coll Surg
.
2013
;
216
(
5
):
955
965
.

Author notes

Eliane M. Shore, MD, MSc, FRCSC, is Clinical Teacher, Department of Obstetrics and Gynaecology, St Michael’s Hospital, and Department of Obstetrics and Gynaecology, University of Toronto, Toronto, Ontario, Canada; Guylaine G. Lefebvre, MD, FRCSC, is Chair of Women’s Health, Department of Obstetrics and Gynaecology, St Michael’s Hospital, and Professor, Department of Obstetrics and Gynaecology, University of Toronto; Heinrich Husslein, MD, is Clinical Fellow, Department of Obstetrics and Gynaecology, St Michael’s Hospital, and University of Toronto, and Assistant Professor, Department of Obstetrics and Gynecology, Medical University Vienna, Vienna, Austria; Flemming Bjerrum, MD, is Resident Physician, Department of Gynecology, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark; Jette Led Sorensen, MD, MMEd, is Associate Professor and Head of Education, Department of Gynecology, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen; and Teodor P. Grantcharov, MD, PhD, FAC, is Associate Professor of Surgery, Department of Surgery, St Michael’s Hospital, and Department of Surgery, University of Toronto.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

This work was presented at the 69th Annual Clinical and Scientific Conference of the Society of Obstetricians and Gynaecologists of Canada, June 11–14, 2013, in Calgary, Alberta, Canada, and at the 42nd Global Congress on Minimally Invasive Gynecology of the American Association of Gynecologic Laparoscopists, November 10–14, 2013, in Washington, DC.

The authors would like to thank the expert panel in both Delphi consensus surveys and the expert gynecologic laparoscopists at St Michael's Hospital in Toronto, Ontario, Canada, and the Juliane Marie Centre for Children, Women and Reproduction at Rigshospitalet in Copenhagen, Denmark, who helped define the proficiency benchmarks.