Background

Whether cognitive and patient care skills attained during simulation-based mastery learning (SBML) are retained is largely unknown.

Objective

We studied retention of intensive care unit (ICU) clinical skills after an SBML boot camp experience.

Methods

Forty-seven postgraduate year (PGY)-1 residents completed SBML intervention designed to increase procedural, communication, and patient care skills. The intervention included ICU skills such as ventilator and hemodynamic parameter management. Residents were required to meet or exceed a minimum passing score (MPS) on a clinical skills examination before starting actual patient care. Skill retention was assessed in 42 residents who rotated in the medical ICU. Residents received a standardized 15-minute booster teaching session reviewing key concepts during the first week of the rotation. During the fourth week of their rotation, PGY-1 residents completed a clinical skills examination at the bedside of an actual ICU patient. Group mean examination scores and the proportion of subjects who met or exceeded the MPS at each testing occasion were compared.

Results

Residents scored a mean 90% (SD  =  6.5%) on the simulated skills examination immediately after training. Residents retained skills obtained through SBML as the mean score at bedside follow-up testing was 89% (SD  =  8.9%, P  =  .36). Thirty-seven of 42 (88%) PGY-1 residents met or exceeded the MPS at follow-up.

Conclusion

SBML leads to substantial retention of critical care knowledge, and patient care skills PGY-1 boot camp is a highly efficient and effective model that can be administered at the beginning of the academic year.

What was known

Learning of clinical skills by using simulation is popular although there is limited knowledge about skills retention after this type of learning intervention.

What is new

A study assessed retention of critical care knowledge and skills in internal medicine interns after simulation-based mastery learning.

Limitations

Single-site study, small sample, and assessment confined to a subset of critical care skills, all reducing generalizability.

Bottom line

A simulation-based boot camp resulted in substantial retention of critical care knowledge and patient care skills, and is an effective and efficient model for training first-year residents.

Patient care in the intensive care unit (ICU) is complex, and trainees are required to display skills across many competency domains. However, many medical schools do not require formal training in critical care despite the increasing acuity level of inpatient medicine.1 As a result, many postgraduate year (PGY)-1 internal medicine residents lack competency in critical care skills.2 Educators have also expressed concern that new duty hour restrictions limit clinical exposure to critically ill patients and represent an additional challenge to resident competence.3,4 

Simulation-based education improves trainees' knowledge and skills in areas such as invasive procedures,59 communication skills,10 and management of medical emergencies.1113 Simulation-based interventions in the ICU setting have also shown promise in promoting resident confidence14 and critical care skills.15 Information regarding skill transfer from the simulated setting to the bedside is limited. However, 1 study showed residents who received simulation-based education performed better on an ICU bedside clinical skills examination than residents who received traditional clinical training alone.16 

Several features promote skill acquisition and retention after simulation-based medical education, including simulation fidelity, outcome measurement, and curriculum integration.17 Deliberate practice of clinical skills with individualized performance feedback is critical for skill acquisition.17,18 Mastery learning is a rigorous form of competency-based education in which knowledge and skills are measured against high achievement standards.19 All learners must reach this standard, although the amount of time may vary among trainees. In simulation-based mastery learning (SBML), all trainees must meet or exceed the mastery standard in a simulated environment before performing the clinical skill in actual patient care.11 

Earlier work shows that skills are retained 12 to 14 months after simulation-based education for Advanced Cardiac Life Support20 and central venous catheter insertion.21 However, little is known about retention of cognitive skills, such as the advanced medical knowledge and patient care skills, needed to provide care for critically ill ICU patients. The aim of this study was to evaluate retention of ICU clinical skills 1 month to 12 months after an SBML intervention.

Study Design and Setting

We conducted a longitudinal cohort study of ICU clinical skill retention after SBML. The study was performed at Northwestern Memorial Hospital (NMH), a tertiary care university-affiliated teaching hospital in Chicago, IL, from July 2011 to June 2012.

The NMH medical ICU is a closed unit staffed by faculty members and fellows from the Division of Pulmonary and Critical Care Medicine at Northwestern University Feinberg School of Medicine. Faculty members and fellows supervise PGY-1 to PGY-3 internal medicine residents rotating through the medical ICU. Each PGY-1 internal medicine resident completes one 4-week rotation and is responsible for daily management of ICU patients.

Procedure

Forty-seven PGY-1 internal medicine residents completed an SBML boot camp intervention designed to increase clinical skills in June 2011.22 Boot camp consisted of 3 days of intensive training and assessment to mastery standards in a range of clinical skills, including (1) cardiac auscultation, (2) paracentesis, (3) lumbar puncture, (4) ICU clinical skills, and (5) code status discussion.22 

The Northwestern University Institutional Review Board approved the study, and trainees provided informed consent before participating.

The ICU clinical skills component of boot camp included didactic content, deliberate skills practice, and assessment to mastery standards. Deliberate practice has consistently shown to be a critical component of skill mastery in medicine and its related domains.18 PGY-1 residents rotated in small groups of 6 to 8 for teaching sessions. Each session contained identical content, including objectives, orientation, didactic material, skills practice, assessment, feedback, additional practice, and individualized assessment.

Didactic content included a review of concepts such as respiratory failure, circulatory shock, and ventilator management. Residents subsequently completed deliberate skills practice of ICU clinical scenarios with individualized feedback.22 Practice sessions took place in the NMH Patient Safety Simulator Center, using a ventilator-compatible, high-fidelity patient simulator displaying vital signs, physical examination findings, and respiratory and circulatory tracings (Medical Education Technologies Inc, Sarasota, FL).

Each practice session had 3 common clinical ICU scenarios. The first scenario involved management of a patient with acute hypoxemic respiratory failure and septic shock. The second scenario required residents to respond to a high-peak pressure ventilator alarm. The final scenario involved assessment of a patient who was clinically appropriate for a spontaneous breathing trial and extubation. A senior faculty member and former medical ICU director (T.C.C.) provided debriefing and feedback.

Assessment

All residents completed an ICU clinical skills examination on ventilator and hemodynamic parameter management after the teaching sessions. We scored each resident's clinical skill examination using a 20-item skills checklist regarding patient and ventilator parameters and readiness for a spontaneous breathing trial. The checklist was used in prior work in both simulated and ICU settings and yielded highly reliable data.16,23 Individual checklist items included using data from the ventilator to calculate airway resistance and static compliance, describing ventilator settings and circulatory parameters, and listing specific clinical features suggesting readiness for a spontaneous breathing trial.23 We expected residents to meet or exceed a minimum passing score (MPS) previously set at 80% by an expert panel using the Angoff (item-based) and Hofstee (group-based) standard setting methods.24 Eleven residents (23%) who did not initially achieve the MPS engaged in up to 1 hour of additional deliberate practice and were retested until the MPS was reached.22 

Skill retention was assessed in 42 PGY-1 internal medicine residents who completed SBML in June 2011 and subsequently rotated through the medical ICU from July 2011 to June 2012 (at least 1 month and up to 1 year after boot camp). During the first week of the rotation, residents received a standardized 15-minute booster teaching session that reviewed key concepts from the SBML ICU intervention. During the fourth week of their ICU rotation, PGY-1 residents completed a single clinical skills examination using the same 20-item checklist at the bedside of an actual ICU patient. A clinically stable patient receiving mechanical ventilation and central venous pressure monitoring was selected for the examination by the instructor.

Simulated skills examinations scores from boot camp (June 2011)22 were compared to bedside follow-up skills examination scores 1 month to 12 months later. A single, unblinded instructor (F.M.) graded each bedside examination. A second rater blind to the study design and aims also scored a random 50% sample of bedside follow-up skills examinations to assess interrater reliability.

Primary outcome measures consisted of performance of simulated skills examinations and ICU bedside follow-up examinations. Skill retention was assessed in 2 ways. First, we compared group mean examination scores at each testing occasion. Second, we compared the proportion of participants who met or exceeded the MPS at each examination.

Statistical Analysis

We estimated checklist score reliability by calculating interrater reliability using the Kappa (κ) coefficient across all checklist items. We compared mean scores at each examination using the paired t test. We compared the proportion of trainees who met or exceeded the MPS at each occasion (simulator versus bedside follow-up) using the chi-square statistic. We used regression analysis to test whether the number of months after SBML was associated with bedside checklist scores. All analyses were performed using SPSS version 20.0 software (IBM SPSS Inc, Chicago, IL).

All 42 PGY-1 internal medicine residents consented to participate and completed the entire protocol. Participants had a mean age of 27 years, almost half were women, and 98% were graduates of US medical schools. United States Medical Licensing Examination scores were a mean of 235.3 (SD  =  16.50) on step 1, and a mean of 251.45 (SD  =  11.22) on step 2.

Interrater reliability was extremely high across the 18 checklist items, with a κ coefficient of 0.99.

Residents in this study retained ICU clinical skills acquired during SBML. The residents scored a mean of 90% (SD  =  6.5%) on the simulated skills examination during boot camp22 and a mean of 89% (SD  =  8.9%) at bedside follow-up (P  =  .36; figure 1). Thirty-seven of 42 (88%) PGY-1 residents demonstrated retention of the material above the MPS.

Mean bedside checklist scores remained constant over time (figure 2). The number of months between boot camp and follow-up testing did not significantly predict bedside checklist scores (b  =  .001, P  =  .76).

This study demonstrates that critical care medical knowledge and patient care skills acquired during SBML, with a 15-minute booster session 3 weeks prior to testing, are retained for up to 1 year. We found no differences between mean checklist scores after boot camp and scores at the bedside of actual ICU patients. Also, PGY-1 residents in this study (who completed SBML during boot camp) scored higher on the skills checklist at the end of their ICU rotation than historical control residents who did not complete SBML.16 We conclude that residents acquired critical care knowledge and skills during SBML, retained these skills, and transferred them to the actual patient care environment up to 12 months later.

We believe the skill retention shown in our study is largely the result of the strength of the primary intervention.25 During boot camp, residents actively participated in small-group training sessions featuring ample time for deliberate practice and feedback. This model has been strongly linked to successful acquisition of scientific knowledge26 and clinical skills.17,18 Additionally, trainees were required to meet or exceed an MPS at immediate posttest. Those interns who were unable to meet the MPS (11%) returned to the simulation laboratory for additional skills practice before starting clinical training.22 Thus, all residents displayed high achievement levels with little variation after SBML. At least 1 other study showed a link between mastery learning and skill retention regarding invasive procedural skills.21 The current study extends what is known about the mastery model by demonstrating skill retention in patient care and medical knowledge skills required to care for critically ill patients.

Our study also demonstrates a model for the effective and efficient use of simulation-based education in graduate medical education. A recent meta-analysis showed that simulation-based education is superior to traditional training.27 Our findings suggest that simulation resources and faculty time can be concentrated at the beginning of the academic year to train residents in important clinical skills. A boot camp model, rather than coordinating faculty and resident schedules and simulation laboratory use throughout the year, produces efficient and cost-effective use of faculty and resources.

This study has several limitations. First, it was conducted at a single medical center using a relatively small group of participants. How residents at other institutions would respond after SBML is unknown. Second, because of practical limitations, the intervention and testing targeted a limited subset of critical care competencies. We believe they are a representative subset of the critical care competencies PGY-1 residents should possess at the end of an ICU rotation. Third, all PGY-1 residents received a short booster session at the start of their ICU rotation. This was done to reduce variation in patient care and instruction received during each rotation. Further information is needed about the impact of withholding the booster session on skill retention. It is also not known how PGY-1 residents would perform if tested at the beginning of their ICU rotation, as ICU clinical experiences might impact bedside checklist scores. Fourth, the primary examiner was not blind to the study protocol. In order to minimize examiner bias, the checklist was designed so each question had only 1 possible correct answer,16 and a second examiner, who had no knowledge of study aims, was present during 50% of the bedside examinations and judged near-equivalent scoring. Finally, we acknowledge that skill retention, while substantial, was not perfect. Five PGY-1 residents (12%) did not reach the MPS at follow-up testing. Further study is needed to develop educational models that ensure that all learners achieve and maintain the skills required to care for critically ill patients.

SBML during PGY-1 boot camp results in critical care knowledge and patient care skills that are retained over time. Simulation-based education featuring deliberate practice and high achievement standards is a highly efficient and effective model that can be administered at the beginning of the academic year for new residents. Further study is required to link SBML during PGY-1 boot camp to improved patient care outcomes.

1
Frankel
HL
,
Rogers
PL
,
Gandhi
RR
,
Freid
EB
,
Kirton
OC
,
Murray
MJ
;
Undergraduate Medical Education Committee of the Society of Critical Care Medicine. What is taught, what is tested: findings and competency-based recommendations of the Undergraduate Medical Education Committee of the Society of Critical Care Medicine
.
Crit Care Med
.
2004
;
32
(
9
):
1949
1956
.
2
Cox
CE
,
Carson
SS
,
Ely
EW
,
Govert
JA
,
Garrett
JM
,
Brower
RG
,
et al.
Effectiveness of medical resident education in mechanical ventilation
.
Am J Respir Crit Care Med
.
2003
;
167
(
1
):
32
38
.
3
Lin
GA
,
Beck
DC
,
Stewart
AL
,
Garbutt
JM
.
Resident perceptions of the impact of work hour limitations
.
J Gen Intern Med
.
2007
;
22
(
7
):
969
975
.
4
Peets
A
,
Ayas
NT
.
Restricting resident work hours: the good, the bad, and the ugly
.
Crit Care Med
.
2012
;
40
(
3
):
960
966
.
5
Blum
MG
,
Powers
TW
,
Sundaresan
S
.
Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy
.
Ann Thorac Surg
.
2004
;
78
(
1
):
287
291
.
6
Sedlack
RE
,
Kolars
JC
.
Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study
.
Am J Gastroenterol
.
2004
;
99
(
1
):
33
37
.
7
Fried
GM
,
Feldman
LS
,
Vassiliou
MC
,
Fraser
SA
,
Stanbridge
D
,
Ghitulescu
G
,
et al.
Proving the value of simulation in laparoscopic surgery
.
Ann Surg
.
2004
;
240
(
3
):
518
525
.
8
Gallagher
AG
,
Lederman
AB
,
McGlade
K
,
Satava
RM
,
Smith
CD
.
Discriminative validity of the minimally invasive surgical trainer in virtual reality (MIST-VR) using criteria levels based on expert performance
.
Surg Endosc
.
2004
;
18
(
4
):
660
665
.
9
Swanstrom
LL
,
Fried
GM
,
Hoffman
KI
,
Soper
NJ
.
Beta test of a new system assessing competence in laparoscopic surgery
.
J Am Coll Surg
.
2006
;
202
(
1
):
62
69
.
10
Szmuilowicz
E
,
Neely
KJ
,
Sharma
RK
,
Cohen
ER
,
McGaghie
WC
,
Wayne
DB
.
Improving residents' code status discussion skills: a randomized trial
.
J Palliat Med
.
2012
;
15
(
7
):
768
774
.
11
Wayne
DB
,
Butter
J
,
Siddall
VJ
,
Fudala
MJ
,
Wade
LD
,
Feinglass
J
,
et al.
Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice
.
J Gen Intern Med
.
2006
;
21
(
3
):
251
256
.
12
Draycott
TJ
,
Crofts
JF
,
Ash
JP
,
Wilson
LV
,
Yard
E
,
Sibanda
T
,
et al.
Improving neonatal outcome through practical shoulder dystocia training
.
Obstet Gynecol
.
2008
;
112
(
1
):
14
20
.
13
Rosenthal
ME
,
Adachi
M
,
Ribaudo
V
,
Mueck
JT
,
Schneider
RF
,
Mayo
PH
.
Achieving housestaff competence in emergency airway management using scenario based simulation training
.
Chest
.
2006
;
129
(
6
):
1453
1458
.
14
Lighthall
GK
,
Barr
J
,
Howard
SK
,
Gellar
E
,
Sowb
Y
,
Bertacini
E
,
et al.
Use of a fully simulated intensive care unit environment for critical event management training for internal medicine residents
.
Crit Care Med
.
2003
;
31
(
10
):
2437
2443
.
15
Musacchio
MJ
Jr,
Smith
AP
,
McNeal
CA
,
Munoz
L
,
Rothenberg
DM
,
von Roenn
KA
,
et al.
Neuro-critical care skills training using a human patient simulator
.
Neurocrit Care
.
2010
;
13
(
2
):
169
175
.
16
Schroedl
CJ
,
Corbridge
TC
,
Cohen
ER
,
Fakhran
SS
,
Schimmel
D
,
McGaghie
WC
,
et al.
Use of simulation-based education to improve resident learning and patient care in the medical intensive care unit: a randomized trial
.
J. Crit Care
.
2012
;
27
(
2
):
219,e7
e13
.
17
McGaghie
WC
,
Issenberg
SB
,
Petrusa
ER
,
Scalese
RJ
.
A critical review of simulation-based medical education (SBME) research: 2003–2009
.
Med Educ
.
2010
;
44
(
1
):
50
63
.
18
Ericsson
KA
.
Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains
.
Acad Med
.
2004
;
79
(
suppl 10
):
70
81
.
19
McGaghie
WC
,
Miller
GE
,
Sajid
A
,
Telder
TV
.
Competency-based curriculum development in medical education: An introduction. Public Health Paper No. 68
.
Geneva, Switzerland
:
World Health Organization
;
1978
.
20
Wayne
DB
,
Siddall
VJ
,
Butter
J
,
Fudala
MJ
,
Wade
LD
,
Feinglass
J
,
et al.
A longitudinal study of internal medicine residents' retention of advanced cardiac life support skills
.
Acad Med
.
2006
;
81
(
suppl 10
):
9
12
.
21
Barsuk
JH
,
Cohen
ER
,
McGaghie
WC
,
Wayne
DB
.
Long-term retention of central venous catheter insertion skills after simulation-based mastery learning
.
Acad Med
.
2010
;
85
(
suppl 10
):
9
12
.
22
Cohen
ER
,
Barsuk
JH
,
Moazed
F
,
Caprio
T
,
Didwania
A
,
McGaghie
WC
,
et al.
Making July safer: simulation-based mastery learning during intern boot cAMP
.
Acad Med
.
2013
;
88
(
2
):
233
239
.
23
Singer
BD
,
Corbridge
TC
,
Schroedl
CJ
,
Wilcox
JE
,
Cohen
ER
,
McGaghie
WC
,
et al.
First-year residents outperform third-year residents after simulation-based education in critical care medicine
.
Simul Healthc
.
2013
;
8
(
2
):
67
71
.
24
Downing
SM
,
Tekian
A
,
Yudkowsky
R
.
Procedures for establishing defensible absolute passing scores on examinations in health professions education
.
Teach Learn Med
.
2006
;
18
(
1
):
50
57
.
25
Cordray
DS
,
Pion
GM
.
Treatment strength and integrity: models and methods
.
In:
Bootzin
RR
,
McKnight
PE
,
eds
.
Strengthening Research Methodology: Psychological Measurement and Evaluation.
.
Washington, DC
:
American Psychological Association
;
2005
:
103
124
.
26
Deslauriers
L
,
Schelew
E
,
Wieman
C
.
Improved learning in a large-enrollment physics class
.
Science
.
2011
;
332
(
6031
):
862
864
.
27
Cook
DA
,
Hatala
R
,
Brydges
R
,
Zendejas
B
,
Szostek
JH
,
Wang
AT
,
et al.
Technology-enhanced simulation for health professions education: a systematic review and meta-analysis
.
JAMA
.
2011
;
306
(
9
):
978
988
.

Author notes

All authors are with the Northwestern University Feinberg School of Medicine. Farzad Moazed, MD, is Chief Medical Resident, Department of Medicine; Elaine R. Cohen, MEd, is Research Coordinator, Department of Medicine; Nicholas Furiasse, MD, is Chief Medical Resident, Department of Medicine; Benjamin Singer, MD, is Chief Medical Resident, Department of Medicine; Thomas C. Corbridge, MD, is Professor of Medicine, Department of Medicine; William C. McGaghie, PhD, is Professor of Medical Education, Center for Education in Medicine; and Diane B. Wayne, MD, is Professor and Vice-Chair, Department of Medicine.

Funding: This study was funded by Northwestern Memorial Hospital and the Excellence in Academic Medicine Act supported by the Illinois Department of Healthcare and Family Services. Dr. McGaghie's contribution was supported in part by the Jacob R. Suker, MD, professorship in medical education and by grant UL 1 RR 025741 from the National Center for Research Resources, National Institutes of Health. These funding sources had no role in study design, data collection, analysis and interpretation of data, writing of the manuscript, or the decision to submit the manuscript for publication.

Presented as a poster at the American Thoracic Society International Conference in San Francisco, CA, May 2012.

The authors thank the Northwestern Internal Medicine residents for their dedication to education and patient care. We acknowledge Douglas E. Vaughan, MD, for support and encouragement of this work.