Retention of Critical Care Skills After Simulation-Based Mastery Learning

Patient care in the intensive care unit (ICU) is complex, and trainees are required to display skills across many competency domains. However, many medical schools do not require formal training in critical care despite the increasing acuity level of inpatient medicine.1 As a result, many postgraduate year (PGY)-1 internal medicine residents lack competency in critical care skills.2 Educators have also expressed concern that new duty hour restrictions limit clinical exposure to critically ill patients and represent an additional challenge to resident competence.3,4 

Simulation-based education improves trainees' knowledge and skills in areas such as invasive procedures,5–9 communication skills,10 and management of medical emergencies.11–13 Simulation-based interventions in the ICU setting have also shown promise in promoting resident confidence14 and critical care skills.15 Information regarding skill transfer from the simulated setting to the bedside is limited. However, 1 study showed residents who received simulation-based education performed better on an ICU bedside clinical skills examination than residents who received traditional clinical training alone.16 

Several features promote skill acquisition and retention after simulation-based medical education, including simulation fidelity, outcome measurement, and curriculum integration.17 Deliberate practice of clinical skills with individualized performance feedback is critical for skill acquisition.17,18 Mastery learning is a rigorous form of competency-based education in which knowledge and skills are measured against high achievement standards.19 All learners must reach this standard, although the amount of time may vary among trainees. In simulation-based mastery learning (SBML), all trainees must meet or exceed the mastery standard in a simulated environment before performing the clinical skill in actual patient care.11 

Earlier work shows that skills are retained 12 to 14 months after simulation-based education for Advanced Cardiac Life Support20 and central venous catheter insertion.21 However, little is known about retention of cognitive skills, such as the advanced medical knowledge and patient care skills, needed to provide care for critically ill ICU patients. The aim of this study was to evaluate retention of ICU clinical skills 1 month to 12 months after an SBML intervention.

We conducted a longitudinal cohort study of ICU clinical skill retention after SBML. The study was performed at Northwestern Memorial Hospital (NMH), a tertiary care university-affiliated teaching hospital in Chicago, IL, from July 2011 to June 2012.

The NMH medical ICU is a closed unit staffed by faculty members and fellows from the Division of Pulmonary and Critical Care Medicine at Northwestern University Feinberg School of Medicine. Faculty members and fellows supervise PGY-1 to PGY-3 internal medicine residents rotating through the medical ICU. Each PGY-1 internal medicine resident completes one 4-week rotation and is responsible for daily management of ICU patients.

Forty-seven PGY-1 internal medicine residents completed an SBML boot camp intervention designed to increase clinical skills in June 2011.22 Boot camp consisted of 3 days of intensive training and assessment to mastery standards in a range of clinical skills, including (1) cardiac auscultation, (2) paracentesis, (3) lumbar puncture, (4) ICU clinical skills, and (5) code status discussion.22 

The Northwestern University Institutional Review Board approved the study, and trainees provided informed consent before participating.

The ICU clinical skills component of boot camp included didactic content, deliberate skills practice, and assessment to mastery standards. Deliberate practice has consistently shown to be a critical component of skill mastery in medicine and its related domains.18 PGY-1 residents rotated in small groups of 6 to 8 for teaching sessions. Each session contained identical content, including objectives, orientation, didactic material, skills practice, assessment, feedback, additional practice, and individualized assessment.

Didactic content included a review of concepts such as respiratory failure, circulatory shock, and ventilator management. Residents subsequently completed deliberate skills practice of ICU clinical scenarios with individualized feedback.22 Practice sessions took place in the NMH Patient Safety Simulator Center, using a ventilator-compatible, high-fidelity patient simulator displaying vital signs, physical examination findings, and respiratory and circulatory tracings (Medical Education Technologies Inc, Sarasota, FL).

Each practice session had 3 common clinical ICU scenarios. The first scenario involved management of a patient with acute hypoxemic respiratory failure and septic shock. The second scenario required residents to respond to a high-peak pressure ventilator alarm. The final scenario involved assessment of a patient who was clinically appropriate for a spontaneous breathing trial and extubation. A senior faculty member and former medical ICU director (T.C.C.) provided debriefing and feedback.

All residents completed an ICU clinical skills examination on ventilator and hemodynamic parameter management after the teaching sessions. We scored each resident's clinical skill examination using a 20-item skills checklist regarding patient and ventilator parameters and readiness for a spontaneous breathing trial. The checklist was used in prior work in both simulated and ICU settings and yielded highly reliable data.16,23 Individual checklist items included using data from the ventilator to calculate airway resistance and static compliance, describing ventilator settings and circulatory parameters, and listing specific clinical features suggesting readiness for a spontaneous breathing trial.23 We expected residents to meet or exceed a minimum passing score (MPS) previously set at 80% by an expert panel using the Angoff (item-based) and Hofstee (group-based) standard setting methods.24 Eleven residents (23%) who did not initially achieve the MPS engaged in up to 1 hour of additional deliberate practice and were retested until the MPS was reached.22 

Skill retention was assessed in 42 PGY-1 internal medicine residents who completed SBML in June 2011 and subsequently rotated through the medical ICU from July 2011 to June 2012 (at least 1 month and up to 1 year after boot camp). During the first week of the rotation, residents received a standardized 15-minute booster teaching session that reviewed key concepts from the SBML ICU intervention. During the fourth week of their ICU rotation, PGY-1 residents completed a single clinical skills examination using the same 20-item checklist at the bedside of an actual ICU patient. A clinically stable patient receiving mechanical ventilation and central venous pressure monitoring was selected for the examination by the instructor.

Simulated skills examinations scores from boot camp (June 2011)22 were compared to bedside follow-up skills examination scores 1 month to 12 months later. A single, unblinded instructor (F.M.) graded each bedside examination. A second rater blind to the study design and aims also scored a random 50% sample of bedside follow-up skills examinations to assess interrater reliability.

Primary outcome measures consisted of performance of simulated skills examinations and ICU bedside follow-up examinations. Skill retention was assessed in 2 ways. First, we compared group mean examination scores at each testing occasion. Second, we compared the proportion of participants who met or exceeded the MPS at each examination.

We estimated checklist score reliability by calculating interrater reliability using the Kappa (κ) coefficient across all checklist items. We compared mean scores at each examination using the paired t test. We compared the proportion of trainees who met or exceeded the MPS at each occasion (simulator versus bedside follow-up) using the chi-square statistic. We used regression analysis to test whether the number of months after SBML was associated with bedside checklist scores. All analyses were performed using SPSS version 20.0 software (IBM SPSS Inc, Chicago, IL).

All 42 PGY-1 internal medicine residents consented to participate and completed the entire protocol. Participants had a mean age of 27 years, almost half were women, and 98% were graduates of US medical schools. United States Medical Licensing Examination scores were a mean of 235.3 (SD  =  16.50) on step 1, and a mean of 251.45 (SD  =  11.22) on step 2.

Interrater reliability was extremely high across the 18 checklist items, with a κ coefficient of 0.99.

Residents in this study retained ICU clinical skills acquired during SBML. The residents scored a mean of 90% (SD  =  6.5%) on the simulated skills examination during boot camp22 and a mean of 89% (SD  =  8.9%) at bedside follow-up (P  =  .36; figure 1). Thirty-seven of 42 (88%) PGY-1 residents demonstrated retention of the material above the MPS.

Mean bedside checklist scores remained constant over time (figure 2). The number of months between boot camp and follow-up testing did not significantly predict bedside checklist scores (b  =  .001, P  =  .76).

This study demonstrates that critical care medical knowledge and patient care skills acquired during SBML, with a 15-minute booster session 3 weeks prior to testing, are retained for up to 1 year. We found no differences between mean checklist scores after boot camp and scores at the bedside of actual ICU patients. Also, PGY-1 residents in this study (who completed SBML during boot camp) scored higher on the skills checklist at the end of their ICU rotation than historical control residents who did not complete SBML.16 We conclude that residents acquired critical care knowledge and skills during SBML, retained these skills, and transferred them to the actual patient care environment up to 12 months later.

We believe the skill retention shown in our study is largely the result of the strength of the primary intervention.25 During boot camp, residents actively participated in small-group training sessions featuring ample time for deliberate practice and feedback. This model has been strongly linked to successful acquisition of scientific knowledge26 and clinical skills.17,18 Additionally, trainees were required to meet or exceed an MPS at immediate posttest. Those interns who were unable to meet the MPS (11%) returned to the simulation laboratory for additional skills practice before starting clinical training.22 Thus, all residents displayed high achievement levels with little variation after SBML. At least 1 other study showed a link between mastery learning and skill retention regarding invasive procedural skills.21 The current study extends what is known about the mastery model by demonstrating skill retention in patient care and medical knowledge skills required to care for critically ill patients.

Our study also demonstrates a model for the effective and efficient use of simulation-based education in graduate medical education. A recent meta-analysis showed that simulation-based education is superior to traditional training.27 Our findings suggest that simulation resources and faculty time can be concentrated at the beginning of the academic year to train residents in important clinical skills. A boot camp model, rather than coordinating faculty and resident schedules and simulation laboratory use throughout the year, produces efficient and cost-effective use of faculty and resources.

This study has several limitations. First, it was conducted at a single medical center using a relatively small group of participants. How residents at other institutions would respond after SBML is unknown. Second, because of practical limitations, the intervention and testing targeted a limited subset of critical care competencies. We believe they are a representative subset of the critical care competencies PGY-1 residents should possess at the end of an ICU rotation. Third, all PGY-1 residents received a short booster session at the start of their ICU rotation. This was done to reduce variation in patient care and instruction received during each rotation. Further information is needed about the impact of withholding the booster session on skill retention. It is also not known how PGY-1 residents would perform if tested at the beginning of their ICU rotation, as ICU clinical experiences might impact bedside checklist scores. Fourth, the primary examiner was not blind to the study protocol. In order to minimize examiner bias, the checklist was designed so each question had only 1 possible correct answer,16 and a second examiner, who had no knowledge of study aims, was present during 50% of the bedside examinations and judged near-equivalent scoring. Finally, we acknowledge that skill retention, while substantial, was not perfect. Five PGY-1 residents (12%) did not reach the MPS at follow-up testing. Further study is needed to develop educational models that ensure that all learners achieve and maintain the skills required to care for critically ill patients.

SBML during PGY-1 boot camp results in critical care knowledge and patient care skills that are retained over time. Simulation-based education featuring deliberate practice and high achievement standards is a highly efficient and effective model that can be administered at the beginning of the academic year for new residents. Further study is required to link SBML during PGY-1 boot camp to improved patient care outcomes.