Progress Toward Improving the Quality of Cardiac Arrest Medical Team Responses at an Academic Teaching Hospital

Adherence to advanced cardiac life support (ACLS) protocols in the inpatient setting is often poor, even when performed by well-trained personnel.1,2 Simulation-based medical education (SBME) has been shown to boost skill in diverse clinical competencies and procedures including ACLS.3–7 In 2003, we instituted SBME for ACLS training of postgraduate year 2 (PGY-2) internal medicine (IM) residents, featuring 8 hours of deliberate practice and feedback using a full-body human patient simulator.6,7 A previous case-control study using 2004 data demonstrated that PGY-2 simulator-trained residents were over 7 times more likely to perform an ACLS response adherent to American Heart Association (AHA) standards than traditionally trained (non-SBME) PGY-3 residents.8 Given these findings, the SBME program was continued each year for all IM PGY-2 residents at our institution.

A prior study showed that ACLS skills are retained by simulator-trained IM residents up to 14 months in the simulated environment.9 Other studies also have shown that SBME prevents skill decay in advanced trauma life support10,11 and airway management.12 These studies evaluated skills in a simulated clinical environment. We have not identified any studies that assessed SBME skill acquisition and retention in clinical practice.

This report presents data from a 2008 case-control study of the impact of simulation-based ACLS training for IM residents. Our study evaluated the quality of team responses to actual in-hospital ACLS events using previously published outcome measures.8 The aim was to compare ACLS responses of PGY-2 with those of PGY-3 residents to evaluate skill retention in actual clinical practice 12 to 18 months after simulation training.

This was a retrospective case-control study13 of cardiac arrest team responses at Northwestern Memorial Hospital (NMH) from January to June 2008. It replicates a similar investigation carried out at NMH in 2004.8 At our institution, all PGY-2 IM residents complete SBME in ACLS (PGY-1 and PGY-3 IM residents do not participate). As shown in figure 1, the 6-month study period was selected because it provided an opportunity to compare ACLS team responses from NMH residents in a natural quasi-experiment from 2 groups14: (a) PGY-2 IM residents who received ACLS simulation training (PGY-2 simulator-trained) within 6 months of the study period; and (b) PGY-3 IM residents who received ACLS simulation training 12 to 18 months earlier (PGY-3 simulator-trained). Cases and controls are team responses to in-hospital ACLS events sorted by PGY-2 resident leaders (group A, controls) and PGY-3 resident leaders who were simulation-trained greater than 1 year earlier (group B, cases).

ACLS team responses measured during the 2008 study period also were compared to team responses from a historical resident cohort measured in 2004.8 That cohort included IM PGY-2 residents who received an 8-hour SBME curriculum in ACLS (cases), and PGY-3 residents without ACLS simulation training (controls). Simulation training remained the same between 2004 and 2008 with the exception of updated AHA algorithms15 and training condensed to 6 hours. An identical study design was used with the 2004 cohort, which compared PGY-2 and PGY-3 resident skills.

The Northwestern University Institutional Review Board approved the study and waived informed consent.

NMH is a tertiary health care facility located in Chicago and is the principal setting for graduate medical education for Northwestern University. Cardiac arrest teams are led by an on-call second- or third-year IM resident for all cardiac arrest calls on all hospital floors with the exception of the Emergency Department and Surgical Intensive Care Unit. All residents complete an AHA ACLS provider course at the beginning of PGY-1 and again at the start of PGY-3.

PGY-2 residents receive a 6-hour SBME intervention.6,7 Simulation training occurs at the beginning of the academic year and includes participation in three 2-hour teaching sessions featuring deliberate practice. All sessions take place in the NMH Patient Safety Simulator Center. The program features the life-sized human patient simulator (HPS, Medical Education Technologies, Inc., Sarasota, FL), which realistically portrays many of the physiologic and pharmacologic responses observed in real ACLS situations. Features include responses of the respiratory system, pupils and eyelids, heart sounds, and peripheral pulses. Monitoring of arterial oxygen saturation, electrocardiogram, and blood pressure were used in the educational program.

Case studies were developed from current ACLS guidelines15 for the 6 most common ACLS scenarios encountered at NMH, including: asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia, and pulseless electrical activity. Evaluation checklists16 were developed and tested using uniform procedures to ensure quality control.17 Teaching sessions allowed groups of 2 to 4 residents time to engage in intense, deliberate practice of protocols and procedures and to receive standardized education and feedback from simulator faculty. Debriefing allowed the residents to ask questions, review algorithms, and receive feedback. The three 2-hour teaching sessions were presented in uniform order as follows: (a) pulseless arrhythmias (asystole, ventricular fibrillation, pulseless electrical activity); (b) tachycardias (supraventricular and ventricular); and (c) bradycardias (second-degree and third-degree atrioventricular block). Content from prior sessions was reinforced during the second and third sessions.

PGY-2 simulator-trained residents (n  =  40, group A) completed the simulation-based intervention in the summer/fall of 2007 (figure 1). PGY-3 simulator-trained residents (n  =  38, group B) received simulator training in the summer/fall of 2006 and completed an additional AHA ACLS provider course in the summer of 2007. The study period provided an opportunity to compare the quality of care provided by these 2 groups during actual ACLS events from January to June 2008.

A log of cardiac arrest team responses is maintained by the NMH Cardiopulmonary Resuscitation (CPR) Quality Committee. This log was used to identify 89 cardiac arrest team responses that occurred during the 6-month study period. All 89 medical records were available for review. Eight records were excluded because the cardiac arrest flow sheet could not be obtained, 30 records were excluded because they were led by non-IM residents in the Emergency Department or surgical intensive care unit, and 4 records were excluded because the ACLS response was not directed by a resident.

A sequential review of the remaining 47 medical records was conducted. Data were abstracted from the cardiac arrest flow sheet completed at each ACLS event by an intensive care unit nurse trained and assigned by the CPR Quality Committee. Because the primary study reviewer was not blinded to residents' training status, 34 randomly selected records (44%) were re-reviewed by a senior hospitalist faculty member blinded to residents' training status and results of the first review. Performance ratings from the second reviewer were compared with ratings by the primary reviewer to assess inter-rater reliability.

Medical abstraction was based on 2005 AHA guidelines that list key steps across ACLS events to compute a total percent correct performance score.15 Four measures were selected because they were common across ACLS scenarios, did not depend on physician judgment, could be scored objectively, and were based on algorithms from the AHA.8,15 The 4 key steps are (1) provision of basic life support (BLS) as a first response; (2) selection and dosage of the first drug administered; (3) sequence and dosage of subsequent drugs administered; and (4) appropriate use of defibrillation, cardioversion, or pacing. These measures were scored in a binary manner based on correct performance of each step (each coded “Yes” or “No” [Y-N]).

Abstracted data also included patient age, gender, prearrest morbidity, and prearrest ventilator and telemetry status. Prearrest morbidity was assessed by chart review for acute or chronic cardiovascular, pulmonary, liver, kidney, or infectious disease and an active history of malignancy.

Adherent responses to each ACLS event were defined as 75% or greater compliance with AHA guidelines, based on adding all 4 performance measures performed correctly. This cutoff value was used because it is identical to the mean of the passing performance standard set by an expert panel in an ACLS standard-setting study18 and had been used to assess quality of care during ACLS responses in an earlier study.8 

Inter-rater reliability was assessed using the kappa (κ) coefficient19 adjusted using the formula of Brennan and Prediger.20 Group differences between PGY-2 simulator- trained and PGY-3 simulator-trained were evaluated using the t test and the χ² test. Logistic regression was computed to determine the odds ratio of an adherent ACLS response, testing the difference between groups controlling for patient age, pre-event ventilator, and telemetry status (each coded Y-N).

The mean inter-rater reliability for the 34 randomly selected ACLS events across the 4 performance measures was high (κ  =  0.83) indicating very little disagreement in abstraction of the ACLS performance data.

Between January and June 2008, 47 ACLS events were led by simulator-trained IM residents. Twenty-five events were led by PGY-2 residents and 22 events by PGY-3 residents. The mean number of ACLS events led over entire number of residents within each cohort by PGY-2s (0.63) and PGY-3s (0.55) had a P value of 0.72. Some residents led 0 events during the study period so the mean is less than 1. Seventeen PGY-2 residents and 13 PGY-3 residents led ACLS events. As shown in the table, there were no significant differences between groups in terms of patient age, gender, prearrest morbidity score as calculated by the summation of comorbid conditions, ventilator status, or telemetry status.

After summing performance of AHA quality indicators for all resuscitation events, the overall mean percentage of correct scores for PGY-2 simulator-trained residents were 86% (SD  =  18%) and 88% (SD  =  17%) for PGY-3 simulator-trained residents. There were no significant differences between the performance scores of simulator-trained second- and third-year residents (P  =  .77) (figure 2).

When we compared ACLS quality indicator performance from 2008 to our prior findings in 2004, we found that simulator-trained groups improved significantly.8 Simulator-trained residents in 2008 (combined mean correct responses, 87% [SD  =  17.%]) outperformed 2004 simulator-trained residents (mean correct responses, 68% [SD  =  20%]; P < .001). In 2004, PGY-2 simulator-trained residents showed significantly higher adherence to AHA standards (68% [SD, 20%]) than nonsimulator-trained PGY-3 residents (44% [SD, 20%]; P < .001) (figure 2). There was no difference in patient survival to hospital discharge between groups (P  =  .83).

Using the cutoff value of ≥75% for an adherent ACLS performance in 2008, 22 (88%) of PGY-2 responses and 20 (91%) PGY-3 responses achieved this standard ( P  =  .75) (table). After controlling for patient age, prearrest morbidity, and ventilator and telemetry status in logistic regression, 2008 simulator-trained residents were more likely to lead an adherent ACLS response than 2004 simulator-trained residents (odds ratio, 5.00; 95% confidence interval, 1.22 to 20.44).

Our findings represent the results of a multiyear intervention to improve the quality of care provided to patients during ACLS events. Our main results indicate high quality of care delivered in actual cardiac arrest events by resident leaders from both PGY-2 and PGY-3 cohorts in 2008 after being trained using SBME with intense deliberate practice. This replicated the high degree of ACLS guideline protocol adherence that had been found with a cohort of simulation-trained residents in 2004.8 

We observed in this study that PGY-3 residents at 12 to 18 months after simulation training have no decay in ACLS skills with actual cardiac arrest events. Prior research shows ACLS skills are not retained without sustained practice and retesting.21 Our finding that skills acquired after SBME are substantially retained at 1 year advance similar findings of studies conducted in the simulated environment.9,22 However, other studies of simulator-based training interventions have demonstrated significant skill decay over time.23,24 Our intervention includes a standardized curriculum with intense deliberate practice, which we believe is responsible for the substantial skill retention as seen in other similarly successful interventions.25 

Our finding is made more evident by comparing results for the 2008 cohorts with those for similarly studied resident cohorts in 2004, where simulator-trained PGY-2 residents performed significantly better than traditionally trained (non-SBME trained) PGY-3 residents.8 The 2004 study convincingly displayed the fact that recent SBME, combined with deliberate practice, improved patient care in ACLS among PGY-2 residents well beyond that of PGY-3 residents who were not simulator-trained but had additional clinical experience and had recently completed an AHA ACLS provider course. Comparing cohorts from 2004 and 2008 indicates that the high quality of ACLS care by PGY-3 residents results from retention of skill from SBME rather than from clinical experience or the AHA ACLS provider refresher course. Our conclusion is also consistent with that of a growing body of literature suggesting that clinical experience is not a proxy for quality.26 

In addition to achieving the primary aim of our study was the interesting finding that the performance of simulator-trained PGY-2 residents improved from 2004 to 2008. There are several plausible explanations for this finding. The December 2005 ACLS guidelines15 emphasize high quality CPR and are more straightforward than the former version in use during the 2004 study. It is possible that streamlined AHA ACLS guidelines contributed to the improved performance of simulator-trained PGY-2 residents in 2008. Another explanation may be that ACLS skills gained by PGY-2 and PGY-3 residents are transferring to PGY-1 residents present at ACLS events. These skills are boosted even further by SBME, as a PGY-2 leading to progressively improved performance. Further study is needed to evaluate the possibility that the impact of SBME is strong enough to transfer to those not directly involved in simulation training.

We learn from the work of Pawson et al27 and Pawson28 that SBME is a complex service intervention in medical education. Such interventions have a variety of key elements including a long implementation chain and features that mutate due to refinement and adaptation to local circumstances and are open systems that feed back on themselves. “As interventions are implemented, they change the conditions that made them work in the first place.”27 Introduction and maintenance of quality improvement innovations modifies the goals, practices, and operational climate of medical education programs and hospital systems.29 Therefore, follow-up of quality improvement projects, such as our use of SBME to improve ACLS patient care, is critical to ensure that any short term performance improvements are maintained.

Our study has several limitations. It represents a small sample of events at a single institution over a relatively short time period. It used a retrospective rather than a prospective design and was not randomized. Due to the low frequency of ACLS events, it was not possible to compare each resident's individual patient care quality during the PGY-2 and PGY-3 years. Thus, we analyzed performance as a cohort. It is possible that unknown differences between cohort performance from year to year could affect our results, but we believe the groups are similar due to common clinical training experiences and baseline characteristics. The possibility of omissions or errors in chart documentation cannot be excluded. Although we are unaware of other education or quality improvement interventions in ACLS at our institution, the existence of other confounding or uncontrolled factors that impacted clinical care quality cannot be definitively ruled out. We accounted for this by focusing on 4 key process measures that would be representative of the team leaders' actions and thus be reflective of their skill level and the educational intervention.

This study replicates our past research, a fundamental principle of science,30 and confirms the key role of SBME in health care process improvement.31 This study adds to the medical education literature by showing that a simulation-based education intervention that includes deliberate practice can result in skill retention and sustained process improvement in actual patient care. We believe that SBME is a powerful tool for shaping the clinical skills of physicians-in-training and should be considered for all ACLS team responders.