Pediatric Intensive Care Simulation Course: A New Paradigm in Teaching

The 2003 Accreditation Council for Graduate Medical Education (ACGME)1 duty hour standards limited the number of hours in which resident education can occur. Medical educators need to develop innovative strategies to improve the efficiency of resident learning, and simulation can be an effective solution. Mock codes with mannequins help residents gain experience and confidence in treating patients with critical illness. High-fidelity simulators can provide lifelike interaction and instantaneous feedback during educational exercises and may be superior to other forms of training in improving long-term performance on important procedure skills, such as resuscitation, intubation, and intraosseous line (IO) placement.2 Simulation builds on the teaching theories of adult and experiential learning,3,4 which focus on “just in time” acquisition of knowledge, incorporation of prior learning experiences, and learning by reflection. Other benefits to simulation include dedicated time for reflection and standardization of the curriculum and clinical situation.

The pediatric intensive care unit (PICU) is a high-risk environment with a potential for errors that can have significant consequences for patients and the residents who care for them.5 The disease spectrum is vast, the patients range from newborns to young adults, and rare events such as cardiopulmonary arrest require rapid and effective intervention to ensure the best chance for survival. While Pediatric Advanced Life Support (PALS) training aims to improve health care providers' performance in codes, the literature reports rapid decay of skills and knowledge.6,7 We believe simulation-augmented education is crucial for residents who need to be prepared to manage rare, life-threatening events.

Hospital mock codes allow the code team, which includes pediatric residents, to gain experience in low-frequency, high-risk events and improve confidence and performance.8–10 However, we11 have reported that even bimonthly in situ mock codes did not lead to individual resident learning; instead they highlighted system problems. Based on these results, our current study sought to improve individual resident confidence and performance in managing pediatric code events and performing procedures on critically ill patients. Our course emphasized IO placement, an important procedure in the PICU,11,12 and one in which our residents perceived themselves to be deficient. We hypothesized that confidence levels with code skills, especially IO placement, intubation skills, and ability to supervise a code would improve after exposure to a simulator-based PICU training course. In addition, we evaluated resident self-perception of their skill level and compared this to video evaluation of their performance. This article reports the results of this educational study.

Our simulation sessions consisted of monthly practice with IO placement followed by weekly videotaped, simulator-based scenarios for 12 months. For each scenario, residents rotated through 3 primary responsibilities: code leader, airway, and cardiac management. Each postgraduate year (PGY)–2 resident had 2 PICU rotations and thus 6 simulation scenarios in 2 blocks. All PGY-2 pediatric residents participated in the simulation scenarios and were videotaped. Nurses, respiratory therapists, pharmacists, and actors playing the roles of parents enhanced the realism of a multidisciplinary team. The Institutional Review Board at the University of Alabama at Birmingham approved the study.

Each session consisted of a 5-minute orientation, a 15-minute videotaped scenario, and a 40-minute debriefing period. During the orientation, residents practiced and successfully placed an IO needle into a chicken leg bone with direct supervision by the authors. The chicken leg bones more closely parallel the “feel” of an infant anterior tibia than the plastic mannequin models used by many PALS courses. Intubation skills were practiced in the context of each scenario and directly observed by the authors. Successful endotracheal placement was confirmed by the computerized mannequin, which also provided feedback on end-tidal carbon dioxide tracing and oxygenation changes. Each scenario was videotaped and followed by a debriefing session; these debriefing sessions composed 60%–70% of the overall sessions. Facilitators probed participants about the thought process behind their actions and decisions during the scenario to identify knowledge and skills gaps.13 

All simulator sessions were conducted in the pediatric simulation center utilizing high-fidelity simulators, including SimBaby (Laerdal Medical, Wappingers Falls, NY) and METI child EPS (METI, Sarasota, FL). All scenarios were adapted from Pediatric Resuscitation: A Practical Approach.14 Checklists were developed for each scenario from PALS curriculum15 and pediatric critical-care experts. Each checklist assessed important items of history, physical exam, diagnosis, and management, as well as skill components of the case such as cardioversion and intubation. The intubation checklist was standard across all scenarios and assessed items such as preparation, preoxygenation, cricoid pressure, sedation, and postintubation assessment of proper endotracheal tube placement (bilateral breath sounds, end-tidal carbon dioxide assessment, and chest radiograph). Checklists for each scenario are provided in the online Supplemental Content A. Two pediatric critical-care physicians independently reviewed all videotapes, assessing each scenario with the preset checklist and recording the percent of skills performed successfully (0%–100%).

We surveyed participating residents (n = 54) before and after intervention. Residents were assigned a unique identifying number on each survey that was blinded to all of the investigators, which allowed comparison of an individual resident's presurvey and postsurvey responses.

Our previously published survey11 that was used in this study was developed from work by Cappelle and Paul.10 The 2-part survey is scored on a 5-point Likert scale. Section A consists of 4 questions concerning residents' attitudes about codes, and section B concerns residents' self-assessment of their perceived ability to perform PALS skills, such as intubation, managing dysrhythmias, and performing chest compressions (the survey is available online as Supplemental Content B). The worry index score was the sum of the first 3 questions from section A (“Codes scare me,” “I need more knowledge about codes,” and “I need more experience with codes” [range 3–15]), with 15 being a high worry index score. The skill index score was the sum of the 10 questions from section B (ability to intubate infants, toddlers, children, and teens; ability to “run” code; ability to treat [respiratory arrest, seizure, cardiac dysrhythmias]; ability to perform chest compressions; and ability to place IO line [range 10–50]), with 50 indicating higher perceived ability to perform code skills.

Third-year residents (PGY-3) who had not participated in simulation sessions during their PICU rotations served as controls and were surveyed just prior to starting their third year after completing their PICU rotations. Residents were assigned a unique identifying number on their survey that was blinded to all of the investigators, which allowed comparison of an individual resident's presurvey and postsurvey responses.

Scores from the surveys of PGY-2 residents with PICU mock-code simulator experience (intervention group) were compared to surveys of PGY-3 residents from the previous year without PICU mock-code simulator experience (control group) using Wilcoxon signed rank analysis. Video scores were averaged by the 2 raters. Pearson correlation was used to assess interrater reliability. Average scores with standard deviations are presented along with 95% confidence intervals (CI). In addition, average videotape scores from the first PICU rotation were compared with average scores from the second PICU rotation using an unpaired t test, and average scores from each scenario were compared using a 1-way analysis of variance with post hoc analysis with a Student-Newman-Keuls test as homogeneity was found between subsets. Participants' presurvey and postsurvey scores and video scores were compared using Wilcoxon signed rank analysis. All tests were 2-tailed, and P < .05 was considered significant. Analysis was performed using SPSS 11.5 (SPSS, Chicago, IL).

All intervention-group residents (PGY-2, n = 16) and 78% (14 of 18) of the control-group residents (PGY-3) completed the first survey; 94% (15 of 16) of PGY-2 residents completed the postsurvey after the 12-month study period. The average worry index before intervention was 14.1 ± 1.2, versus 11.5 ± 1.9 after the simulator course (P = .001, 95% CI, 1.72, 3.48). The worry index was higher in the PGY-3 residents without PICU code sessions than in PGY-2 residents with PICU code sessions (12.5 ± 1.6 versus 11.5 ± 1.9, respectively). All PGY-2 residents had improvements in their worry index, indicating less worry about codes (figure 1). The average presurvey skill index was 25.1 ± 7.7, improving to 38.0 ± 5.3 postintervention (P < .001, 95% CI, 9.7, 16). The skill index was similar in both groups (38.4 ± 4.4 versus 38.0 ± 5.3, respectively; see figure 2).

The table illustrates the confidence level improvement for each individual skill during the study period. All perceived improvements of individual skills were statistically significant except for the ability to perform chest compressions. The 4 largest self-assessment improvements were the perceived ability to place an IO line (P = .001), the perceived ability to intubate 1- to 3-year-olds (P = .001), the perceived ability to intubate 3- to 12-year-olds (P = .001), and the ability to supervise a code (P < .001). When compared to PGY-3 controls, PGY-2 residents with PICU simulation experience had significant improvement in their perceived ability to intubate 3- to 12-year-olds (P = .05) and trended toward improvement in their perceived ability to place an IO line (P = .07). When compared to the control group, the confidence of all skills improved. Only the ability to intubate toddlers and children 3–12 years old was statistically significant, with IO line placement nearly reaching significance (figure 3).

Objective review of videotapes using checklists showed an average success score of 54% ± 12% of skills performed. The Pearson correlation for the 2 reviewers was 0.866. The average score on checklists from the first PICU month, which included scenarios of hypovolemic shock, supraventricular tachycardia, and croup was 58% ± 12%. The average score on checklists from the second PICU month, which included scenarios of septic shock, tricyclic antidepressant ingestion/wide complex tachycardia, and foreign body was 48% ± 10%. This was lower than the scores from the first month (P = .02). This difference was accounted for in the change of scores from the 2 cardiac scenarios. The average scores for individual scenarios varied significantly from one another (P = .002). The highest average score was with supraventricular tachycardia (68% ± 13%), followed by croup (58% ± 13%), hypovolemic shock (55% ± 7%), septic shock (54% ± 9%), foreign body (51% ± 11%), and finally tricyclic antidepressant ingestion/wide complex tachycardia, the lowest score (40% ± 3%), which contributed to the lower checklist scores for the second month. Finally, each pair of cases was compared. The only statistically significant difference was found between the 2 cardiac simulations (P = .001).

Residents rated the simulation course as a very positive learning experience, citing the high degree of realism, “thinking on their feet,” and timely feedback. A sampling of individual resident feedback included the following:

• “The practice sessions simulate real-life situations, but allow us time to review every decision in a calm, nonthreatening environment. This only helped to deepen my understanding of the disease process and the rationale behind interventions. Having done a couple of these sessions, I've found that repetition has helped build confidence in my decision-making skills and solidify previously learned lessons.”

• “Codes are a time for quick, organized thinking which is very difficult without practice. I liked the variety of the scenarios and the different directions each code could take depending on your choices. I also liked having the opportunity to practice the IO placement. During my [emergency room] rotation, I was much more comfortable attempting the IO placement in a real code situation after our practice.”

• “I think the mock codes are one of our best learning experiences in residency.”

A weekly simulation-based PICU course improved second-year residents' confidence about their ability to perform effective overall resuscitations and was especially helpful in allowing pediatric residents to gain additional confidence in performing skills such as intubation and IO placement. Although all residents' confidence improved, video-based performance reviews failed to improve during the study period.

Our scenarios were based on common severe pediatric emergencies due to respiratory insufficiency and shock.16 When not recognized and treated promptly and correctly, these often progress to cardiac arrest. Survival rates following such an arrest are dismal.17,18 We felt it was important for PGY-2 residents to focus on recognizing these emergencies early and intervening before cardiopulmonary arrest ensues.

Our findings are similar to a multi-institution “boot camp” simulation course for first-year pediatric critical-care fellows.19 Their course development was driven by factors similar to ours: acute, complex patients in the PICU, an increased emphasis on patient safety, and work hour limitations. Although training different levels of learners, that course focused on similar skills including managing a compromised airway, cardiac dysrhythmias, and shock management. Their learning environment was high intensity, whereas our setting consisted of weekly hour-long sessions during the residents' 2 months of PICU rotations. Several studies20,21 suggest that learning and retention of psychomotor skills can be improved if opportunities for deliberate practice are distributed over time, rather than provided on a single occasion.

Both the worry index and the skill index were found to improve significantly during each resident year, such that PGY-3 scores were higher than PGY-2 scores, which were higher than the first-year pediatric residents' scores.11 The skill index assessed performance of tasks regarded to be mandatory by the ACGME Pediatric Program Requirements as well as important for PALS certification.22 These factors add to the content validity of these indexes. However, the indexes have not undergone rigorous independent reliability and validity testing.

Two skills in which the greatest perceived improvement occurred was the ability to place an IO properly and intubation of children (nonneonates). Although IO placement is discussed in all PALS courses and is considered easy to learn, we found our residents were not confident with this important procedure.11 Gaies et al12 found similar findings when surveying pediatric program directors. More than 60% of program directors felt IO placement was a very important skill, but just over 20% felt their graduating residents were competent to perform IO placement. Residents improved their confidence with IO placement with regular practice using chicken bones. Residents were surprised at the degree of force required to place an IO needle in a bone and often slipped off the side of the bone versus going into the IO cavity. Regular IO practice with the chicken bones appears to give residents both the confidence and competence to place an IO in a real clinical situation.

Confidence with tracheal intubation of children also improved during our course. Many of our scenarios require intubation, and this practice with direct supervision and immediate feedback helped improve confidence. There was a differential skill perception in intubating neonates versus nonneonates. This was also seen by Gaies et al,12 who found only 30% of program directors felt their residents were competent to intubate nonneonates versus 60% who felt their residents were competent to intubate neonates. This discrepancy is likely due to the increased clinical experience of intubating neonates during neonatal intensive care rotations. This subjective rate of competence is similar to the findings of Falck et al,23 where direct observation of neonatal intubations revealed the success rate of PGY-2 residents to be 55% and PGY-3 residents to be 62%. We demonstrated that confidence performing nonneonatal intubation can also be improved with practice. Further studies are needed to determine if this improved confidence translates into improved skill.

Our debriefing used a technique developed by the Center for Medical Simulation (Cambridge, MA), known as “Debriefing with Good Judgement.”13 This technique is based on behavioral science theory to improve reflective practice of professionals24 and reveals the importance of narrative comments. Further work could be done to further quantify residents' responses to this form of debriefing. During videotape reviews, we found that only a moderate percentage of skills were performed properly. The percentage of skills performed properly in our study was 54%, which is comparable to Donoghue et al,5 who found 50%–61% effective performance when evaluating PALS scenarios. The types of skills assessed in each study, such as cardiopulmonary resuscitation, intravenous/intraosseous access, and use of epinephrine, was similar. Our self-assessment tool asked about specific skills such as intubation and chest compressions and did not assess overall ability to care for critically ill children. Our checklist, on the other hand, evaluated overall patient care, looking at aspects of integrating history and physical elements of case with decision making during a crisis. We may have found better improvements if we had focused on skill acquisition. In addition, our scenarios were not the same during both assessment time periods. Specifically, when comparing our 2 cardiac scenarios, supraventricular tachycardia and tricyclic antidepressant overdose/wide complex tachycardia, the second case had significantly lower scores and may have been more difficult.

It is concerning that although residents' perception of their abilities improved during the study period, the objective performance checklist did not demonstrate improvements in the percentage of successfully completed skills. Lum and Galletly25 also found that performance was not related to confidence, an important finding to consider when developing any simulation study. Though self-assessment can be inaccurate, it helps form our sense of confidence and ability.26 When caring for patients, often there is no objective data of our care, only subjective self-assessment. Our study found that this self-assessment is inaccurate, which is similar to research reporting that humans in general have a poor ability to accurately perceive their abilities.26 Interestingly, our ability to self-perceive skills when there is an observable outcome may be better than our ability to self-judge mental processes.27 Simulation allows for both of these processes—skills with observable outcomes and cognitive processing—to be witnessed. The learner is able to practice a skill and, with expert debriefing augmented with videotaped feedback, may be able to better align self-perception with reality. By taking advantage of simulation's ability to objectively review performance, we have a unique chance to potentially improve performance and emphasize the inaccuracies of our own self-perceptions. Our study found this inaccuracy of self-assessment to be a result of overconfidence, but Peyre et al28 found that surgical residents were more self-critical of their own laparoscopic performance than were expert faculty reviews.

The lack of improvement in performance despite an improvement in worry index bears further examination and study. There may be important study design reasons for this lack of improvement. First, Nishisaki et al19 found that the “train to success” strategy with repetitive practice resulted in higher perceived training effectiveness compared with a single scenario followed by a debriefing session, as we used in our study. Second, 6 hours of dedicated simulation training may not be enough time to show the benefit of this educational modality. Nadel et al8 found that pediatric residents as a group have very little experience, either real or simulated, in leading resuscitations during their residency. This complex skill may take more time to develop, possibly hindering our ability to find a difference in actual skill performance using our study design.

Our study has some limitations. First, we did not account for differences in residents' abilities and experiences; we assumed the only differences between the control and intervention groups were the simulation sessions. A second limitation of our study was that we did not use the same scenarios. Although the simulation cases were similar during the first half and second half of the study, the level of difficulty may have changed, and this may have contributed to some of the lack of objective improvement. We thought that experiencing different scenarios would enhance learning by providing increased learning opportunities; however, repeated practice may be a preferable strategy. Third, we used a survey tool that appears to have construct validity but has not been rigorously tested. Finally, our scenario checklists were developed from the PALS curriculum and adapted by critical-care experts but may have overemphasized or underemphasized some treatment steps. Our checklists have not as yet been rigorously evaluated for content validity.

Simulation in health care is well aligned with adult learning principles and the recent emphasis on patient safety. At the same time, simulation represents a culture change in learning that needs to be measured in the long term. Our simulation-based course targeting all PGY-2 residents improved learner confidence in running codes and in pediatric resuscitation skills, especially skills that were practiced during each session. Videotaped reviews revealed areas of discordance between observed resident performance and perceived ability during simulation codes. With further study, assessing skills with observable outcomes may be more useful than cognitive-processing self-assessments. In addition, it is vital to understand if similar differences occur in actual patient care settings. As most of our perceptions of ability are shaped by our self-assessment and not objective performance measures, this may be an important area for further study to attempt to better align perception and reality.