Pediatrics Residents' Preparedness for Neonatal Resuscitation Assessed Using High-Fidelity Simulation Open Access

Pediatrics residents complete a neonatal resuscitation program (NRP) at the beginning of their residency and every 2 years thereafter. Following completion of these programs, they are expected to demonstrate preparedness for neonatal resuscitation (NR).1–4 Throughout the years, gaps in residents' readiness to perform NR have been noted while testing cognitive skills,5,6 via self-efficacy surveys,7–9 by observation of real-life interactions,10,11 and during simulated scenarios using low- and high-fidelity (HF) mannequins.12–15 

More than a decade ago, in a landmark contribution, Halamek et al16 noted that simulation-based training in NR was possible and offered benefits not inherent in traditional paradigms of medical education. Advanced technology (eg, HF mannequins, video recordings) allows placement of trainees in a situation that mimics the real environment closely enough to achieve suspension of disbelief.13–16 Recently, the American Academy of Pediatrics and the American Heart Association modernized NRP courses and included simulation in the methods used to educate and assess technical aspects of NR.17–19 The objective of this project was to evaluate procedural skills and team performance of pediatrics residents with simulated NR twice during their monthly neonatal intensive care delivery room (DR) rotation using an HF mannequin. A secondary objective was to ascertain whether trainees' confidence in their resuscitation skills correlated with their performance.

Each month, 2 teams participated in two 90-minute sessions (2 to 3 weeks apart) in an off-site delivery room (DR) located in the adjacent simulation center. Before each session, pediatrics residents' confidence in their ability to lead or assist an NR under 4 hypothetical scenarios was measured with a questionnaire that used a 9-point Likert-type scale20 (provided as online supplemental material). A score ≥6 for each scenario was considered the cutoff for definite confidence.20 Each team consisted of a resident and 2 interns (general pediatrics and/or medicine-pediatrics), and each session included an introduction to simulation, a short skills practice, and review of the video “Just a Routine Operation” (http://vimeo.com/970665, accessed July 2, 2013), which highlights gaps in team performance. Those activities were followed by a 15-minute scenario (modified from SimNewB NRP 2010, American Academy of Pediatrics, Elk Grove Village, IL), which involved an NR of a 37-week gestational age infant (HF mannequin SimNewB; www.laerdal.com, accessed July 2, 2013) born by cesarean delivery to a woman with preeclampsia and placental abruption. Apgar scores were 1 at 1 and 5 minutes, respectively. The NR required proficiency in 5 skills: positive pressure ventilation (PPV), chest compressions (CC), endotracheal intubation (INT), umbilical vein catheterization (UVC), and intratracheal (ETepi) and intravenous (IVepi) administration of epinephrine. The PPV was provided via a flow-inflating bag connected to a pressure gauge and a manometer. Pulse oximetry was available. By design, there was not a heart rate response ≥60 beats/min to ventilation and ETepi until the correct dosage of IVepi had been given through the UVC. Teams were expected to perform without instructors' guidance or redirection. Video recording of the scenario was used for a 20- to 30-minute debriefing and later for independent scoring by a neonatologist, a neonatal fellow, and a neonatal nurse practitioner (NRP instructor). Each of the 5 skills included 4 technical elements, given 1 point each if performed correctly (table 1). A total score ≥3 (≥75% of 4 points) for each individual skill and a cumulative score ≥15 (≥75% of 20 points) were considered acceptable performance.

Each procedure was expected to start or be completed within a predetermined time and scored accordingly (0 points if not done, 1 point if done late, and 2 points if done on time). The times for initiation of PPV (≤60 seconds) and for CC (≤120 seconds if preceded by bag/mask ventilation or ≤180 seconds if preceded by INT) were derived from NRP guidelines.17,18 The maximum time for INT (as documented by end tidal Pco₂ color change) was 180 seconds. This limit was selected from data of successful INT in real-life or in simulated NR.11,12 The ETepi administration was expected by 300 seconds, UVC placement by 420 seconds, and IVepi by 480 seconds. These time expectations were in line with those observed in real-life NR.21 For each individual skill, a timeliness score of 2 was considered acceptable. A total cumulative score for timeliness ≥9 (≥75% of 12 points) was considered acceptable.

Team behaviors were evaluated by a method modified from Thomas et al22 that consisted of 3 items (scored 0 to 4 each): communication (information sharing and inquiry), management (workload and vigilance), and leadership (assertion and evaluation of the plan). A total score ≥3 (≥75% of 4 points) for each individual team behavior and a cumulative score ≥15 (≥75% of 20 points) were considered acceptable performance.

Procedural skills, timeliness, and behavioral scores were expressed as medians and were compared with Wilcoxon signed rank test. Self confidence and acceptable performance scores were compared using the McNemar χ² test with correction for continuity. Interrater reliability, measured by Krippendorff κ coefficients23 for skills were PPV κ  =  0.26, INT κ  =  0.78, CC κ  =  0.48; ETepi and IVepi κ  =  0.65; and UVC κ  =  0.63; for team behaviors were communication κ  =  0.37, management κ  =  0.40, and leadership κ  =  0.57.

This project was declared exempt from review by the Institutional Review Board.

Eleven teams (pediatric level [PL]-1, 10 [38%]; PL-2, 12 [46%]; and ≥PL-3, 4 [15%]) completed the 2 sessions. The NRs were always led by senior residents. During the 2- to 3-week interval between sessions, residents increased their clinical experience by attending approximately 50 low-risk and 40 high-risk deliveries that required varying degrees of NR.

Before both sessions, all 16 senior residents (100%) felt very confident assisting with all NR scenarios. Before the first session, 14 of 16 senior residents (88%) felt comfortable leading the NR of the first 3 hypothetical scenarios, but only 1 of 16 (6%) felt confident leading the NR of the 26-week gestational age infant. Before the second session, the senior residents' level of confidence for the hypothetical cases increased significantly (P < .01). Before the first session, half of the 10 interns (50%) felt confident assisting but none (0%) felt confident leading the NR scenarios. Two weeks later, however, all (10 of 10, 100%) felt comfortable assisting and half (5 of 10, 50%) felt comfortable leading the NR of the first 3 scenarios, but none (0%) felt confident leading the NR of the 26-week gestational age infant. These changes were also significant (P < .01).

Median PPV scores were 3 during the first and 3 (P  =  .44) during the second session. Drying/stimulation, suction, and evaluation of heart rate and respiratory rate were consistently done by all teams. Bag/mask ventilation, however, was incorrectly performed by 5 of 11 teams (45%) because of their inability to maintain bag inflation by not controlling the continuous positive airway pressure valve or the oxygen flow.

Median INT scores were 3 during the first and 4 (P  =  .44) during the second session. Good technique for INT was shown by most teams; however, 5 teams needed 2 attempts while 6 teams needed 3 attempts. Securing the tube with a NeoBar (Neotech Products Inc, Valencia, CA) was not common, thus, operators kept their hands on the tube, limiting their participation in other tasks. That error did not reoccur during the second session.

Median CC scores were 3 during the first and 4 (P  =  .68) during the second session. Although technique was adequate, ventilation count was either too slow or too fast in 7 of 11 NRs (64%) during the first session. Only 2 of 11 team leaders (18%) recognized that the operators were tired and reassigned the task to another team member. During the second session, these problems were less common.

Median ETepi and IVepi scores were 3 during the first and 4 (P  =  .11) during the second session. During the first session, only 6 of 11 teams (55%) knew the correct doses. Most residents did not know how to draw epinephrine, operate the 3-way connectors, or flush epinephrine with saline. These errors did not reoccur during the second session.

Median UVC placement scores were 2 during the first and 3 (P  =  .26) during the second session. Only 5 of 11 teams (45%) placed umbilical tape around the cord to control bleeding and only 5 teams (45%) withdrew “blood” from the UVC to evaluate placement. Six teams (55%) placed a UVC without saline, thus injecting air into the mannequin, and only 1 of 11 teams (9%) secured the UVC. These performance gaps reoccurred during the second session. Median combined scores for procedural skills did not change between sessions (14 and 17, P  =  .13).

Median scores for the first and second session were 2 and 2 for PPV; 1 and 2 for INT 1; 2 and 1 for CC; 1 and 1 for ETepi; 1 and 1 for UVC; and 0 and 1 for IVepi. Median combined scores for all procedures did not change from the first to the second session (8 and 8, P  =  .16). None of the differences in scores was statistically significant.

Median communication scores improved from the first to the second session (3 and 4, P  =  .05), but management scores did not (3 and 3, P  =  .18). Distribution of tasks without considering the capabilities of each resident was common. Vigilance was also compromised by allowing team members to focus only on their assignments while ignoring their potential contributions to other tasks. Neither median leadership scores (3 and 3, P  =  .39) nor median combined team scores (9 and 9, P  =  .16) changed between sessions. Lack of assertion and situational awareness compromised team performance.

The number of teams with acceptable performance (individual and combined) for technical skills, timeliness, and team behaviors are presented in table 2. Although none of the observed changes in individual technical skills between the 2 sessions reached significance, the overall trend for improvement was definite (Wilcoxon signed rank test, P  =  .02). The apparent dramatic change in teams with acceptable performance in UVC placement (from 2 to 7) as well as in the timeliness of IVepi (from 5 to 0) are a consequence of the inconsistencies of the different teams' performances.

Limitations of traditional NRP teaching and those of similar programs have been recognized, and modernized educational resources have been incorporated.4,10,13,17,18,24–27 As demonstrated here, video recordings of simulated scenarios are valuable tools to assess individual as well as team performance.12–15,25–27 

The participation of our residents during the NR and video debriefing, together with the 2 to 3 weeks of clinical experience, may have increased the residents' confidence in their ability to lead or assist in NR. The contrast between gains in self-confidence and actual performance has been observed in real-life interactions and during mock resuscitations.4,22,27,28 

Performance gaps in procedural skills during real-life and simulated resuscitation of newborns have been previously noted.10,12,29 A high rate of failure in intubation and an inability to correctly assemble and use the laryngoscope during simulation training have also been reported.7,12,13,29 Like others, we observed an inability to draw epinephrine from a vial, as well as an inability to place and correctly secure an endotracheal tube or a UVC, even after the completion of traditional NRP.3,4,7,20,24,30 

Most investigators have focused on the technical aspects of procedural skills and not on the timeliness of the task.31,32 Our assessment also included an evaluation of the timeliness (time to initiation or time to completion) of each individual skill. Important delays in task initiation (PPV or CC) or task completion (INT, epinephrine administration, UVC placement) observed during the first session continued in spite of the additional clinical experience. It is possible that following debriefing of the first session, our residents recognized the need for performing each procedure precisely, but in doing so, they may have prolonged the time to completion.

Current NRP guidelines define when to start PPV, but they are less clear about CC and even INT.18,19 For this reason, we chose different start times for these skills, depending on whether the pediatrics residents chose PPV or INT first. Our time expectations for completion for the remaining tasks were selected from clinical data taken from real-life scenarios.11,21 White et al12 recorded the time to completion of clinical skills, but no conclusions were drawn because of the lack of standards for comparison. Other investigators who compared video and traditional, oral debriefing of simulated NRs reported no differences in time to intubation—a finding of questionable significance because, during the performance of the skill, their instructors interrupted and redirected the trainees.15 Finally, without providing specifics, Bismilla et al3 observed that the prolonged time taken by pediatrics residents and fellows to intubate in real life was concerning.

In this group of previously NRP-certified residents assessed in a realistic NR scenario, no significant increases in acceptable performance were noted over time. This could be explained by the few opportunities available during their neonatal intensive care unit DR rotation for more simulation training and by the fact that, in our institution, real-life NRs are lead by attendings, fellows, or neonatal nurse practitioners thus further limiting the residents' participation. In light of the above, it is possible that a more structured, delivery room–centered, short intervention that includes simulation and deliberate practice may be necessary to improve performance.30,33 

Team behaviors can be assessed during real-life DR resuscitations or in simulated scenarios similar to ours.2,21,34 Video recordings facilitate evaluation of behaviors and global team performances.10,22,34 Between the sessions, communication improved, whereas management and leadership did not. Communication is a fundamental behavior for effective team performance.2 These preliminary observations are encouraging, considering that ineffective communication and teamwork contribute in up to 70% of the cases of medical errors and patient harm.35 

A post hoc analysis showed that the small sample size had sufficient power to detect only large effects. Another limitation of the study is that adjustments for multiple outcomes would further reduce the power reported above. Finally, it is possible that residents' performance during a simulated NR may not be reflective of their reactions during real-life situations where other providers with different degrees of expertise participate.

The HF simulation offers opportunities for NR and team skills training and assessment. In a typical neonatal intensive care unit rotation, pediatrics residents' self-confidence increased with simulation training and clinical exposure; however, that improvement did not correlate with gains in procedural skills and some team behaviors.