A Multirater Instrument for the Assessment of Simulated Pediatric Crises

During the past decade, simulation has been increasingly used to teach the knowledge and skills needed to manage pediatric medical crises.1–4 In a typical pediatric crisis simulation, participants receive a clinical story and are then asked to manage the patient as they would in real life, using a high-fidelity mannequin as a patient proxy. Participants are then debriefed regarding their experience. This debriefing encourages reflection upon clinical performance and gives the opportunity to modify future behavior.5,6 Traditionally, written feedback is not a part of debriefing, even though it could enhance learning by giving participants materials to reflect on after the session's conclusion. Written feedback, however, is contingent on the existence of valid assessment instruments. To date, several tools have been constructed to meet this need, and their characteristics are presented in table 1.7–11 Although each has its strengths, none take advantage of recent developments in multirater feedback methodology.

Multirater feedback is a technique derived from the business domain that has been successfully adapted to medical education.12–17 Its advantage lies in the synthesis of multiple perspectives to achieve a more stable, global rating.18–20 An additional property is the ability to incorporate learner self-assessment in what is known as gap analysis.16,18 Gap analysis examines the difference between the combined scores of “expert” faculty raters and the learner's self-score to obtain a measure of that individual's self-appraisal. “Positive” gaps indicate areas where learners underappraise their abilities, while “negative” gaps indicate areas where those abilities are overappraised, a concerning phenomenon suggesting an inability to accurately reflect on performance. By quantifying self-appraisal, gap analysis enables faculty to give focused feedback tailored to alter inaccurate learner self-perception.16 This technique has been applied to communication skills assessment16 but has not been used to date in the assessment of code team skills.

Given the enhanced feedback made possible by these techniques, we sought to develop a multirater tool with gap analysis for use in simulation-based pediatric crisis resource management (CRM) courses. An explicit goal of our development process was to adhere as closely to the Accreditation Council for Graduate Medical Education (ACGME) core competencies as possible. In July 2008 we introduced this tool, the Team Performance during Simulated Crises Instrument (TPDSCI), into our pediatric CRM course at the University of Louisville and Kosair Children's Hospital. The goal of this article is to report the psychometric and gap analysis data derived from this pilot.

This study was approved by the University of Louisville Institutional Review Board.

During the initial development phase, we discovered that competency scores based on individual performance have limited utility due to the team-oriented nature of medical crises. Therefore, we chose to use the entire code team, rather than the team member, as the unit of assessment. The code management literature, which suggests that team cohesiveness affects outcome more significantly than individual performance,21,22 supported this decision.

During tool development, we assessed the ACGME core competencies (patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice23) as they related to code team skill. We sought construct validity by adhering to this framework. Practice-based learning did not appear measurable in the context of a time-limited training session, as this competency reflects long-term patterns of growth and development, and was therefore not included.24 We chose to focus the patient care competency on procedural aspects of patient care, as other elements of this competency do not pertain to medical crises,24 and further limited the procedures involved to key airway, breathing, and circulatory interventions such as bag-mask ventilation, intubation, and chest compressions.

We used a short, rubric-based format to allow for faster, less educationally disruptive assessment. Rubrics were developed by listing key code team behaviors that might fit within each competency and ranking them in order of importance for a successful outcome. This list was transformed into descriptive statements that were further compiled into a series of 5 ranked paragraphs per competency. Each ranked paragraph was given a descriptor based on content and assigned a numerical value (1 = poor, 2 = fair, 3 = good, 4 = very good, 5 = excellent) to enable score averaging. Paragraphs containing multiple behaviors were phrased such that all must be present for a particular score to be merited. The completed tool was reviewed by 2 additional pediatric intensivists for accuracy and transparency of language. Suggested alterations were incorporated into the final version of the tool (online supplemental material).

The tool was piloted for 1 year during sessions of the Simulation for Pediatric Assessment, Resuscitation, and Communication Program's CRM training program. This program consists of 1-hour simulation-based CRM sessions conducted in the University of Louisville's Paris Simulation Center, Kosair Children's Hospital Pediatric Critical Care Center, and Kosair Children's Hospital Emergency Department. Participants include pediatrics and medicine-pediatrics residents, pediatric critical care and emergency nurses, respiratory therapists (RTs), and pharmacists. All sessions included participants with varying levels of clinical experience as defined by postgraduate year for physicians and years spent in critical or emergency care settings for nurses, RTs, and pharmacists. Each session begins with an introduction to the simulator followed by a discussion of CRM principles and a brief case history. Session participants then engage in a simulated pediatric crisis, organizing into an interdisciplinary code team to address the situation. Cases included primary cardiac, respiratory, and neurologic derangements. After each simulation the team is debriefed regarding teamwork, procedural skills, and medical knowledge.

During the pilot, the TPDSCI was completed by course faculty. Up to 3 raters were used, depending on faculty availability. A total of 15 faculty raters contributed during the course of the study. Participant teams reached consensus and completed self-evaluations using the tool. Raters received no instruction prior to using the TPDSCI, as our intent was to create a tool transparent enough to be usable “off-the shelf” without need for training sessions. Pilot data were then analyzed. Sessions in which intensive care unit faculty participated were excluded from analysis to minimize bias.

Competency-specific scores were calculated by averaging individual faculty scores for each competency. Overall score was determined by first averaging the individual competency scores for each faculty rater and then averaging this score between all faculty raters. Average scores of 3 (good) or greater were defined as meeting or exceeding expectations, and scores of less than 3 (poor or fair) were defined as needing improvement. Gap analyses were calculated by subtracting the self-score for each competency and, for the overall resuscitation, from the corresponding average faculty score. An absolute gap of 0.5 has been suggested in the literature as a cutoff for significance.25 Based on this, we defined a gap of 0.5 or greater as signifying self-underappraisal, and a gap of −0.5 or less as signifying self-overappraisal.

We examined the TPDSCI for internal consistency using Cronbach α, calculated using aggregate data from all included sessions. To address the issue of potential intrasubject correlation, a second α value was calculated by randomly sampling 1 rater from each simulated session, determining an α for that subset, and then repeating this process until all raters were accounted for. The resulting numbers were then averaged to provide a composite Cronbach α.

To determine interrater reliability, intraclass correlation (ICC) coefficients were calculated for each competency as well as for the overall score. Data from sessions rated by less than 3 faculty members were excluded from this analysis as a stable number of raters are needed to produce the statistic. A statistical assessment of construct validity could not be performed as each team was a heterogeneous mix of clinicians with differing experience levels, and we possessed no other information that could serve as a “gold standard” for team crisis management ability.26 We instead chose to present the range of scores descriptively (figure 1).

Gap analysis results were examined for frequency of significant positive (self-underappraisal) and negative (self-overappraisal) gaps. Gap data were correlated with faculty rater scores for domains in which the groups scored less than 3 (defined as needing improvement) or achieved a score of 3 or greater (defined as meeting or exceeding expectations). These results are presented descriptively in table 2.

After initial exclusions, 54 teams, composed of a total of 79 residents, 128 nurses, 9 RTs, and 8 pharmacists, were found suitable for analysis by Cronbach α. Ten of those teams were subsequently excluded from ICC analysis. figure 2 gives a schematic of the exclusion process. Individual teams consisted of, on average, 4 residents (range, 1–7) and 4 nurses (range 0–8). Experience levels varied between participants, with teams typically containing both experienced and inexperienced members. Pharmacists and RTs attended 19% (10/54) and 13% (7/54) of sessions, respectively. Due to logistical issues, only residents were able to attend Paris Simulation Center sessions.

Overall Cronbach α for the TPDSCI was 0.72. Using the averaging process described, we found the Cronbach α to be 0.69. ICC values for the medical knowledge, clinical skill, communication skill, and systems-based practice competencies ranged between 0.72 and 0.87. ICC for the professionalism domain was 0.22. The overall ICC for the TPDSCI was 0.82. Details regarding these statistics are displayed in table 3. figure 1 depicts the range of scores present within each individual sessions arranged from lowest to highest average score.

Significant gaps were found among all 5 competencies assessed by the tool. Forty-three (97%) sessions assessed were found to have significant gaps between faculty and self-assessment in at least 1 competency; 38 (86%) sessions had at least one significant gap indicating participant team self-overappraisal, and 15 (34%) sessions had at least one significant gap indicating self-underappraisal. Significant gaps were found between overall faculty and team scores for 23 (52%) sessions. Individual domain scores for each team were further examined to determine the prevalence of correlated significant gaps. Of 220 possible correlations, individual domains were associated with a significant gap (either self-overappraisal or self-underappraisal) 136 times (62%). As data regarding self-appraisal in a given area must be interpreted in light of the participant's objective abilities in that area to be useful for feedback,16 we examined these 136 instances further by correlating instances of self-overappraisal or self-underappraisal with average faculty scores in those areas. These results are shown in table 2.

There is strong evidence for the internal consistency and overall interrater reliability of the TPDSCI when used in a multirater context. Cronbach α for the instrument ranges between 0.69 and 0.72, indicating that the individual domains contribute to the description of a single construct, namely overall resuscitation competence. This further suggests that an overall score obtained by averaging individual domain scores is meaningful. Although higher alphas are typically sought, our tool contains a deliberate degree of heterogeneity that would render higher alpha levels suspect. The overall ICC of 0.82 further suggests that global TPDSCI scores are reproducible between multiple rater groups. As this study deployed the TPDSCI in the actual situation for which it was designed, these statistics provide strong evidence for the internal consistency and interrater reliability of the TPDSCI when used by multiple raters to assess team performance during simulated pediatric crises.

The ICC scores for most individual competency scores were similarly high, and strong evidence exists for the stability and reproducibility of these scores between rater groups. Professionalism is a significant exception, as the ICC for this competency was quite low. Further examination of the raw data for professionalism revealed that only 1 rater gave a professionalism score of fair, with all other scores ranging from good to excellent, suggesting a significant positive skew in this competency. This is not entirely surprising, as there is reason to believe that this type of assessment methodology may not be ideal for the assessment of professionalism.27–30 

Construct validity was difficult to assess given the team-oriented nature of the tool. Although it would have been ideal to compare scores to some external anchor of expected performance, such as participant experience,26 this was not possible because of the heterogeneous experience levels in the participant teams. In addition to this, we did not possess information regarding any other experience measures, such as previous code participation, that could function as a proxy for expected performance. Still, the contrast between the relatively large range of average session scores and relatively small range of individual rater scores within each session suggests that the tool is able to discriminate effectively between sessions. It is not unreasonable to assume that these differences at least partially correspond to real performance variance.

Examination of the gap analysis data revealed a high number of competencies in which team self-perception differed significantly from course faculty. This is not surprising given the reported unreliability of self-perception and is consistent with previously reported material regarding gap analysis in a medical context.16,31,32 Although some gaps may be attributable to subjective factors such as differing perceptions or perspectives regarding the session, the TPDSCI assesses most competencies using objective, behavioral criteria, and thus most gaps likely represent truly inaccurate participant self-evaluation. Most (86%, 38/44) sessions had at least some degree of self-overappraisal, indicating that participants felt that they performed significantly better than they actually did, a concerning finding given the high stakes nature of pediatric resuscitations. A more worrisome observation is that 31% (43/136) of gaps indicating self-overappraisal occurred in competency domains rated as needing improvement, a combination indicating an area of weakness not recognized by the team. Such areas need focused intervention. This finding illustrates the value of gap analysis, as its use enables faculty raters to uncover “blind spots” and address them in a way that will enhance participant self-reflection.

One potential limitation is the small number of raters (3) per session. However many simulation-based CRM programs will likely not have more than 3 faculty members present for any given session. Although sample size is a possible limitation, our subject number is equivalent to or greater than those of comparable pediatric CRM assessment tool validation studies.7–10,13 Finally, we could not assess construct validity statistically and our observations are suggestive at best. Further study will be needed to examine this aspect of the tool.

Multirater feedback with gap analysis is a robust means for the assessment of team competence and self-appraisal in the context of simulation-based pediatric CRM training. The TPDSCI has demonstrated good internal consistency and overall interrater reliability. Good interrater reliability has also been demonstrated for all component competencies except professionalism. Data are encouraging with respect to construct validity, but significant further work will be needed to demonstrate this statistically. The use of gap analysis can further delineate the competency-specific self-perception of the resuscitation team, allowing for enhanced self-reflection and enabling faculty to intervene in areas in which inadequate self-appraisal coincides with a need for improvement.