The Nightmares Course: A Longitudinal, Multidisciplinary, Simulation-Based Curriculum to Train and Assess Resident Competence in Resuscitation

Due to increased clinical responsibilities and expectations, the transition from undergraduate to postgraduate medical education can be a challenging time for resident physicians. Junior residents are often the first physicians to attend to inpatients with acute illnesses. With the movement toward competency-based medical education, there is an increasing mandate to ensure that residents demonstrate the knowledge, skills, and attitudes required to progress through the stages of training.¹  Patient resuscitation requires specific skills and can be a daunting task for junior residents. A recent meta-analysis of 182 publications endorsed the utility of simulation in resuscitation curricula.²  A second meta-analysis found that 67% of transition to residency curricula provided formative feedback to residents, but no programs provided a summative assessment.³  It would be beneficial for postgraduate training programs to establish a robust process for training in and assessment of minimum competencies in resuscitation early in residency.

This article describes and evaluates the Nightmares Course at Queen's University in Kingston, Ontario, Canada. The course is a novel, competency-based, transitional curriculum in resuscitation medicine that uses distributed and deliberate practice facilitated by high-fidelity simulation over a 6-month period. It also provides a summative assessment in the form of an entrustment decision⁴  on the course entrustable professional activity (EPA) common to junior postgraduate residents from several specialties, along with remediation for residents not meeting the competency threshold.

Course participants in the 2015–2016 academic year included residents at the foundations of discipline level of training at an academic teaching hospital. This stage roughly corresponds to the level of residents in their first or second postgraduate year, where a breadth of foundational abilities is acquired prior to advancing to increasingly specialized training.¹  Residents from internal medicine, orthopedic surgery, general surgery, anesthesiology, obstetrics and gynecology, urology, ophthalmology, and neurology participated in the course.

An EPA common to several specialty training programs was defined with expert consensus input from 2 physicians with expertise in resuscitation (A.K.H. and T.C.; table 1).

Nightmares Course sessions were run on a weekly basis from August 2015 through February 2016. Over this period, residents self-scheduled 1 session every 4 weeks using an online system. Each session was held in the simulation laboratory at the Kingston Resuscitation Institute within Kingston General Hospital, and 2 simulation mannequins were used: SimMan 3G and SimMan Essential (Laerdal, Toronto, Canada).

Each 90-minute session was attended by 4 to 6 residents and involved 3 simulated scenarios. Topics for each session were chosen based on collected data from the most common calls to our hospital's Rapid Assessment of Critical Events team for acutely unwell admitted patients. A structured debrief⁵  with formative feedback followed each scenario. The sessions were taught by 8 attending physicians with an interest in resuscitation medicine and simulation-based instruction from a variety of specialties (emergency medicine, critical care medicine, anesthesiology, and general internal medicine). Faculty participation was rewarded with acknowledgment via a points-based reward structure from our department of postgraduate medicine.

Approval to perform this study was obtained through the Research Ethics Board at Kingston General Hospital.

We used a multi-station objective structured clinical examination (OSCE) on March 1, 2016. The OSCE included 4 independent simulation scenarios assessed by 4 attending physician examiners from a variety of specialties (emergency medicine, anesthesiology, general surgery, and general internal medicine). The examiners were not blinded to the identity of the residents because all were already instructors for the Nightmares Course. The residency programs encouraged all residents to participate in the OSCE, but it was not mandatory. The total OSCE time per resident was 60 minutes, about 10 minutes per station and 3 minutes to transition. Residents completed a brief questionnaire following the OSCE.

Stations were mapped by the study authors (L.M., A.K.H., T.C.) to the core 18 scenarios of the Nightmares Course via a blueprinting process⁶  (provided as online supplemental material), and 7 experts with emergency medicine and critical care medicine backgrounds reviewed the stations for content validity. All stations were rehearsed with nursing staff and simulation technicians 1 day prior to the OSCE to ensure a standardized experience.

The Queen's Simulation Assessment Tool (QSAT), an anchored global assessment scoring tool, was modified for use in this setting (provided as online supplemental material). A validity argument for the use of the QSAT in simulated resuscitations has been previously presented.^7,8  Modifications for the Nightmares Course OSCE included adding an EPA to anchor examiners and using an entrustment decision⁴  instead of the previously utilized global assessment scale. All examiners underwent a 30-minute training session led by 1 study investigator (A.K.H.).

Considering the expected level of independence of first-year residents, the competency threshold was set at 3 out of 5 on all stations. This indicates an entrustment decision of indirect supervision required (table 2), and was determined a priori by expert consensus with input from 2 authors (A.K.H. and T.C.). While individual QSAT domain scores were completed by examiners, these scores did not contribute to the summative competency decision.

A generalizability (G) study was conducted to determine the stability of residents' entrustment scores and the magnitude of the identified sources of variance. With separate examiners for each station, our analysis was limited to a 2-facet, person-by-item design, with item denoting station/examiner. The G-study was performed using SPSS version 22 (IBM Corp, Armonk, NY), and confirmed independently by a second researcher using GENOVA (generalized analysis of variance). Phi coefficients were calculated as a measure of score dependability, and a decision study was conducted.

A total of 40 residents participated in the Nightmares Course in the 2015–2016 academic year. Our hospital's Rapid Assessment of Critical Events team database identified the following as the most common reasons for calls to inpatient units: decreased level of consciousness, shortness of breath or hypoxia, tachycardia, hypotension, and clinical concern on the part of the patient's nurse. After reviewing this list, study authors (L.M. and T.C.) generated 6 session topics that formed the Nightmares Course curriculum (provided as online supplemental material).

Six 4-week sessions were held between July 2015 and March 2016. The majority of residents participated in at least 4 of 6 sessions prior to the OSCE in March (mean = 4.25; range, 0–6). All program directors supported the Nightmares Course and considered it protected academic time for their residents. The primary reason for absence was out-of-town clinical rotations.

A total of 23 of 40 residents (58%) participating in the Nightmares Course completed the OSCE. The frequencies of entrustment scores were calculated for each OSCE station (figure). Person variance was found to be greater than item variance, but a large portion (51%) of the variance remained unaccounted for (person-by-item interaction and error; table 3). The use of 4 stations/examiners yielded acceptable dependability (phi = 0.733).

Of the 23 residents, 8 (35%) scored less than 3 on at least 1 OSCE station and therefore did not meet the competency threshold. These 8 residents participated in the subsequent remediation sessions that ran from March through June 2016. The response rate of the post-OSCE questionnaire was 23 of 23 (100%). The comments were uniformly positive; residents indicated that they found the examination to be fair, a valuable learning experience, and representative of their clinical ability.

Each weekly session cost approximately $240 for human resources, and there were a total of 44 weekly sessions, yielding a total curriculum cost of $10,560. The multi-station OSCE required 4 lab technicians and 6 registered nurses for a full 9-hour day, as well as a half-day preparatory session, contributing to a total approximate cost of $3,900 for the OSCE (table 4).

The Nightmares Course was valued by residents and faculty from all programs as it provided an opportunity to teach and assess residents using direct observation, frequent formative feedback, and a summative OSCE. Most importantly, this program identified residents requiring additional remediation and directed their further training.

Most of the literature pertaining to transitional curricula (or boot camps) in postgraduate medical education describes the preparation of residents with basic technical skills required to begin surgical programs.^9–11  Nishisaki et al¹²  described a resuscitation-focused boot camp for 22 pediatrics critical care fellows, employing a 2.5-day course with both didactic and simulation-based methods, with no assessment of residents. Pliego et al¹³  described a boot camp for 23 obstetrics and gynecology residents involving 4 high-fidelity simulation scenarios with no assessment, demonstrating improved perceived confidence. There are several examples of crisis resource management curricula; however, these courses are discipline specific, provide mainly formative feedback, and are not described as transitional curricula.^14,15 

Several attributes make the Nightmares Course a novel intervention. It is a longitudinal course that takes place over the academic year, and it reiterates a standard approach to acutely unwell patients. Repetitive practice distributed over time is recognized as an important ingredient for learning.¹⁶  Most transitional curricula are short in duration, with the longest reported boot camp in a recent meta-analysis taking place over a 7-week period.³  The course also provides a summative assessment in the form of an entrustment decision on the course EPA, which is not reported in any of the published curricula as previously mentioned.³  The course is interdisciplinary and addresses an EPA common to residents providing inpatient care. Finally, remediation is built into the curriculum by design, with the infrastructure in place to continue weekly remedial sessions following the summative OSCE.

The G-study results for the entrustment scores in our OSCE demonstrate that resident variance (41%) was much larger than station/examiner variance (8%), indicating that the scores were sensitive in differentiating resident performance. However, there was a large portion of unaccounted variance, suggesting that additional factors were contributing errors to the performance rating. Separating examiners from stations to develop a fully crossed design would help to account for a larger percentage of the observed variance.

The course was well attended and rated highly by residents. With 8 faculty contributing to weekly sessions, the teaching burden was manageable, and all faculty volunteered to continue teaching the course in the upcoming year.

There are several limitations. The cost for the Nightmares Course and the OSCE was substantial, estimated at $14,760. Participation in the OSCE was optional and low, and this may have biased the results in favor of competent performance; the small number of OSCE stations was also a limitation. The use of nonblinded examiners may have resulted in potential assessment bias. Finally, the use of a single entrustment score to inform the summative assessment decision, instead of individual domain scores, could be viewed as a limitation in the assessment process.

Future work will be required to ensure that competency is translated into the hospital setting and, ultimately, improved patient-level outcomes. Moving forward, we will also evaluate the implementation and effectiveness of similar curricular designs at other centers for similar EPAs.

The Nightmares Course is a novel, competency-based, educational intervention and assessment program in resuscitation medicine applicable to a majority of postgraduate medical training programs. This course was feasible and supported by residents and faculty at our institution. The entrustment-based assessment detected several residents not meeting a minimum competency threshold, and directed their further remediation.