Perceptions on the Impact of a Just-in-Time Room on Trainees and Supervising Physicians in a Pediatric Emergency Department

Increased supervision and duty hour restrictions have led to fewer clinical opportunities for trainees. Concurrently, the Accreditation Council for Graduate Medical Education has developed milestones for residents, which include procedural or technical skills under the core competency of patient care. Just-in-time (JIT) procedural training allows for practice of a skill immediately prior to performing a procedure in an effort to maximize trainee learning, confidence, and patient safety. While similar to in situ simulation in its geographic proximity to procedural performance, JIT training adds temporal proximity as well.

A number of studies^1–9  have examined JIT training, yet there are no known studies examining the use of dedicated JIT space to practice common procedures near the clinical space, rather than a simulation center. Simulation-based education, as in the JIT room, can provide a reproducible standard environment for deliberate practice as well as both formative and summative assessment.¹⁰  Gaining a better understanding of learners' perspectives can assist educators in the implementation of a specific simulation curriculum.³ 

Seattle Children's Hospital opened a new emergency department (ED) in April 2013 that includes a JIT training room, conveniently located within the ED clinical space, and is available 24 hours a day for procedural education in splinting, suturing, and lumbar puncture (LP). This room is used daily for physician and nurse education, but is always available for JIT training. This was thought to be a cost-effective approach. Location within the ED clinical space maximizes accessibility to trainees working clinical shifts and allows for deliberate practice, a structured repetitive training designed to acquire or improve a critical aspect of performance for a specific level of performance or mastery.^8,9 

We sought to examine the impact of the JIT room on resident and supervisor procedural experiences in the ED. We hypothesized that the JIT room would be a convenient, cost-effective space for preprocedural training that would potentially improve trainee self-confidence and supervisor confidence in the trainee, and allow for less supervisor intervention during procedures.

We conducted a cross-sectional survey study using a convenience sample of trainees and supervising physicians performed at a tertiary care pediatric hospital, including family medicine, emergency medicine (EM), and pediatrics residents who worked in the ED, as well as pediatric emergency medicine (PEM) attending physicians and fellows. Medical students and visiting residents were excluded.

Two surveys were administered: 1 to trainees and 1 to PEM supervising physicians. The surveys were developed using expert consultation in the fields of education and PEM. They were pretested by various PEM faculty members at the study institution for content and ease of use. Demographic survey data included years of training/years of posttraining, specialty, prior experience with JIT training, and baseline comfort with suturing, splinting, and LP. Using 5-point Likert scales and divided time percentages, the surveys queried use of the JIT room, subsequent confidence in procedure performance, and supervisor intervention during procedures. Open-ended questions about the JIT room included barriers to use, improvement recommendations, and procedures to add.

The anonymous surveys were disseminated using Research Electronic Data Capture, or REDCap (Vanderbilt University, Nashville, TN), using an e-mail link sent to eligible subjects. The survey was open from June 6 through July 11, 2014. A maximum of 8 reminders were e-mailed inviting subjects to complete the surveys.

The study was granted a review exemption by the Seattle Children's Hospital Institutional Review Board.

Demographic characteristics and JIT room use were summarized. Paired t tests compared responses to contrasting questions to assess use of the JIT room based on trainee self-reported competence, whether usage was associated with higher trainee perceived confidence or trainee procedural skills, and if usage was related to supervisor intervention during procedures. For each t test, the paired difference, its 95% confidence interval, and the P value are reported. P values below .05 were considered significant. Adjustments were not made for multiple testing, as this is an exploratory study. Analyses were performed using Stata version 12 (StataCorp LP, College Station, TX) and Microsoft Excel 2010 (Microsoft Corp, Redmond, WA). Subsequent to the survey, JIT room material costs were estimated as a means of describing generalized feasibility.

Thirty-one of 32 supervising physicians (97%) and 122 of 186 residents (66%) completed the survey. Table 1 shows response rates by resident specialty.

Residents reported that the JIT room is often used if they feel insufficiently skilled in a procedure (figure 1). Residents felt more proficient when using the JIT room (mean = 4.0 on a Likert scale of 1 to 5) than when not using the JIT room (mean = 2.6; P < .05; paired difference +1.4; table 2). Trainees reported that JIT room use has little effect on supervisor decision to procedurally intervene (mean score = 2.6 for both; P = .30; paired difference −0.04; table 2).

Among supervisors, 94% (29 of 31) reported that they were more likely to use JIT training for procedural practice with the JIT room within the clinical ED space. All supervisors noted that they use the JIT room if a trainee states that he or she is not proficient in a procedure, but only 19% (6 of 31) would do so if a trainee states that he or she is procedurally proficient. The JIT room is most frequently used to teach suture repair and LP skills and less frequently used to teach splinting: 32% (10 of 31) of supervising physicians used the JIT room more than 10 times to teach suturing, 32% (10 of 31) used it more than 10 times to teach LP, and only 3% (1 of 31) of supervisors used it more than 10 times to teach splinting (figure 2).

Supervisors strongly agreed that residents feel more proficient if they use the JIT room (mean = 4.7 on a Likert scale of 1–5) and were neutral on residents' feelings of proficiency when they do not use the JIT room (mean = 2.8 on a Likert scale of 1–5; P < .05; paired difference +1.8; table 2). Supervisors were more inclined to intervene if the JIT room was not used (P < .05; paired difference −0.4; table 2).

Lack of time was reported as a major barrier to JIT room use. While most felt that there was not anything to be improved in the JIT room, some respondents suggested standardizing regular use of the room and scheduled supply restocking. Suggestions for added procedures to the JIT room included central and peripheral line placement, intubation, foreign body removal, bladder catheterization, gastric tube replacement, ultrasound, incision and drainage, chest tube placement, and tracheostomy tube replacement.

The JIT room was conceptualized early in the planning of the new ED to allow for efficient JIT education of trainees prior to procedures within the ED. Suture, splinting, and LP material is recycled and reused. See table 3 for cost estimation of materials. Ongoing costs for the space are part of ED overhead.

Both residents and supervisors reported improved trainee skills and confidence with JIT room use. In addition, supervisors reported that they intervene less in procedures after JIT room use; however, there was no difference seen in supervising faculty intervention in procedures as perceived by trainees.

The educational theory of situated learning proposes that effective learning occurs through workplace context and experiential participation,¹¹  allowing learners and teachers to offer unique perspectives on educational interventions. The JIT room's location in the clinical space creates an optimally situated learning environment, as travel to a simulation center is not possible during a clinical shift. Use of a JIT room located within the clinical space can occur between tasks; however, learners must make time to utilize the room and balance clinical demands.

While JIT learning is well grounded in educational theory, studies of the effect of JIT teaching have produced mixed results. In a study of JIT training in infant LP,¹²  frequency of procedural success was not affected by JIT training. Other markers of procedural proficiency, however, such as accurate initial placement of central lines, improved after JIT practice.⁷  This may point to a contextual issue with situated cognition in that learning is inherent to the situation in which it occurs.¹³  A dedicated procedural training space such as a JIT training room that removes the learner from a distracting clinical environment, while still physically being in the clinical space, might improve trainee procedural success by providing a protected space for JIT deliberate practice.

A majority of residents and supervising physicians felt that trainee procedural confidence and competence improved after use of the JIT room. Trainees did not feel that use of the JIT room reduced supervising physician intervention. In contrast, supervising physicians felt that use of the JIT room reduced their interventions in trainee-conducted procedures. This difference may suggest that supervising physicians perform an informal summative assessment of trainees in the JIT room and therefore believe they intervene less in trainee procedures. Trainees may not perceive this change. Additionally, given their limited procedural opportunities, trainees may not have the perspective to sense a difference in physician supervisor intervention.

The JIT room is primarily utilized for skill training and formative assessment. Trainees practice their procedural skills in the space, but are not explicitly required to demonstrate a certain level of competence before performing a procedure.

Trainee and supervising physicians reported that lack of time was the major barrier to use of the JIT room. Another study¹  found this to be a similar barrier for JIT infant LP trainer use, as reported by pediatrics interns. Standardizing the expectation that the room should be used for summative assessment prior to performing a procedure on a real patient may transform use of the room into a standard preprocedure step.

Limitations of this study include that it was a single center study, it used a convenience sample, and the results may not generalize to other settings. Our survey was developed by knowledgeable faculty, but lacks validity evidence, and respondents may not have interpreted questions as intended. The ED at which this study took place is heavily staffed by pediatrics and family medicine residents (more so than EM residents), as reflected in the distribution of trainee survey respondents. Given that these learners inherently spent less time in the ED, family medicine and pediatrics residents may be less confident in their procedural skills than EM residents, potentially making them more likely to use the JIT room. The study's reliance on self-report makes it susceptible to recall and reporting bias. We attempted to avoid reporting bias by keeping surveys anonymous and protected from program leaders. Additionally, survey methodology did not allow us to measure definitive endpoints such as patient satisfaction or procedure outcome. Lastly, the JIT room was built with a new ED, so translating this space to other institutions may not be feasible. However, existing space proximate to the ED could be modified to include dedicated JIT training space.

Future research should include evaluation of the JIT room use relative to the experience level of the trainee, as well as explore the effects of summative JIT assessment on procedural performance and, ultimately, quality and safety of patient care.

A dedicated room near the clinical space may be an efficient, time-saving method to provide JIT education, potentially improve trainee skills and confidence, and decrease supervisor intervention in procedures.