Musculoskeletal problems are the reason for one-quarter of primary care visits. Opportunities for internal medicine residents to perform joint aspirations and injections have declined. Simulation has been shown to improve procedure skills post completion of simulation courses, yet controversy exists about the durability of simulation-acquired skills.
To investigate whether web-based review preserves residents' joint procedure skills 6 to 30 months after a simulation course.
Postgraduate year–1 internal medicine residents participated in a simulation-based Joint Aspiration Injection Course consisting of web-based instructional material, guided practice on joint models, and a multiple-choice test. Procedure proficiency was scored by using a 3-component skills checklist. Six to 30 months later, residents who had participated in the simulation were randomly assigned to review or not to review the original web-based instructional material before retesting. The groups were compared by using Wilcoxon rank sum and matched pairs signed rank tests.
Compared to the performance at the end of the simulation course, scores of all 3 procedure components declined (informed consent, 64.7–43.0 versus 30.6–23.8, P < .001; procedure proficiency, 63.4–61.7 versus 46.4–44.3, P < .001; and postprocedure instructions, 58.0–54.1 versus 29.9–29.4, P < .001). However, the review group outperformed the nonreview group on informed consent (shoulder: 37.1 versus 24.6, P = .01) and postprocedure instructions (shoulder: 34.0 versus 25.2, P = .01; knee: 35.5 versus 24.8, P < .001). Residents who reported doing actual procedures maintained a higher confidence level, compared with those reporting none (6.8–5.1 versus 4.1–3.6, P < .001).
Shoulder and knee simulation-acquired skills declined 6 to 30 months after a simulation course. However, rereview of web-based instructional material improved proficiency in informed consent, shoulder, and postprocedure instructions, shoulder and knee.
Musculoskeletal problems are the reason for one-fourth of the visits to primary care physicians, yet internal medicine residents have relatively few opportunities to perform joint aspirations and injections.
Internal medicine interns completed a simulation-based joint aspiration injection course. They were subsequently randomly assigned to review or not to review the original web-based instructional material and were retested.
Small sample, single-site study; utility of checklist items for reassessment was reduced because a substantial number of residents obtained “perfect” scores.
Skills developed during the simulation-based joint aspiration injection course declined in the 6- to 30-month period before reassessment. Rereview of the web-based materials improved proficiency in informed consent and postprocedure instructions.
Patients with musculoskeletal problems constitute about a quarter of primary care visits.1 The 2006 American College of Physicians Survey not only reaffirmed the leading role of joint aspirations and injections among procedures performed by internists in their practice, but also revealed a significant decline in the number of internists performing them.2
Inadequate training and a low comfort level in performing these procedures have led primary care physicians to underuse intra-articular and periarticular injections.3 Owing to the decreasing length of office visits, resident work hour restrictions, and concern for patient safety, residents learn and perform fewer office procedures.4
Simulation-based learning provides a low risk, predictable, and reproducible environment that has been embraced as a solution to teach learners new procedures without compromising patient safety.5,6 Many studies,7–11 including a few that specifically examined teaching arthrocentesis and intra-articular injection techniques, report preintervention to postintervention increases in knowledge, skill, and confidence levels of the participants. Controversy exists about the durability of simulation-acquired skills. Most studies, with a few exceptions,12,13 report significant skill attrition by 6 months or earlier.14–16
To date, only 1 study17 has assessed the self-reported comfort level of residents 10 months after a joint model injection workshop.
Our study investigated resident retention of simulation-acquired skills 6 to 30 months after completing a simulation-based knee and shoulder aspiration and injection course, and whether prior review of web-based instructional material would mitigate skill attrition and instead improve procedure performance. We hypothesized that prior review of web-based instructional material would improve procedure performance.
Beginning in October 2006, the postgraduate year (PGY)–1 internal medicine residents at the University of Pittsburgh participated in a simulation-based Joint Aspiration Injection (JAI) Course as part of their PGY-1 ambulatory rotation. It consisted of 2 sessions, one dedicated to the shoulder and the second to the knee. One instructor taught 1 to 4 residents per session. Using course-specific web-based content, the instructor reviewed shoulder and knee anatomy; the differential diagnosis and examination of a painful shoulder and knee; the indications, contraindications, and complications of joint injections; handling of joint fluid; informed consent; and postprocedure instructions. The instructor then demonstrated the proper injection technique on realistic joint models. The shoulder model used a visual cue for correct needle placement (ie, a light turned green when the needle entered the correct spot on the model), and the knee model allowed aspiration of synthetic joint fluid. Residents had an unlimited number of practice attempts with immediate faculty feedback. At the conclusion of the first session, each resident was tested on his/her shoulder procedure skills. Using the model as a mock patient, each resident orally obtained informed consent, performed the injection, and orally provided postprocedure instructions to the patient. The residents repeated this process till they reached proficiency at a composite score of 170.
The second session was focused on the knee. The process was similar to that of the shoulder, but the cutoff score for the knee was 150. The session also included a multiple-choice test, which consisted of 12 vignette questions examining both the shoulder and knee content.
The faculty instructor documented the residents' performance by using a skills evaluation checklist. The checklist consisted of 3 components: informed consent, procedure proficiency, and postprocedure instructions. Each component had multiple items, and each item had 3 choices of (1) performed correctly, (2) performed incorrectly, and (3) did not perform. A score of 10, 5, or 0 was assigned to the choices, resulting in a separate sum for each of the 3 components.
Residents were also asked to rate their procedure skill confidence level (“I am fully confident that I can perform a…”) on a 9-point Likert scale before and after the course (1 = strongly disagree, 3 = disagree, 5 = neutral, 7 = agree, 9 = strongly agree).
In 2009, 6 to 30 months following the simulation course, the original 44 PGY-1 residents (now PGY-2 and PGY-3 residents) were randomly assigned to either review or not to review the original web-based instructional material before being retested. One-half of the group (JAI-1) was notified and given access to the web-based material in order to prepare for the testing, while the other half of the residents (JAI-2) were neither notified nor given access. All were resurveyed about their current procedure performance confidence levels and the total number of actual shoulder and knee procedures they had completed since the initial simulation session. The same precourse confidence Likert scale, skills evaluation checklist, joint models, and multiple-choice questions were used. A single faculty member, who was blinded to the 2 groups, evaluated the residents' skills by using the same methods as in the original PGY-1 JAI course, as detailed above.
The study was approved both by the University of Pittsburgh and VA Healthcare System Institutional Review Boards.
Descriptive statistics of mean, median, and standard deviation were computed for all scores and confidence levels. Changes in outcomes between the PGY-1 survey and the PGY-2/3 survey were assessed with the Wilcoxon matched pairs signed rank test. Comparisons of scores and confidence levels between the JAI-1 and JAI-2 groups were examined via Wilcoxon rank sum test. The differences of proportions of correct answers were tested with the McNemar test for the PGY-1 and PGY-2/3 residents, and the Fisher exact test for the JAI-1 and JAI-2 groups.
The overall significance level for tests was retained as α = .05 and all values were given as mean ± SD (median) or proportion (%) of scores. All statistical analyses were done with STATA10 (2008; StataCorp LP, College Station, TX).
Performance for all 3 components (informed consent, procedure proficiency, and postprocedure instructions), involving both the knee and shoulder models, decreased upon retesting when compared with the results at the end of the original PGY-1 course (table 1). A subgroup analysis of residents retested at 6 to 18 versus 18 to 30 months showed a similar decline in performance and no difference between the 2 groups (table 2). The distribution of when all PGY-2 and PGY-3 residents were retested is shown in table 3.
All 3 components of the skills evaluation checklist, for both shoulder and knee, were examined for internal consistency by using Cronbach α (table 4). The shoulder informed consent and postprocedure instructions showed moderate internal consistency (0.5–0.8). The knee informed consent, procedure proficiency, and postprocedure instructions just reached moderate internal consistency (0.5–0.6), but for the review group only.
All the PGY-2 and PGY-3 residents repeated the original 12-vignette multiple-choice test. Residents' test scores decreased significantly after the 6- to 30-month time frame, as compared to their PGY-1 scores at the completion of the simulation course (76% versus 90%, respectively; P < .001). There was no difference between the review and nonreview groups (77% and 74%, respectively, P = .56).
The resident group randomly assigned to review the web-based instructional material before retesting (JAI-1) outperformed the nonreview group (JAI-2) on informed consent for the shoulder and postprocedure instructions for both the shoulder and the knee (table 1).
Residents were also asked to rate their procedure skill confidence level on a 9-point Likert scale. The confidence levels reported by the PGY-1 group before the simulation course were significantly lower than those at the retesting preclass survey 6 to 30 months later. However, the confidence levels of the PGY-1 group at the completion of the simulation course were significantly higher than the confidence levels of the PGY-2 and PGY-3 groups (table 5). In contrast to the shoulder procedure, the confidence levels in performing the knee procedure were better preserved, remaining above the neutral category on the Likert scale.
Thirty residents reported performing no actual shoulder injections before retesting, while 11 reported doing 1 to 5 injections. With respect to knee injections, 8 residents reported performing none, while 32 reported having done 1 to 5, and 2 reported having done 6 to 10 injections. Residents who reported doing actual procedures showed the same decline in skills (table 6) but maintained a higher confidence level at retesting than those who reported doing none (table 7).
Simulation-based learning has become a well-established tool that increases learner confidence and skill immediately after a simulation course, although controversy persists as to the durability of simulation-acquired skills. Our study assessed the durability of shoulder and knee aspiration and injection skills, and demonstrated that the residents' overall procedure proficiency declined after 6 to 30 months. We also examined the impact of prior review of web-based instructional material, which was predominantly textual, on the preservation of these skills. Although prior review did not maintain technical competence, it did improve proficiency with the verbal portions of the procedure: shoulder informed consent and shoulder and knee postprocedure instructions. We believe that if the web-based content had included a procedure video instead of predominantly textual information, it might have also improved the technical or procedure competence. The shoulder informed consent contained more items than the one for the knee. These additional items included uncommon complications associated more frequently with the shoulder. It is more likely that residents in the review group included these than residents in the nonreview group.
Two major studies demonstrate preservation of simulation-acquired skills beyond 6 months. Wayne et al13 document no decay of medical residents' advanced cardiac life support skills at 6 and at 14 months. Their residents, after baseline testing, were randomly assigned to either 4 weekly 2-hour simulation teaching sessions or no sessions for 3 months. After 3 months, both groups were retested and then crossed over, to wait 3 months or to participate in the 4 weekly 2-hour teaching sessions. All were retested at 6 and 14 months. Although our curriculum used deliberate practice,19 as did Wayne et al, we differed in the use of distributive practice. Our residents benefited from only 2 sessions in contrast to the 4 sessions of Wayne and colleagues. Also, our earliest retesting occurred at 6 months, in contrast to the group of Wayne et al, which was retested at 3 and 6 months before the 14-month evaluation. Testing itself has been shown to improve long-term retention (ie, the “testing effect”20). Furthermore, there is no mention of the actual number of codes run by the residents in the studies of Wayne et al, compared to the limited number of actual joint procedures done by our residents.
A second study, by Stefanidis et al,12 demonstrated the durability of simulation-acquired laparoscopic suturing skills at 6 months. Medical students were trained to proficiency by reaching an expert-derived predetermined score; deliberate practice was encouraged and overtraining built into each session. The subjects in the study by Stefanidis et al were tested at even shorter intervals than in the study of Wayne et al: at 2 weeks, 1 month, 3 months, and 6 months, thus exposing their subjects to an even higher “testing effect.” In contrast to Stefanidis et al,12 we did not incorporate overtraining into our sessions and our earliest retesting interval was at 6 months, with most of our residents being tested even further from their initial training session.
Studies are mixed as to the reliability of self-reported confidence levels, with a few even showing a reciprocal relationship between actual skill and self-reported confidence.7,9 Our residents' self-reported confidence levels declined in correspondence with the lack of experience in performing actual procedures. For example, confidence to perform shoulder injections (specifically, the subacromial bursa) decreased more than the confidence to perform knee injections, reflecting the fewer opportunities for residents to perform shoulder injections as compared with knee injections.
The paucity of actual intra-articular injections done during residency is unfortunately common. For example, Jolly and Curran3 demonstrated that only 11% of faculty reported performing more than 5 intra-articular injections during their residency training, although 60% of them considered 5 to 10 intra-articular injections to be optimal for competence. The optimal number of knee or shoulder injections necessary for competence or for mastery is unclear. However, our study suggests that 1 to 5 actual knee procedures is insufficient, since the skill levels did not differ between the group performing 1 to 5 procedures and the group performing none.
Our study is the first to measure the retention of simulation-acquired knee and shoulder injection skills at 6 to 30 months and to investigate whether prior web-based review is sufficient to preserve these skills; our sample size of 44 residents compares favorably to that of other studies in this area. It is also the first to compare confidence levels in performing these procedures beyond 10 months after a teaching session. At the same time, our study has several potential limitations. It was conducted in a single residency program at a single academic institution, limiting generalizability. The calculations of our skills checklist items' internal consistency, the Cronbach α, were low in some cases because a substantial number of our residents obtained full scores. This is a common limitation of Cronbach α calculations. Although our retesting interval was wide (ie, 6–30 months), it represents a realistic time frame during which knee and shoulder procedure opportunities arise in an internal medicine residency program.
Our study demonstrates that prior review of web-based instructional material improves the proficiency with informed consent for the shoulder and postprocedure instructions for knee and shoulder. There also seems to be a congruent decline between the number of procedures performed and the self-reported confidence level. Future research should focus on identifying what factor or combination of factors, including dose learning, overtraining, feedback, retraining intervals, or other methods, are the most critical for preservation of simulation-acquired skills.
Ruth Preisner, MD, is Associate Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; Harish Jasti, MD, is Assistant Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; Michael Elnicki, MD, is Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; and Kwonho Jeong, BA, is in the Epidemiology Data Center at the University of Pittsburgh.
Funding: The authors report no external funding source for this study.
The authors wish to thank John Lutz, Director of Information Technology, and Peter M. Winter, Institute for Simulation, Education, and Research.