Abstract

Background

Musculoskeletal problems are the reason for one-quarter of primary care visits. Opportunities for internal medicine residents to perform joint aspirations and injections have declined. Simulation has been shown to improve procedure skills post completion of simulation courses, yet controversy exists about the durability of simulation-acquired skills.

Objective

To investigate whether web-based review preserves residents' joint procedure skills 6 to 30 months after a simulation course.

Methods

Postgraduate year–1 internal medicine residents participated in a simulation-based Joint Aspiration Injection Course consisting of web-based instructional material, guided practice on joint models, and a multiple-choice test. Procedure proficiency was scored by using a 3-component skills checklist. Six to 30 months later, residents who had participated in the simulation were randomly assigned to review or not to review the original web-based instructional material before retesting. The groups were compared by using Wilcoxon rank sum and matched pairs signed rank tests.

Results

Compared to the performance at the end of the simulation course, scores of all 3 procedure components declined (informed consent, 64.7–43.0 versus 30.6–23.8, P < .001; procedure proficiency, 63.4–61.7 versus 46.4–44.3, P < .001; and postprocedure instructions, 58.0–54.1 versus 29.9–29.4, P < .001). However, the review group outperformed the nonreview group on informed consent (shoulder: 37.1 versus 24.6, P  =  .01) and postprocedure instructions (shoulder: 34.0 versus 25.2, P  =  .01; knee: 35.5 versus 24.8, P < .001). Residents who reported doing actual procedures maintained a higher confidence level, compared with those reporting none (6.8–5.1 versus 4.1–3.6, P < .001).

Conclusion

Shoulder and knee simulation-acquired skills declined 6 to 30 months after a simulation course. However, rereview of web-based instructional material improved proficiency in informed consent, shoulder, and postprocedure instructions, shoulder and knee.

What was known

Musculoskeletal problems are the reason for one-fourth of the visits to primary care physicians, yet internal medicine residents have relatively few opportunities to perform joint aspirations and injections.

What is new

Internal medicine interns completed a simulation-based joint aspiration injection course. They were subsequently randomly assigned to review or not to review the original web-based instructional material and were retested.

Limitations

Small sample, single-site study; utility of checklist items for reassessment was reduced because a substantial number of residents obtained “perfect” scores.

Bottom line

Skills developed during the simulation-based joint aspiration injection course declined in the 6- to 30-month period before reassessment. Rereview of the web-based materials improved proficiency in informed consent and postprocedure instructions.

Introduction

Patients with musculoskeletal problems constitute about a quarter of primary care visits.1 The 2006 American College of Physicians Survey not only reaffirmed the leading role of joint aspirations and injections among procedures performed by internists in their practice, but also revealed a significant decline in the number of internists performing them.2 

Inadequate training and a low comfort level in performing these procedures have led primary care physicians to underuse intra-articular and periarticular injections.3 Owing to the decreasing length of office visits, resident work hour restrictions, and concern for patient safety, residents learn and perform fewer office procedures.4 

Simulation-based learning provides a low risk, predictable, and reproducible environment that has been embraced as a solution to teach learners new procedures without compromising patient safety.5,6 Many studies,711 including a few that specifically examined teaching arthrocentesis and intra-articular injection techniques, report preintervention to postintervention increases in knowledge, skill, and confidence levels of the participants. Controversy exists about the durability of simulation-acquired skills. Most studies, with a few exceptions,12,13 report significant skill attrition by 6 months or earlier.1416 

To date, only 1 study17 has assessed the self-reported comfort level of residents 10 months after a joint model injection workshop.

Our study investigated resident retention of simulation-acquired skills 6 to 30 months after completing a simulation-based knee and shoulder aspiration and injection course, and whether prior review of web-based instructional material would mitigate skill attrition and instead improve procedure performance. We hypothesized that prior review of web-based instructional material would improve procedure performance.

Methods

Beginning in October 2006, the postgraduate year (PGY)–1 internal medicine residents at the University of Pittsburgh participated in a simulation-based Joint Aspiration Injection (JAI) Course as part of their PGY-1 ambulatory rotation. It consisted of 2 sessions, one dedicated to the shoulder and the second to the knee. One instructor taught 1 to 4 residents per session. Using course-specific web-based content, the instructor reviewed shoulder and knee anatomy; the differential diagnosis and examination of a painful shoulder and knee; the indications, contraindications, and complications of joint injections; handling of joint fluid; informed consent; and postprocedure instructions. The instructor then demonstrated the proper injection technique on realistic joint models. The shoulder model used a visual cue for correct needle placement (ie, a light turned green when the needle entered the correct spot on the model), and the knee model allowed aspiration of synthetic joint fluid. Residents had an unlimited number of practice attempts with immediate faculty feedback. At the conclusion of the first session, each resident was tested on his/her shoulder procedure skills. Using the model as a mock patient, each resident orally obtained informed consent, performed the injection, and orally provided postprocedure instructions to the patient. The residents repeated this process till they reached proficiency at a composite score of 170.

The second session was focused on the knee. The process was similar to that of the shoulder, but the cutoff score for the knee was 150. The session also included a multiple-choice test, which consisted of 12 vignette questions examining both the shoulder and knee content.

The faculty instructor documented the residents' performance by using a skills evaluation checklist. The checklist consisted of 3 components: informed consent, procedure proficiency, and postprocedure instructions. Each component had multiple items, and each item had 3 choices of (1) performed correctly, (2) performed incorrectly, and (3) did not perform. A score of 10, 5, or 0 was assigned to the choices, resulting in a separate sum for each of the 3 components.

Residents were also asked to rate their procedure skill confidence level (“I am fully confident that I can perform a…”) on a 9-point Likert scale before and after the course (1  =  strongly disagree, 3  =  disagree, 5  =  neutral, 7  =  agree, 9  =  strongly agree).

In 2009, 6 to 30 months following the simulation course, the original 44 PGY-1 residents (now PGY-2 and PGY-3 residents) were randomly assigned to either review or not to review the original web-based instructional material before being retested. One-half of the group (JAI-1) was notified and given access to the web-based material in order to prepare for the testing, while the other half of the residents (JAI-2) were neither notified nor given access. All were resurveyed about their current procedure performance confidence levels and the total number of actual shoulder and knee procedures they had completed since the initial simulation session. The same precourse confidence Likert scale, skills evaluation checklist, joint models, and multiple-choice questions were used. A single faculty member, who was blinded to the 2 groups, evaluated the residents' skills by using the same methods as in the original PGY-1 JAI course, as detailed above.

The study was approved both by the University of Pittsburgh and VA Healthcare System Institutional Review Boards.

Statistical Analysis

Descriptive statistics of mean, median, and standard deviation were computed for all scores and confidence levels. Changes in outcomes between the PGY-1 survey and the PGY-2/3 survey were assessed with the Wilcoxon matched pairs signed rank test. Comparisons of scores and confidence levels between the JAI-1 and JAI-2 groups were examined via Wilcoxon rank sum test. The differences of proportions of correct answers were tested with the McNemar test for the PGY-1 and PGY-2/3 residents, and the Fisher exact test for the JAI-1 and JAI-2 groups.

The overall significance level for tests was retained as α  =  .05 and all values were given as mean ± SD (median) or proportion (%) of scores. All statistical analyses were done with STATA10 (2008; StataCorp LP, College Station, TX).

Results

Performance for all 3 components (informed consent, procedure proficiency, and postprocedure instructions), involving both the knee and shoulder models, decreased upon retesting when compared with the results at the end of the original PGY-1 course (table 1). A subgroup analysis of residents retested at 6 to 18 versus 18 to 30 months showed a similar decline in performance and no difference between the 2 groups (table 2). The distribution of when all PGY-2 and PGY-3 residents were retested is shown in table 3.

TABLE 1

Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 30 Months, With and Without Web-Based Review

Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 30 Months, With and Without Web-Based Review
Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 30 Months, With and Without Web-Based Review
TABLE 2

Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 18 Versus 18 to 30 Months

Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 18 Versus 18 to 30 Months
Shoulder and Knee Procedure Skills at Completion of Simulation Course and at 6 to 18 Versus 18 to 30 Months
TABLE 3

Number of PGY-2/3 Residents per Retesting Interval in Review, Nonreview, and Combined Groups

Number of PGY-2/3 Residents per Retesting Interval in Review, Nonreview, and Combined Groups
Number of PGY-2/3 Residents per Retesting Interval in Review, Nonreview, and Combined Groups

All 3 components of the skills evaluation checklist, for both shoulder and knee, were examined for internal consistency by using Cronbach α (table 4). The shoulder informed consent and postprocedure instructions showed moderate internal consistency (0.5–0.8). The knee informed consent, procedure proficiency, and postprocedure instructions just reached moderate internal consistency (0.5–0.6), but for the review group only.

TABLE 4

Shoulder and Knee Procedure Skill Checklist Cronbach α at Completion of Simulation Course, and at 6 to 30 Months, With and Without Web Review

Shoulder and Knee Procedure Skill Checklist Cronbach α at Completion of Simulation Course, and at 6 to 30 Months, With and Without Web Review
Shoulder and Knee Procedure Skill Checklist Cronbach α at Completion of Simulation Course, and at 6 to 30 Months, With and Without Web Review

All the PGY-2 and PGY-3 residents repeated the original 12-vignette multiple-choice test. Residents' test scores decreased significantly after the 6- to 30-month time frame, as compared to their PGY-1 scores at the completion of the simulation course (76% versus 90%, respectively; P < .001). There was no difference between the review and nonreview groups (77% and 74%, respectively, P  =  .56).

The resident group randomly assigned to review the web-based instructional material before retesting (JAI-1) outperformed the nonreview group (JAI-2) on informed consent for the shoulder and postprocedure instructions for both the shoulder and the knee (table 1).

Residents were also asked to rate their procedure skill confidence level on a 9-point Likert scale. The confidence levels reported by the PGY-1 group before the simulation course were significantly lower than those at the retesting preclass survey 6 to 30 months later. However, the confidence levels of the PGY-1 group at the completion of the simulation course were significantly higher than the confidence levels of the PGY-2 and PGY-3 groups (table 5). In contrast to the shoulder procedure, the confidence levels in performing the knee procedure were better preserved, remaining above the neutral category on the Likert scale.

TABLE 5

Confidence Level in Performing Shoulder Injections and Knee Aspirations Before, at Completion of, and 6 to 30 Months After Simulation Course

Confidence Level in Performing Shoulder Injections and Knee Aspirations Before, at Completion of, and 6 to 30 Months After Simulation Course
Confidence Level in Performing Shoulder Injections and Knee Aspirations Before, at Completion of, and 6 to 30 Months After Simulation Course

Thirty residents reported performing no actual shoulder injections before retesting, while 11 reported doing 1 to 5 injections. With respect to knee injections, 8 residents reported performing none, while 32 reported having done 1 to 5, and 2 reported having done 6 to 10 injections. Residents who reported doing actual procedures showed the same decline in skills (table 6) but maintained a higher confidence level at retesting than those who reported doing none (table 7).

TABLE 6

Shoulder and Knee Procedure Skills at 6 to 30 Months: Residents Who Reported Doing Actual Procedures Versus Residents Who Reported Doing None

Shoulder and Knee Procedure Skills at 6 to 30 Months: Residents Who Reported Doing Actual Procedures Versus Residents Who Reported Doing None
Shoulder and Knee Procedure Skills at 6 to 30 Months: Residents Who Reported Doing Actual Procedures Versus Residents Who Reported Doing None
TABLE 7

Comparison of Confidence Level Between Individuals Who Reported Performing Actual Shoulder and Knee Injections Versus Those Who Reported Performing None

Comparison of Confidence Level Between Individuals Who Reported Performing Actual Shoulder and Knee Injections Versus Those Who Reported Performing None
Comparison of Confidence Level Between Individuals Who Reported Performing Actual Shoulder and Knee Injections Versus Those Who Reported Performing None

Discussion

Simulation-based learning has become a well-established tool that increases learner confidence and skill immediately after a simulation course, although controversy persists as to the durability of simulation-acquired skills. Our study assessed the durability of shoulder and knee aspiration and injection skills, and demonstrated that the residents' overall procedure proficiency declined after 6 to 30 months. We also examined the impact of prior review of web-based instructional material, which was predominantly textual, on the preservation of these skills. Although prior review did not maintain technical competence, it did improve proficiency with the verbal portions of the procedure: shoulder informed consent and shoulder and knee postprocedure instructions. We believe that if the web-based content had included a procedure video instead of predominantly textual information, it might have also improved the technical or procedure competence. The shoulder informed consent contained more items than the one for the knee. These additional items included uncommon complications associated more frequently with the shoulder. It is more likely that residents in the review group included these than residents in the nonreview group.

Two major studies demonstrate preservation of simulation-acquired skills beyond 6 months. Wayne et al13 document no decay of medical residents' advanced cardiac life support skills at 6 and at 14 months. Their residents, after baseline testing, were randomly assigned to either 4 weekly 2-hour simulation teaching sessions or no sessions for 3 months. After 3 months, both groups were retested and then crossed over, to wait 3 months or to participate in the 4 weekly 2-hour teaching sessions. All were retested at 6 and 14 months. Although our curriculum used deliberate practice,19 as did Wayne et al, we differed in the use of distributive practice. Our residents benefited from only 2 sessions in contrast to the 4 sessions of Wayne and colleagues. Also, our earliest retesting occurred at 6 months, in contrast to the group of Wayne et al, which was retested at 3 and 6 months before the 14-month evaluation. Testing itself has been shown to improve long-term retention (ie, the “testing effect”20). Furthermore, there is no mention of the actual number of codes run by the residents in the studies of Wayne et al, compared to the limited number of actual joint procedures done by our residents.

A second study, by Stefanidis et al,12 demonstrated the durability of simulation-acquired laparoscopic suturing skills at 6 months. Medical students were trained to proficiency by reaching an expert-derived predetermined score; deliberate practice was encouraged and overtraining built into each session. The subjects in the study by Stefanidis et al were tested at even shorter intervals than in the study of Wayne et al: at 2 weeks, 1 month, 3 months, and 6 months, thus exposing their subjects to an even higher “testing effect.” In contrast to Stefanidis et al,12 we did not incorporate overtraining into our sessions and our earliest retesting interval was at 6 months, with most of our residents being tested even further from their initial training session.

Studies are mixed as to the reliability of self-reported confidence levels, with a few even showing a reciprocal relationship between actual skill and self-reported confidence.7,9 Our residents' self-reported confidence levels declined in correspondence with the lack of experience in performing actual procedures. For example, confidence to perform shoulder injections (specifically, the subacromial bursa) decreased more than the confidence to perform knee injections, reflecting the fewer opportunities for residents to perform shoulder injections as compared with knee injections.

The paucity of actual intra-articular injections done during residency is unfortunately common. For example, Jolly and Curran3 demonstrated that only 11% of faculty reported performing more than 5 intra-articular injections during their residency training, although 60% of them considered 5 to 10 intra-articular injections to be optimal for competence. The optimal number of knee or shoulder injections necessary for competence or for mastery is unclear. However, our study suggests that 1 to 5 actual knee procedures is insufficient, since the skill levels did not differ between the group performing 1 to 5 procedures and the group performing none.

Our study is the first to measure the retention of simulation-acquired knee and shoulder injection skills at 6 to 30 months and to investigate whether prior web-based review is sufficient to preserve these skills; our sample size of 44 residents compares favorably to that of other studies in this area. It is also the first to compare confidence levels in performing these procedures beyond 10 months after a teaching session. At the same time, our study has several potential limitations. It was conducted in a single residency program at a single academic institution, limiting generalizability. The calculations of our skills checklist items' internal consistency, the Cronbach α, were low in some cases because a substantial number of our residents obtained full scores. This is a common limitation of Cronbach α calculations. Although our retesting interval was wide (ie, 6–30 months), it represents a realistic time frame during which knee and shoulder procedure opportunities arise in an internal medicine residency program.

Conclusion

Our study demonstrates that prior review of web-based instructional material improves the proficiency with informed consent for the shoulder and postprocedure instructions for knee and shoulder. There also seems to be a congruent decline between the number of procedures performed and the self-reported confidence level. Future research should focus on identifying what factor or combination of factors, including dose learning, overtraining, feedback, retraining intervals, or other methods, are the most critical for preservation of simulation-acquired skills.

References

References
1
Kahl
LE
.
Musculoskeletal problems in the family practice setting: guidelines for curriculum design
.
J Rheumatol
.
1987
;
14
:
811
814
.
2
Wigton
RS
,
Alguire
P
.
The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians
.
Ann Intern Med
.
2007
;
146
:
355
360
.
3
Jolly
M
,
Curran
JJ
.
Underuse of intra-articular and periarticular corticosteroid injections by primary care physicians: discomfort with the technique
.
J Clin Rheumatol
.
2003
;
9
:
187
192
.
4
Vogelgesang
SA
.,
Karplus
TM
,
Kreiter
CD
.
An instructional program to facilitate teaching joint/soft-tissue injection and aspiration
.
J Gen Intern Med
.
2002
;
17
:
441
445
.
5
Dunn
W
,
Murphy
JG
.
Simulation: about safety, not fantasy
.
Chest
.
2008
;
133
:
6
9
.
6
McGaghie
WC
.,
Issenberg
SB
,
Petrusa
ER
,
Scalese
RJ
.
A critical review of simulation-based medical education research: 2003–2009
.
Med Educ
.
2010
;
44
:
50
63
.
7
Barilla-LaBarca
M
.,
Tsang
JC
,
Goldsmith
M
,
Furie
R
.
Design, implementation, and outcome of a hands-on arthrocentesis workshop
.
J Clin Rheumatol
.
2009
;
15
:
275
279
.
8
Cuccurullo
S
.,
Brown
D
,
Petagna
AM
,
Platt
H
,
Strax
TE
.
Musculoskeletal injection skills competency in physical medicine and rehabilitation residents: a method for development and assessment
.
Am J Phys Med Rehabil
.
2004
;
83
:
479
485
.
9
Leopold
SS
.,
Morgan
HD
,
Kadel
NJ
,
Gardner
GC
,
Schaad
DC
,
Wolf
FM
.
Impact of educational intervention on confidence and competence in the performance of a simple surgical task
.
J Bone Joint Surg Am
.
2005
;
87
:
1031
1037
.
10
Stitik
TP
.,
Foye
PM
,
Nadler
SF
,
Chen
B
,
Schoenherr
L
,
Von Hagen
S
.
Injections in patients with osteoarthritis and other musculoskeletal disorders: use of synthetic injection models for teaching physiatry residents
.
Am J Phys Med Rehabil
.
2005
;
84
:
550
559
.
11
Wilcox
T
.,
Oyler
J
,
Harada
C
,
Utset
T
.
Musculoskeletal exam and joint injection training for internal medicine residents
.
J Gen Intern Med
.
2006
;
21
:
521
523
.
12
Stefanidis
D
.,
Korndorffer
JR
,
Markley
S
,
Sierra
R
,
Scott
DJ
.
Proficiency maintenance: impact of ongoing simulator training on laparoscopic skill retention
.
J Am Coll Surg
.
2006
;
202
:
599
603
.
13
Wayne
DB
.,
Siddall
VJ
,
Butter
J
,
Fudala
MJ
,
Linquist
LA
,
Feinglass
J
,
et al .
A longitudinal study of internal medicine residents' retention of advanced cardiac life support skills
.
Acad Med
.
2006
;
81
:
S9
S12
.
14
Berden
HJJM
.,
Willems
FF
,
Hendrick
JMA
,
Pijls
NHJ
,
Knape
JTA
.
How frequently should basic cardiopulmonary resuscitation training be repeated to maintain adequate skills
?
BMJ
.
1993
;
306
:
1576
1577
.
15
Kovacs
G
,
Bullock
G
,
Ackroyd-Stolarz
S
,
Cain
E
,
Petrie
D
.
A randomized controlled trial on the effect of educational interventions in promoting airway management skill maintenance
.
Ann Emerg Med
.
2000
;
36
:
301
309
.
16
Kuduvalli
PM
.,
Jervis
A
,
Tighe
SQM
,
Robin
NM
.
Unanticipated difficult airway management in anaesthetized patients: a prospective study of the effect of mannequin training on management strategies and skill retention
.
Anaesthesia
.
2008
;
63
:
364
369
.
17
Jolly
M
,
Hill
A
,
Mataria
M
,
Agarwal
S
.
Influence of an interactive joint model injection workshop on physicians' musculoskeletal procedural skills
.
J Rheumatol
.
2007
;
34
:
1576
1579
.
18
Wayne
DB
,
Buttler
J
,
Siddall
VJ
,
Fudala
MJ
,
Linquist
LA
,
Feinglass
J
,
et al .
Simulation-based training of internal medicine residents in advanced cardiac life support protocols: a randomized trial
.
Teach Learn Med
.
2005
;
17
(
3
):
202
208
.
19
Ericsson
KA
.
Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains
.
Acad Med
.
2004
;
79
:
S70
S81
.
20
Roediger
HL
,
Karpicke
JD
.
Test-enhanced learning: taking memory tests improves long-term retention
.
Psychol Sci
.
2006
;
17
:
249
255
.

Author notes

Ruth Preisner, MD, is Associate Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; Harish Jasti, MD, is Assistant Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; Michael Elnicki, MD, is Professor of Medicine, Division of General Internal Medicine, University of Pittsburgh School of Medicine; and Kwonho Jeong, BA, is in the Epidemiology Data Center at the University of Pittsburgh.

Funding: The authors report no external funding source for this study.

The authors wish to thank John Lutz, Director of Information Technology, and Peter M. Winter, Institute for Simulation, Education, and Research.