Simulation-Based Education with Mastery Learning Improves Paracentesis Skills

Internal medicine residents and gastroenterology fellows commonly perform paracentesis procedures. The American Board of Internal Medicine (ABIM) requires residents to be familiar with the indications, complications, and interpretation of this procedure.1 Recently, the Accreditation Council for Graduate Medical Education added paracentesis skills as a 2012 program requirement for gastroenterology fellowship training as well as a requirement that fellows have access to simulation training.2 Despite these recommendations, many procedures are performed by learners who are not appropriately proficient or confident,3,4 and there is variable adherence to the gastroenterology core curriculum regarding hepatology in many programs.5 

Simulation-based mastery learning6 featuring deliberate practice7 gives residents and fellows the opportunity for procedural skills development and feedback. Mastery learning requires that learners meet or exceed a minimum passing score (MPS) on a simulated examination prior to performing the procedure in actual clinical care.6 We previously used simulation-based mastery learning to improve skills of residents and fellows in thoracentesis,8 Advanced Cardiac Life Support,9 and central venous catheter (CVC) insertion.10–12 

We hypothesized that the mastery model of procedural training could be successfully extended to paracentesis skills on an ultrasound-compatible simulator that was designed and created at our institution.

From May to August 2008, various materials and strategies were tested to create an ultrasound-compatible paracentesis model. A prototype was produced and tested to obtain feedback and suggested modifications. Ten identical simulators were pilot tested at a national procedure skills conference held at Northwestern University in October 2009. Twenty-one practicing physicians who participated in the conference tested the simulator and provided feedback about the simulator's realism (provided as online supplemental material, Appendix A).

The paracentesis simulator represents the human abdomen from rib margin to pubis. A foam chassis contains a “cartridge” containing simulated liver, kidneys, pancreas, aorta, intestines, and ascitic fluid. The chassis and cartridge are covered with an ultrasound-compatible silicone-based “skin” which is affixed to the chassis. Realistic ultrasound images can be obtained (figure 1), and a paracentesis can be performed (figure 2). Research and development of the simulator required partial effort of 2 part-time staff to complete. The estimated cost of disposable supplies was $1500 and labor cost $9216. The final cost of materials for each simulator was $57.

An educational intervention was performed within the internal medicine residency at Northwestern University from July to September 2010. Participants were all 58 postgraduate year-one (PGY-1) internal medicine residents. The Northwestern University Institutional Review Board approved the study, and all participants provided informed consent.

PGY-1 residents received standardized simulation-based education regarding paracentesis. Two faculty members supervised educational sessions for a total of 5 days. Residents underwent baseline skill assessment (pretest) on the simulator, using a 25-item checklist (provided as online supplemental material, Appendix B). The checklist was developed by one author (J.H.B.) using evidence-based guidelines for paracentesis13,14 and checklist design.8–12,15,16 The checklist was reviewed for completeness by 2 other authors with expertise and experience in paracentesis, simulation-based education, and checklist design (J.A.V. and D.B.W.).

After undergoing pretesting, all study residents completed a 3-hour educational session featuring a lecture, the New England Journal of Medicine's paracentesis video,14 ultrasound training, and deliberate practice with the paracentesis simulator with directed feedback.7 

Following training, residents were retested (posttest) using the 25-item checklist and were expected to meet or exceed the MPS for simulated paracentesis procedures. Residents who failed to meet the MPS the underwent additional deliberate practice until they were able to meet the MPS. The MPS for the paracentesis clinical skills examination was determined by 9 clinical experts (ABIM-certified physicians, 6 physicians from internal medicine, and 1 physician each from gastroenterology, pulmonary/critical care, and infectious disease) using the Angoff and Hofstee standard setting methods.17–19 

Pretest and posttest scores were compared to assess the impact of the educational intervention. Residents responded to a questionnaire about their experience with the simulator and the training sessions after the intervention.

Demographic data were obtained from participants including age, sex, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, and prior clinical experience with paracentesis procedures. All pretests and posttests were graded by a single unblinded instructor (E.R.C.) and were videotaped. A dichotomous scoring scale (done correctly or incorrectly) was used for each item.

A faculty instructor with expertise in scoring clinical skills examinations, who was blinded to pretest and posttest status of each examinee (J.H.B.), rescored a 50% random sample of the videotaped examinations to assess interrater reliability. Course evaluation questionnaires have been previously published,8,9,12 and answers were recorded on a 5-point Likert scale (where 1  =  strongly disagree, and 5  =  strongly agree).

The Cohen kappa coefficient was used to assess interrater reliability. Pretest and posttest checklist scores were compared using a paired t-test. Multiple linear regression was used to assess the association of posttest score with pretest performance, age, sex, USMLE score, and clinical experience. Analyses were performed using PASW version 18.0 statistical software (SPSS, Inc., Chicago, IL).

All 58 PGY-1 residents consented to participate and completed the entire protocol (table 1).

The MPS for the skills checklist was set at 83%. Interobserver agreement for the educational intervention skills checklist was high (κ_n  =  .87). Paracentesis checklist skills improved from an average pretest score of 8.25/25 (33.0%; SD  =  15.2%) to 23.2/25 (92.7%; SD  =  5.4%) at posttest (P < .001) after the training intervention (figure 3).

Two residents (3%) failed to meet the MPS at posttest. They subsequently underwent additional deliberate practice and were retested. Both residents reached the MPS within 1 hour of further practice.

Regression analysis showed no associations between posttest scores and pretest performance, age, sex, USMLE Step 1 and 2 scores, or prior clinical experience with paracentesis procedures. Residents rated the training sessions highly (table 2).

To our knowledge, this is the first study to use the mastery learning model for simulation-based paracentesis education. Our data demonstrate significant improvement in paracentesis skills of PGY-1 residents after simulation-based education. The magnitude of skill improvement using the paracentesis simulator was similar to that shown in earlier studies with commercial-grade simulators for thoracentesis,8 Advanced Cardiac Life Support,9,15 and CVC insertion.10–12 These interventions have also been linked to improved patient care quality.10,11,20,21 Our results also support the policy of the ABIM1 which states that simulation should be used to verify trainees′ skill before performing invasive procedures in actual patient care.

This study contributes to what is known about paracentesis procedural skill acquisition among medical learners. When assessed rigorously, there was no association among demographic variables such as age, sex, or USMLE examination scores, and paracentesis clinical skills. Additionally, our study found no relationship between the number of paracentesis procedures performed in actual clinical care and measured procedural skills. However, a difference may have been unlikely, as most residents had previously performed 0–1 procedures. Past research on thoracentesis,8 CVC insertion,10,11 and endoscopic balloon dilation of the esophagus22 also shows no association between the number of procedures performed and procedural skill. This is problematic because a 2003 Core Curriculum developed by the Gastroenterology Leadership Council for gastroenterology fellows recommends that fellows demonstrate competence in paracentesis by performing at least 20 procedures.23 Clinical experience is not a proxy for skill,24 and our findings are a reminder that competence should not be assumed after a set number of procedures.

This study has several limitations. The tools used to create a “homemade” simulator and the faculty commitment required for simulation training may not be available at other institutions. Training programs should consider the benefits of simulation as well as new training requirements when planning educational interventions. The degree of improvement in posttest scores recorded in our study may be reduced because our paracentesis model was used for both education and testing and we did not use a true “control” group. However, this should not diminish the pronounced impact simulation-based training had on paracentesis skills of first-year residents. Additionally, post-testing was performed immediately after training, thereby enhancing recall of procedural steps. In the service of translational science goals,25 we have not yet linked improved paracentesis skills in the simulated environment to improved health care delivery (higher procedural success rates, lower referral rate to interventional radiology) or better patient outcomes (lower complication rates).

This study demonstrates the ability of simulation-based education to improve paracentesis skills in internal medicine residents. Use of simulation as an adjunct to clinical training helps students meet residency and fellowship accreditation requirements and is feasible and well accepted by learners. Further study should address the effect of training on the long-term retention of paracentesis skills and health care outcomes.