ABSTRACT

Background

There is insufficient knowledge about how personal access to handheld ultrasound devices (HUDs) improves trainee learning with point-of-care ultrasound (POCUS).

Objective

To assess whether HUDs, alongside a yearlong lecture series, improved trainee POCUS usage and ability to acquire images.

Methods

Internal medicine intern physicians (n = 47) at a single institution from 2017 to 2018 were randomized 1:1 to receive personal HUDs (n = 24) for patient care/self-directed learning vs no-HUDs (n = 23). All interns received a repeated lecture series on cardiac, thoracic, and abdominal POCUS. Main outcome measures included self-reported HUD usage rates and post-intervention assessment scores using the Rapid Assessment of Competency in Echocardiography (RACE) scale between HUD and no-HUD groups.

Results

HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in non-HUD arm (P = .66). There was no relationship between the number of self-reported examinations per week and a trainee's post-intervention RACE score (rho = 0.022, P = .95). HUD interns did not have significantly higher post-intervention RACE scores (median HUD score 17.0 vs no-HUD score 17.8; P = .72). Trainee confidence with cardiac POCUS did not correlate with RACE scores.

Conclusions

Personal HUDs without direct supervision did not increase the amount of POCUS usage or improve interns' acquisition abilities. Interns who reported performing more examinations per week did not have higher RACE scores. Improved HUD access and lectures without additional feedback may not improve POCUS mastery.

Objectives To assess whether handheld ultrasound devices (HUDs), alongside a yearlong lecture series, improved trainee point-of-care ultrasound (POCUS) usage and ability to acquire images.

Findings Personal HUDs did not increase the amount of POCUS usage or improve interns' acquisition abilities.

Limitations Single center design and self-reported HUD usage rates.

Bottom Line Access to personal HUDs may not be a major limiting factor in promoting POCUS usage among trainees.

Introduction

Point-of-care ultrasound (POCUS) usage is increasing across the clinical training spectrum as more investigations have demonstrated its positive effect on patient care.15  However, there is a risk trainees may be performing POCUS with minimal oversight (including for procedures).6  Outside of emergency medicine, there are few guidelines on how to effectively teach POCUS or measure competency.68  There is a growing need to investigate how to optimally teach trainees this ever-increasing diagnostic modality.35,8 

Previous investigations of POCUS education have mainly focused on the effect of lectures on image interpretation.6,915  POCUS educational guidelines have advocated for improved device availability,8,1618  but there have been no studies examining whether access to personal handheld ultrasound devices (HUDs) improves trainee competency. In theory, improved access to HUDs increases opportunities for deliberate practice, which is needed to improve competency with POCUS beyond the classroom setting.19  Such practice may improve technical skill with acquiring images, which is a key feature of POCUS competency.6,19 

In this randomized controlled study, we investigated whether increased access to HUDs increases the frequency POCUS is performed and whether increased HUD access improves a trainee's ability to acquire POCUS images. We also investigated whether trainees who report performing more examinations per week are more skilled at acquiring images.

Methods

Participants and Setting

The 2017 intern class (n = 47) at an internal medicine residency program participated in this study. The Stanford University Institutional Review Board (IRB) approved this investigation.

Study Design

Interns (n = 47) received didactics related to POCUS from June 2017 to June 2018 (Figure). In addition, the interns were randomized 1:1 to receive personal HUDs that could be used for patient care and/or self-directed learning (n = 24) vs no-HUDs (n = 23). POCUS usage rates and trainee ability to acquire POCUS images were assessed.

Figure

Overview of Study

Note: In period 1, assessments were performed on incoming 2017 interns (pre-intervention/baseline measurement) who were at the beginning of their year. No point-of-care ultrasound (POCUS)–related didactics were administered prior to the pre-intervention measurement. In period 2, they all received weekly didactics in POCUS while rotating on the inpatient wards service. In addition, they were randomized 1:1 to have personal access to a handheld ultrasound device (HUD; n = 24) or no such device (non-HUD; n = 23). In period 3, post-intervention assessments were performed after completing their year of ultrasound training. We compared image acquisition performance between pre- and post-intervention groups, as well as HUD vs no HUD groups. Due to scheduling constraints, 25 interns completed the post-intervention measurement (HUD, n = 11; no-HUD, n = 14).

Figure

Overview of Study

Note: In period 1, assessments were performed on incoming 2017 interns (pre-intervention/baseline measurement) who were at the beginning of their year. No point-of-care ultrasound (POCUS)–related didactics were administered prior to the pre-intervention measurement. In period 2, they all received weekly didactics in POCUS while rotating on the inpatient wards service. In addition, they were randomized 1:1 to have personal access to a handheld ultrasound device (HUD; n = 24) or no such device (non-HUD; n = 23). In period 3, post-intervention assessments were performed after completing their year of ultrasound training. We compared image acquisition performance between pre- and post-intervention groups, as well as HUD vs no HUD groups. Due to scheduling constraints, 25 interns completed the post-intervention measurement (HUD, n = 11; no-HUD, n = 14).

Outcomes

Main outcome measures included self-reported HUD usage rates and differences in post-intervention assessment scores between HUD vs no-HUD interns (see “Assessments” below). Additional outcomes included the relationship between the number of self-reported scans and technical skill assessment scores, differences in pre- vs post-intervention assessment scores, and trainee confidence with POCUS via surveys administered pre- and post-intervention (provided as online supplementary data).

Curriculum Implementation and Design

All interns (n = 47) received a 1-hour weekly lecture whenever they rotated on the inpatient medicine wards rotation (Figure). This rotation is 4 weeks in duration. Each lecture contained 2 parts: (1) 20 to 30 minutes of traditional didactics via PowerPoint, and (2) 30 to 40 minutes of supervised practice using HUDs on standardized patients. Trainees were given personalized feedback on their image acquisition technique during this portion of the class. Four lectures were given each month: (1) introduction to POCUS, (2) thoracic ultrasound, (3) echocardiography, and (4) abdominal ultrasound. These 4 lectures were repeated each month as new interns rotated on service. Most interns experienced the same content multiple times (2–3 times/year), which was intentional to assess learning rates with repeated lecture exposure, as has been previously reported.20  The contents of each lecture were based on previously published guidelines or expert consensus for POCUS training.7,18,2028 

HUDs

This study used the Philips Lumify HUD, an FDA-approved device. Interns randomized to HUDs received their device at the start of the rotation. Based on our IRB protocol, any saved patient images could not be reviewed by the researchers. However, interns could share their findings with their attending physicians for clinical care purposes. Attending discretion and comfort with POCUS determined whether to incorporate these images into clinical decision making or to provide feedback, which were not measured. The interns not randomized to HUDs could access a single Lumify device that was shared among residents and fellows in the hospital. This control HUD was kept in a centrally stored container that participants had to sign out from the unit secretary. Interns reported the average number of POCUS examinations performed each week via a survey sent during the last week of the rotation.

Assessments

Trainees were assessed on their ability to obtain cardiac and abdominal ultrasound images during the pre- vs post-intervention periods. This study utilized 2 technique assessments: (1) the Rapid Assessment of Competency in Echocardiography (RACE) scale, and (2) the Brightness Mode Quality Ultrasound Imaging Examination Technique (B-QUIET).29,30  Both methods have excellent interrater reliability and are well-described assessment tools for image acquisition ability regarding cardiac and abdominal POCUS.29,30  The RACE scale assesses image acquisition skills for trainees performing bedside echocardiography (provided as online supplementary data).6,29  The B-QUIET is a technique assessment that can be applied toward abdominal ultrasound (provided as online supplementary data).6,30 

All pre- and post-intervention assessments were performed on 3 healthy male volunteers who were screened to ensure they had easily obtainable image windows. Trainees were randomly assigned a volunteer for scanning. Trainees were asked to obtain the following views/structures: parasternal long, parasternal short, apical 4-chamber, subcostal, inferior vena cava, splenic-diaphragm interface (from the mid-axillary line), and longitudinal right kidney (provided as online supplementary data). Trainees were given 15 minutes to scan and were instructed to save an image when they believed they had obtained an optimal image for evaluation. A study author was present for the scanning sessions to provide instruction and to set up the device but did not directly observe or comment on the images being acquired. Three of the study authors who teach POCUS at our institution (J.K., L.W., A.K.) independently reviewed the saved images and assigned scores using the RACE and B-QUIET scales. The study authors were blinded to the participant's identification when reviewing images. RACE scores were applied toward cardiac images, and B-QUIET scores were applied toward splenic and renal images. The average score across the 3 reviewers was used for the final trainee score.

Statistical Analysis

Outcomes were compared between the 2 groups using unpaired 2-sample Wilcoxon signed-rank tests. Correlations between interns' self-reported confidence levels and assessment scores were evaluated using Spearman's rank test of correlation using the R Project for Statistical Computing. A priori power analysis was performed with assumptions of a 0.0125 type I error threshold, standard deviation of 0.5, and effect size of 0.3. A sample size of 128 interns (64 per group) would be needed for 80% power to detect statistical meaningful differences. Therefore, this study was underpowered because we could only perform it on the 2017–2018 intern class.

Results

Baseline Characteristics

There were 47 categorical and preliminary interns in the study. Image acquisition skills were evaluated for 47 interns at the beginning of the study and 25 interns (HUD, n = 11; no-HUD, n = 14) at the end of the study (51% completion rate; Table 1). Lecture attendance was similar between the groups (Table 1).

Table 1

Participation and Performance of the 2017 Intern Class

Participation and Performance of the 2017 Intern Class
Participation and Performance of the 2017 Intern Class

Outcomes

Ultrasound Usage Rates:

HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in the non-HUD arm (P = .66). There was no correlation between an intern's final RACE or B-QUIET score and the amount of times they reported performing POCUS per week (Table 2).

Table 2

Ultrasound Usage Rates and Post-Intervention Score

Ultrasound Usage Rates and Post-Intervention Score
Ultrasound Usage Rates and Post-Intervention Score

RACE/B-QUIET Scores:

There were no significant differences in post-intervention RACE scores (including overall image quality and image generation) or B-QUIET scores between the HUD vs. no-HUD groups (Table 1). The median RACE image generation score increased from 0.0 pre-intervention (IQR 0.0–6.0) to 17.3 post-intervention (IQR 15.0–19.0; P < .0001; provided as online supplementary data) for all interns. Similarly, the B-QUIET scores for kidney and splenic POCUS significantly increased pre- vs post-intervention for all interns (provided as online supplementary data).

Trainee Confidence:

Overall, the interns' final RACE scores did not correlate with their confidence to correctly interpret POCUS images of the heart (Table 3). Similarly, the B-QUIET composite scores for both renal and splenic POCUS did not correlate with the intern's self-reported confidence to identify these structures on ultrasound (provided as online supplementary data).

Table 3

Correlation Between Trainee Confidence With Post-Intervention Assessment Scores

Correlation Between Trainee Confidence With Post-Intervention Assessment Scores
Correlation Between Trainee Confidence With Post-Intervention Assessment Scores

Discussion

POCUS educational guidelines have advocated for improved device availability,8,16,18,19  but it is unknown if increased device access improves POCUS learning. In this study, we found that personal HUDs and a lecture series were neither correlated with an increase in POCUS usage nor did they improve technique assessment scores for trainees. Furthermore, trainee RACE or B-QUIET scores did not correlate with higher self-reported rates of POCUS usage or trainee confidence. Together, these findings suggest that improved HUD access with a lecture series that provides hands-on scanning may not be enough to promote POCUS mastery. Supplementary pedagogies, such real-time feedback with actual patients, are likely needed.6,8,19 

Regarding the observation that POCUS usage rates did not increase with HUD access, it is possible that no-HUD interns used the community device more frequently because of the intervention. Alternatively, it is possible that the demands of being a trainee prevented HUD interns from performing more examinations, especially since time-motion studies have revealed that internal medicine interns spend less than 15% of their time toward direct patient care.32  Future studies should stringently track POCUS usage among trainees using time-motion methodologies and device logs.

Irrespective of identical HUD usage rates, we observed that trainees who reported higher HUD usage did not achieve higher RACE or B-QUIET scores. Previous investigations have shown that trainees can become proficient in acquiring cardiac and abdominal POCUS images in as few as 20 to 30 examinations,29,33,34  while other authors have shown that the mastery of skills requiring manual dexterity takes years of deliberate practice.3537  It is possible the interns quickly achieved a plateau effect with POCUS proficiency, as observed with Millington et al.33  Additional mastery to detect a difference between the HUD groups or superusers may have required additional practice beyond the time frame of this study. Finally, we observed that there was no correlation between trainee confidence and their actual skill. This phenomenon has been described as the Dunning–Kruger effect and it may be common in POCUS learning.6,38 

There are several limitations to this study. It was performed at a single academic institution with an underpowered sample size. It examined only intern physicians due to funding and rotation constraints. We were unable to perform the post-intervention assessment on all interns due to scheduling limitations (eg, days off and off-site rotations). We were unable to track the timing/location of the devices' usage, and the interns' self-reported usage rates may be subject to recall bias. To our knowledge, there were no significant lapses in device availability/functionality, and interns were discouraged from sharing devices. Due to timing constraints, we did not perform interval assessments of learning over the year. Intern physicians in the HUD arm didn't receive formal feedback on personally acquired patient images, which may have limited the intervention's impact.

Conclusions

In this small randomized study, we did not observe an improvement in internal medicine interns' acquisition of cardiac, splenic, or renal POCUS images despite having personal HUDs. Additionally, interns did not scan more frequently than peers without HUDs, suggesting that personal device access may not be a major limiting factor in promoting POCUS usage.

References

References
1. 
Park
YH,
Jung
RB,
Lee
YG,
Hong
CK,
Ahn
J-H,
Shin
TY,
et al.
Does the use of bedside ultrasonography reduce emergency department length of stay for patients with renal colic?: A pilot study
.
Clin Exp Emerg Med
.
2016
;
3
(4)
:
197
203
.
2. 
Glomb
N,
D'Amico
B,
Rus
M,
Chen
C.
Point-of-care ultrasound in resource-limited settings
.
Clin Pediatr Emerg Med
.
2015
;
16
(4)
:
256
261
.
doi:10.1053/j.sult. 2018.03.007.
3. 
Bahner
DP,
Goldman
E,
Way
D,
Royall
NA,
Liu
YT.
The state of ultrasound education in U.S. medical schools: results of a national survey
.
Acad Med
.
2014
;
89
(12)
:
1681
1686
.
4. 
Hall
JWW,
Holman
H,
Bornemann
P,
Barreto
T,
Henderson
D,
Bennett
K,
et al.
Point of care ultrasound in family medicine residency programs: a CERA study
.
Fam Med
.
2015
;
47
(9)
:
706
711
.
5. 
Schnobrich
DJ,
Gladding
S,
Olson
APJ,
Duran-Nelson
A.
Point-of-care ultrasound in internal medicine: a national survey of educational leadership
.
J Grad Med Educ
.
2013
;
5
(3)
:
498
502
.
6. 
Kumar
A,
Jensen
T,
Kugler
J.
Evaluation of trainee competency with point-of-care ultrasonography (POCUS): a conceptual framework and review of existing assessments
.
J Gen Intern Med
.
2019
;
34
(6)
:
1025
1031
.
7. 
Stolz
LA,
Stolz
U,
Fields
JM,
Saul
T,
Secko
M,
Flannigan
MJ,
et al.
Emergency medicine resident assessment of the emergency ultrasound milestones and current training recommendations
.
Acad Emerg Med
.
2017
;
24
(3)
:
353
361
.
8. 
LoPresti
CM,
Schnobrich
DJ,
Dversdal
RK,
Schembri
F.
A road map for point-of-care ultrasound training in internal medicine residency
.
Ultrasound J
.
2019
;
11
(1)
:
10
.
9. 
Razi
R,
Estrada
JR,
Doll
J,
Spencer
KT.
Bedside hand-carried ultrasound by internal medicine residents versus traditional clinical assessment for the identification of systolic dysfunction in patients admitted with decompensated heart failure
.
J Am Soc Echocardiogr
.
2011
;
24
(12)
:
1319
1324
.
doi:10.1016/j.echo. 2011.07.013.
10. 
Levitov
A,
Frankel
HL,
Blaivas
M,
Kirkpatrick
AW,
Su
E,
Evans
D,
et al.
Guidelines for the appropriate use of bedside general and cardiac ultrasonography in the evaluation of critically ill patients—part II: cardiac ultrasonography
.
Crit Care Med
.
2016
;
44
(6)
:
1206
.
11. 
Kobal
SL,
Trento
L,
Baharami
S,
Tolstrup
K,
Naqvi
TZ,
Cercek
B,
et al.
Comparison of effectiveness of hand-carried ultrasound to bedside cardiovascular physical examination
.
Am J Cardiol
.
2005
;
96
(7)
:
1002
1006
.
12. 
Ceriani
E,
Cogliati
C.
Update on bedside ultrasound diagnosis of pericardial effusion
.
Intern Emerg Med
.
2016
;
11
(3)
:
477
80
.
13. 
Keil-Ríos
D,
Terrazas-Solís
H,
González-Garay
A,
Sánchez-Ávila
JF,
García-Juárez
I.
Pocket ultrasound device as a complement to physical examination for ascites evaluation and guided paracentesis
.
Intern Emerg Med
.
2016
;
11
(3)
:
461
466
.
14. 
Whitson
MR,
Mayo
PH.
Ultrasonography in the emergency department
.
Crit Care
.
2016
;
20
(1)
:
227
.
15. 
Kelm
DJ,
Ratelle
JT,
Azeem
N,
Bonnes
SL,
Halvorsen
AJ,
Oxentenko
AS,
et al.
Longitudinal ultrasound curriculum improves long-term retention among internal medicine residents
.
J Grad Med Educ
.
2015
;
7
(3)
:
454
457
.
16. 
LoPresti
CM,
Jensen
TP,
Dversdal
RK,
Astiz
DJ.
Point-of-care ultrasound for internal medicine residency training: a position statement from the Alliance of Academic Internal Medicine
.
Am J Med
.
2019
;
132
(11)
:
1356
1360
.
17. 
Soni
NJ,
Tierney
DM,
Jensen
TP,
Lucas
BP.
Certification of point-of-care ultrasound competency
.
J Hosp Med
.
2017
;
12
(9)
:
775
776
.
18. 
Ma
IWY,
Arishenkoff
S,
Wiseman
J,
Desy
J,
Ailon
J,
Martin
L,
et al.
Internal medicine point-of-care ultrasound curriculum: consensus recommendations from the Canadian Internal Medicine Ultrasound (CIMUS) Group
.
J Gen Intern Med
.
2017
;
32
(9)
:
1052
1057
.
19. 
Lucas
BP,
Tierney
DM,
Jensen
TP,
Dancel
R,
Cho
J,
El-Barbary
M,
et al.
Credentialing of hospitalists in ultrasound-guided bedside procedures: a position statement of the Society of Hospital Medicine
.
J Hosp Med
.
2018
;
13
(2)
:
117
125
.
20. 
Kumar
A,
Weng
Y,
Wang
L,
Bentley
J,
Almli
M,
Hom
J,
et al.
Portable ultrasound device usage and learning outcomes among internal medicine trainees: a parallel-group randomized trial
.
J Hosp Med
.
2020
;
15
(2)
:
e1
e6
.
21. 
Sabath
BF,
Singh
G.
Point-of-care ultrasonography as a training milestone for internal medicine residents: the time is now
.
J Community Hosp Intern Med Perspect
.
2016
;
6
(5)
:
33094
.
22. 
American College of Emergency Physicians.
Policy Statement. Ultrasound Guidelines: Emergency, Point-of-Care and Clinical Ultrasound Guidelines in medicine.
2020
.
23. 
Ramsingh
D,
Rinehart
J,
Kain
Z,
Strom
S,
Canales
C,
Alexander
B,
et al.
Impact assessment of perioperative point-of-care ultrasound training on anesthesiology residents
.
Anesthesiology
.
2015
;
123
(3)
:
670
682
.
24. 
Keddis
MT,
Cullen
MW,
Reed
DA,
Halvorsen
AJ,
McDonald
FS,
Takahashi
PY,
et al.
Effectiveness of an ultrasound training module for internal medicine residents
.
BMC Med Educ
.
2011
;
11
:
75
.
25. 
Townsend
NT,
Kendall
J,
Barnett
C,
Robinson
T.
An effective curriculum for focused assessment diagnostic echocardiography: establishing the learning curve in surgical residents
.
J Surg Educ
.
2016
;
73
(2)
:
190
196
.
26. 
Hoppmann
RA,
Rao
VV,
Bell
F,
Poston
MB,
Howe
DB,
Riffle
S,
et al.
The evolution of an integrated ultrasound curriculum (iUSC) for medical students: 9-year experience
.
Crit Ultrasound J
.
2015
;
7
(1)
:
18
.
27. 
Skalski
JH,
Elrashidi
M,
Reed
DA,
McDonald
FS,
Bhagra
A.
Using standardized patients to teach point-of-care ultrasound-guided physical examination skills to internal medicine residents
.
J Grad Med Educ
.
2015
;
7
(1)
:
95
97
.
28. 
Chisholm
CB,
Dodge
WR,
Balise
RR,
Williams
SR,
Gharahbaghian
L,
Beraud
A-S.
Focused cardiac ultrasound training: how much is enough?
J Emerg Med
.
2013
;
44
(4)
:
818
822
.
29. 
Millington
SJ,
Arntfield
RT,
Hewak
M,
Hamstra
SJ,
Beaulieu
Y,
Hibbert
B,
et al.
The rapid assessment of competency in echocardiography scale: validation of a tool for point-of-care ultrasound
.
J Ultrasound Med
.
2016
;
35
(7)
:
1457
1463
.
30. 
Bahner
DP,
Adkins
EJ,
Nagel
R,
Way
D,
Werman
HA,
Royall
NA.
Brightness mode quality ultrasound imaging examination technique (B-QUIET): quantifying quality in ultrasound imaging
.
J Ultrasound Med
.
2011
;
30
(12)
:
1649
1655
.
31. 
Frankel
HL,
Kirkpatrick
AW,
Elbarbary
M,
Blaivas
M,
Desai
H,
Evans
D,
et al.
Guidelines for the appropriate use of bedside general and cardiac ultrasonography in the evaluation of critically ill patients-part I: general ultrasonography
.
Crit Care Med
.
2015
;
43
(11)
:
2479
2502
.
32. 
Desai
SV,
Asch
DA,
Bellini
LM,
Chaiyachati
KH,
Liu
M,
Sternberg
AL,
et al.
Education outcomes in a duty-hour flexibility trial in internal medicine
.
N Engl J Med
.
2018
;
378
(16)
:
1494
1508
.
33. 
Millington
SJ,
Hewak
M,
Arntfield
RT,
Beaulieu
Y,
Hibbert
B,
Koenig
S,
et al.
Outcomes from extensive training in critical care echocardiography: identifying the optimal number of practice studies required to achieve competency
.
J Crit Care
.
2017
;
40
:
99
102
.
34. 
Millington
SJ,
Arntfield
RT,
Guo
RJ,
Koenig
S,
Kory
P,
Noble
V,
et al.
The Assessment of Competency in Thoracic Sonography (ACTS) scale: validation of a tool for point-of-care ultrasound
.
Crit Ultrasound J
.
2017
;
9
(1)
:
25
.
35. 
Ericsson
KA.
Deliberate practice and acquisition of expert performance: a general overview
.
Acad Emerg Med
.
2008
;
15
(11)
:
988
994
.
36. 
Chi
MTH,
Glaser
R,
Farr
MJ,
eds.
The Nature of Expertise
.
New York, NY
:
Psychology Press;
2014
.
37. 
Ericsson
KA,
Hoffman
RR,
Kozbelt
A,
Williams
AM,
eds.
The Cambridge Handbook of Expertise and Expert Performance. 2nd ed
.
New York, NY
:
Cambridge University Press;
2018
.
38. 
Kruger
J,
Dunning
D.
Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments
.
J Pers Soc Psychol
.
1999
;
77
(6)
:
1121
1134
.

Author notes

Editor's Note: The online version of this article contains the survey used in the study, RACE score sheet, B-QUIET assessment, pre-intervention vs post-intervention RACE and B-QUIET scores, correlation between trainee confidence with post-intervention assessment scores, and survey results pre- vs post-intervention by HUD and no-HUD.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: Dr Kumar is a paid consultant for Vave Health, which manufactures a handheld ultrasound device not used in this study. This consultancy began after this study was conducted.

Supplementary data