Development and Validation of an Assessment Tool for Competency in Critical Care Ultrasound

Portable ultrasound technology is increasingly used in the diagnosis and management of critically ill patients. As an easily accessible and frequently used bedside tool, critical care ultrasound (CCUS) has the potential to improve patient safety; however, in the hands of untrained providers, this technology may be harmful. The current standard of procedure logs with arbitrary thresholds and subjective assessment of skill is not adequate to assess competency.^1,2  This has led to greater interest in developing tools to assess complex procedures such as bronchoscopy that include technical skills, medical knowledge, and clinical judgment.^3,4 

With the introduction of the new accreditation system⁵  by the Accreditation Council for Graduate Medical Education,⁵  critical care medicine (CCM) and pulmonary/CCM fellowship programs are required to demonstrate trainee competency in procedures, including ultrasound.^6–8  Performing ultrasound safely and efficiently is an important entrustable professional activity for pulmonary medicine and CCM.⁹  Although the specific components required for CCUS competence have been defined,¹⁰  and a framework for training standards developed,¹¹  there is no certification program to ensure uniform standards.

Training standards include didactics in general CCUS and basic critical care echocardiography, practical training on clinically normal volunteers, bedside scanning on patients with a range of conditions, and pathology.¹¹  There is no consensus on the number of scans needed to achieve competence.¹¹  Developing a competency assessment instrument using psychometric principles with strong evidence for validity is essential in the era of outcomes-based education. A structured approach to validity for instrument development has been used for procedural skills.^12–15  However, we are not aware of any valid and reliable instruments to measure comprehensive CCUS skill.

The purpose of this study was to develop a competency assessment tool for CCUS and provide evidence for its validity.

We conducted a prospective study to develop and provide validity evidence for a CCUS competency assessment tool.

The New York University Institutional Review Board approved this study.

We adopted the framework of Downing¹⁶  to provide a validity argument for this tool. Elements include content, response process, internal structure, relationship to other variables, and consequences.

The instrument was developed by the modified Delphi method.^17,18  The Delphi method is a consensus-based content generation method frequently used to develop guidelines, policy statements, and assessment tools. The modified Delphi method differs in that an initial structure is given to participants, from which feedback is generated and incorporated until consensus is reached. This method has been used to develop performance checklists with high content-related validity evidence.^17,18  One author (P.P.), a critical care physician and ultrasound educator, created comprehensive step-by-step procedure checklists for CCUS modules based on existing literature and published guidelines.¹⁰  A modified Delphi method was used to edit and finalize the checklists. In round 1, an expert panel (N = 4) reviewed checklists, made free text edits, and classified step importance using a 9-point Likert scale (1 to 3, not important; 4 to 6, somewhat important; 7 to 9, very important). Steps with a mean rating ≤ 3 were eliminated, and suggested edits were incorporated. Repeated iterations of this method were continued until consensus was reached.

The expert panel included faculty with expertise in CCUS, checklist development, and educational outcome measures. All raters were faculty with extensive experience in CCUS application and education and Delphi panel members. Information regarding the Delphi process is presented descriptively. Means (ranges) are reported for checklist development.

We standardized rater activities through the use of scripted instructions. Participants imaged the same healthy actor. Feedback was formalized using a structured report card.

First-year CCM fellows from the New York metropolitan area attended a regional, 3-day, introductory CCUS course that meets multisociety-recommended training standards¹¹  and includes didactic presentations, image interpretation, and image acquisition on healthy actors. The training included essential CCUS modules as defined by a consensus statement:¹⁰  basic echocardiography, vascular access, vascular diagnostic, pleural, lung, and abdominal ultrasound.

Six to 30 months after attending the course, fellows from 3 training programs completed a comprehensive CCUS assessment. Cognitive skill was assessed with an image-based multiple-choice test. One live and 1 video rater reviewed 28 fellows, and a second video rater reviewed a subset of 10 using the skills checklists. The live rater completed checklists on specific tasks (ie, “Perform DVT study” [DVT] and “Qualify left ventricular function using parasternal long axis and parasternal short axis views” [Echo]). The live rater assigned each task using a script, directly observed the fellow, and then provided checklist-guided feedback. Time was allotted for deliberate practice until competence was achieved. Video raters completed checklists remotely by examining a video recording (1 camera on the probe, 1 on the console, and direct capture of ultrasound images via Arcadia 5.08, Education Management Solutions). Fellow performance on each task was categorized as competent, competent with areas for improvement, or not competent based on the global rating step (GRS). Fellows and program directors received standardized report cards.

Cronbach α coefficients were calculated using individual item scores completed by the live rater and 1 video rater on 28 subjects. Only items viewable by video raters were used to assess reliability. Total percent agreement and a Cohen kappa were used to measure interrater reliability between live and video raters (N = 28) and between 2 video raters (N = 10). The relationship between individual steps and the GRS is presented descriptively. Mean SD is presented for the multiple-choice test. Analysis was performed using SPSS version 22 (IBM Corp).

This element requires correlation to an external performance measure. Lack of other existing performance measures for this construct prevented our ability to assess this element.

Evidence of consequences was derived from the psychometric data.

Round 1 results of the modified Delphi method are presented for DVT (table 1) and Echo (table 2). All steps had a mean rating > 3 and therefore none were eliminated. Addition of a GRS to each checklist was the only edit. Consensus of 100% was reached with round 2. The final DVT checklist consists of 13 dichotomous (yes-no) steps, including the GRS, and the Echo checklist consists of 14 steps, including the GRS for each of 2 Echo views.

The DVT checklist had a Cronbach α of 0.85 across both live and video raters. The Echo checklist had a Cronbach α of 0.92 for both raters.

Interrater reliability was assessed using percent agreement and a Cohen kappa (tables 3 and 4). Steps poorly visualized by the video rater were excluded. Prevalence for some steps was at the extremes (eg, all learners completed the item correctly or no learner completed the item correctly). Kappas for DVT steps ranged from 0.21 to 1.00 between live and video raters and from 0 to 0.62 between 2 video raters (table 3). Total percent agreement between live and video raters and between 2 video raters was 100% for the DVT GRS (table 5). Kappas for Echo steps ranged from 0.29 to 0.58 between live and video raters and from 0.74 to 1.00 between 2 video raters (table 4). Kappas were 0.44 for the GRS of parasternal long axis and 0.58 for parasternal short axis between live and video raters. Between 2 video raters, kappa was 0.74 for parasternal short axis (table 5).

Performance on DVT steps 6 to 9 was the primary determinant of the DVT GRS. These steps required raters to simultaneously see and hear learner identification of anatomy, which was inconsistently audible on video. Echo step 7 primarily influenced parasternal long axis GRS, and step 10, the parasternal short axis GRS. These steps necessitated careful evaluation of ultrasound images. Video raters were able to replay without restriction, likely accounting for the higher reliability between video raters.

The mean SD score on the cognitive portion of the assessment was 81.9% (10.7), reflecting the skill homogeneity of these learners.

We describe a novel tool for assessing CCUS competence with preliminary evidence of its validity using criteria recommended by Downing.¹⁶  Lack of CCUS expertise is an area for improvement in many fellowship programs,¹⁹  and remote asynchronous video review offers an alternative to the time-intensive method of direct supervision. Importantly, our study provides valuable information on the benefits and limitations of live versus video review in CCUS.

We present validity evidence of this instrument following current theories.^16,20  We offer evidence for content validity, as the authors are content experts in CCUS and educational outcomes assessment. We also standardized rater activities through scripted instructions, offering evidence for response process validity. All participants imaged the same healthy actor, and feedback was formalized through a structured report card. We also offer evidence for internal consistency of both checklists. The Echo GRSs showed moderate interrater reliability, and the DVT GRS showed excellent interrater reliability. With expert raters, GRS may in fact be more reliable than checklists.^21–23  However, there was variability in the reliability of individual steps. We could not assess relationship to other variables, as the task included no alternative performance measures, nor can we offer evidence of consequences validity. The effect of the assessment on the examinee and learning is often used to determine if the instrument should be used for high-stakes or low-stakes assessment. However, no high-stakes CCUS examination exists, limiting our ability to assess this element.

Our tool was evaluated in a safe environment on a healthy actor. Fellows were excused from clinical responsibilities and they commented that this provided a unique learning opportunity. The assessment, including a multiple-choice examination, checklists, and deliberate practice, was typically completed within 1 hour. Faculty volunteered their time, and the only cost was for the actor.

Our study had several limitations. Raters were members of the modified Delphi panel, limiting generalizability to raters who are not content experts or did not develop this tool. Tool dissemination will require a rater guide and validation among raters not involved in tool development. Further detail on correct and incorrect task performance should be added to the behavioral anchors to improve interrater reliability.

The extreme prevalence of some items complicates reliability analysis. This relates to the skill homogeneity of subjects (ie, reflected in the similarity of their cognitive examination scores) and the lack of variability in the test subject. Interrater reliability was poor or showed no correlation for several individual checklist items. Several checklist items were excluded as they were poorly seen on video. Video raters also commented that camera footage was inadequate to assess some checklist items.

Implementation of a CCUS competency-based assessment is needed to ensure a new generation of CCUS-competent clinicians. Poor reliability of specific checklist items demonstrates the challenges of remote, asynchronous assessment by video raters. A live rater may best use this tool, particularly given the advantage of providing supervised performance improvement. This tool should be combined with other sources of evaluation for summative assessment or certification.²⁴  This tool needs additional validity evidence among live raters not involved in checklist development and will also need further evaluation in patients with pathology. Further study should be performed in a heterogeneous group of learners to assess this tool's ability to discriminate learner levels.

We designed a formative assessment tool to establish existing skills, provide checklist-directed feedback, and guide deliberate practice. We also present preliminary validity evidence for a competency assessment tool in critical care ultrasound. This instrument may be used in combination with other measures for formative assessment in critical care medicine fellows.