The TeleHealth OSCE: Preparing Trainees to Use Telemedicine as a Tool for Transitions of Care

The transition of care between inpatient and outpatient settings represents a particularly vulnerable time for patients.^1,2  Telemedicine, the provision of health care remotely, holds promise to bridge this transition, expand access to care, and potentially reduce subsequent hospital admission.³  Among the many applications of telemedicine, the potential for real-time post-discharge surveillance makes telemedicine a timely and effective means of providing post-hospital care. In 2020, with widespread social distancing regulations due to the COVID-19 pandemic, telemedicine has become an even more essential modality for patients to access care.

Despite this, telemedicine-specific communication and technical skills required for successful virtual encounters are not routinely taught in graduate medical education (GME). While didactics and elective clinical telemedicine experiences exist in some medical schools, few are required in core curricula, and published data regarding the specific content of such offerings are limited.^4,5  Many advocate for adoption of telemedicine competencies to facilitate training of residents^6–8 ; however, there are few telemedicine curricula or structured assessment tools in GME.

We developed an objective structured clinical examination (OSCE) case and new assessment tool to evaluate residents' telemedicine-specific skills and identify potential gaps in our residency program's curriculum.

This study was conducted with internal medicine (IM) residents at New York University (NYU) Grossman School of Medicine, a large, urban, multisite, university-based training program. A telemedicine case was developed for the 2019 multi-station OSCE, a formative assessment conducted over several dates throughout the academic year. Participants comprised residents from NYU's categorical, primary care, and Brooklyn community health tracks.

The case consisted of a telemedicine (video) visit between a resident and recently discharged standardized patient (SP). Two experienced SPs underwent 3 hours of case- and assessment-specific instruction (provided as online supplemental material). Residents were instructed in advance with relevant SP clinical information for the 10-minute scenario (provided as online supplemental material); however, they were not primed with the assessment items. SPs and residents were stationed in different rooms and communicated via video conference. Participants had not previously taken part in any didactics or clinical experiences regarding telemedicine. Participants were debriefed at the conclusion of the OSCE program and were provided with optional post-encounter self-assessment forms.

We developed a behaviorally anchored assessment tool that evaluated core communication skills and unique telemedicine skills (table 1). Core communication items used in our assessment tool have been previously described⁹  and are widely used to evaluate these domains. Specifically, the core communication assessment items are internally consistent in OSCE encounters across multiple samples of leaners, settings, and clinical scenarios, have demonstrated sufficient test/retest reliability and interrater reliability and show consistent patterns over time.

Telemedicine-specific assessment items were developed to capture key behaviors necessary for successful virtual encounters. To generate these items, 2 authors (D.J.S. and S.R.Z.) convened focus groups with experienced telemedicine clinicians from NYU Virtual Urgent Care, Steven A. Cohen Military Family Center, and “Doctor on Demand.” One author (D.J.S.) directly observed a series of virtual visits and interviewed clinicians to generate telemedicine behavioral anchors. Our assessment tool captures similar skills described by Cantone and colleagues,¹⁰  mirroring the following key skills: evaluating/optimizing technical aspects of virtual visits, adjusting posture/camera to maintain eye contact, verbalizing actions while documenting, and acknowledging technical glitches with the interface. The assessment tool also reflects a relevant subset of proposed nursing telehealth entrustable professional activities,¹¹  proposed telepsychiatry competencies,⁸  and skills described by the American Telemedicine Association.¹² 

Table 1 includes specific descriptors that represent “well done” behaviors. SPs evaluated residents across each domain with responses anchored to “not done,” “partly done,” and “well done.” Evaluations are presented as “% not done,” “% partly done,” and “% well done” for each assessment item, as well as mean % well done (SD) for each domain as a whole. Telemedicine-specific domain items were further stratified by resident training track and postgraduate year (PGY); 3×3 Pearson's chi square tests were used to assess for association of either training track or PGY with frequency of not done, partly done, and well done telemedicine evaluations.

This project met NYU's criteria for certification as a quality improvement and not a human subject research project and was exempt from institutional review board review.

Seventy-eight residents comprising all PGY and 3 training tracks participated in the TeleHealth OSCE case. Evaluations from all 78 participants were obtained. When core communication domains were analyzed in aggregate, a mean 95% (SD 3.3%) of residents received well done within the Information Gathering domain, and 91% (SD 4.9%) received well done within Relationship Development. A total of 78% (SD 14%) of residents received well done within the Education/Counseling domain. There were only 2 assessment items evaluated as not done (table 1).

In contrast, resident performance was variable within the Telemedicine domain. Only 46% (SD 45%) of residents received well done evaluations in this domain. Specifically, 24% (n = 19) of residents assessed technical barriers during the encounter, and 18% (n = 14) attempted a virtual physical examination. Only 17% of residents (n = 13) received well done evaluations for using video to augment information gathering—a key item that included virtual medicine reconciliation and discussing care plans with onsite caregivers providing collateral (table 1). Interestingly, self-assessments, provided by a subset of residents (n = 23), demonstrated that residents felt confident with performance despite SP evaluations. Most residents (91%, 21 of 23) reported that they felt prepared for this telemedicine encounter, and only 9% (2 of 23) of residents reported that the encounter could have gone better.

Our program's training tracks reflect distinct areas of focus that may influence telemedicine proficiency; therefore, we stratified evaluations to assess whether telemedicine-specific evaluations differed by track. There were no significant associations between telemedicine skill evaluations and training track (X = 2.88, DF 4, P = .57). There were no significant associations between telemedicine skill evaluation and training year (X = 1.26, DF 4, P = .91; table 2).

This post-discharge telemedicine OSCE demonstrated resident achievement of core communication competencies but revealed deficiencies in several telemedicine-specific skills across all tracks and PGY levels in a large IM residency program. Residents self-assessed their telemedicine performance higher than the SPs who rated resident performance with a new behaviorally anchored assessment tool.

To our knowledge this is the first assessment of specific telemedicine skills among IM residents, such as performing a virtual physical examination, appropriately identifying patients remotely, optimizing the audio-video interface, and using video to augment history taking. Our study supports assertions that telemedicine requires distinct interpersonal and technical skills that warrant dedicated assessment and training.^7,13  Somewhat striking are our findings that trainee's self-reported confidence with telemedicine differs significantly from their objective telemedicine-specific performance. GME trainees may not recognize that telemedicine represents far more than medicine via FaceTime⁷  and requires distinct skills.

Limitations of our study include the single institution sample of participants that may not be generalizable to other settings or specialties. In addition, the case focused on a post-discharge telemedicine visit, which may not reflect skills required in virtual encounters in other settings, such as urgent care. As we did not examine interrater reliability, the 2 different SPs may have evaluated residents differently. Lastly, we had a rather low self-assessment response rate (n = 23 of 78) and thus conclusions regarding perceived telemedicine skill apply to a subset of learners.

Future studies will focus on expanding assessment of residents' telemedicine skills to different types of telemedicine OSCE encounters, for example those requiring urgent evaluation and triage and those focusing on chronic disease surveillance. These will provide further evaluation of the reliability of our assessment tool.

This study found that IM residents participating in a post-discharge telemedicine OSCE assessed using a new behaviorally anchored assessment tool demonstrated good core communication skills but were deficient in several telemedicine-specific skills, regardless of training year or training track. Residents did not recognize their lack of telemedicine-specific skills.