Implementing an Experiential Telehealth Training and Needs Assessment for Residents and Faculty at a Veterans Affairs Primary Care Clinic

When the COVID-19 pandemic began, telehealth services, particularly synchronous video services, tripled across Veterans Affairs (VA) platforms.^1-3  To address this crisis, the VA made extensive efforts to provide telehealth-based primary care for veterans nationally. Our team at the VA New York Harbor Healthcare System (NYHHS) Primary Care clinic recognized an opportunity for telehealth evaluation and training for clinicians. A survey of physicians conducted 6 months after the onset of the pandemic at 5 sites in the NYU Langone hospital system, including the VA, found that physicians struggle with establishing and maintaining relationships with patients and addressing technical issues.^4,5  For clinicians, telehealth optimization requires the development of skills in virtual rapport building, “facilitated” physical examinations, intentional nonverbal communication, and counseling, in addition to computer etiquette and technical skills.^6-8  In our medical education programs, recent performance-based assessments have consistently shown that the majority of residents have not yet mastered the specific skills required for telehealth.⁹ 

The extensive benefits of telehealth, including reduced cost and travel time, lower risk of exposure to disease, and greater accessibility for some populations, suggest it will remain an integral part of the evolving health care landscape.⁷  Most existing telehealth resources are limited to general guidelines, best practices, and objective structured clinical examinations (OSCEs).^10-13  Targeted, individualized, and adaptable educational methods are necessary to address disparate needs among clinicians in different stages of practice with different educational backgrounds, access to telehealth software, and institutional support.

Announced standardized patient (ASP) programs, in which the clinician is informed that they are encountering a standardized patient (SP) in their regular clinical practice, offer a practical method for workplace-based assessment of and skill development in telehealth. ASPs are distinct from OSCEs and other simulated trainings in that they integrate assessment and feedback into clinicians’ regular patient schedules, allowing for individualized, cost-effective, and real-time feedback in situations, and without observation from clinical leadership. For this reason, ASPs are most effectively used for assessing the clinical practice of resident and faculty clinicians with full caseloads and limited time for external training. ASP programs apply experiential learning theory, a model of action and reflection that suggests exposure to simulated clinical experiences allows learners to develop skills.¹⁴  Using ASPs via virtual modalities is relatively new, and literature on remote-based SPs is limited to the last decade,^15,16  with most studies confined to the last 3 years.^9,17,18 

Our team of medical educators, residency program directors, and clinic leaders developed a telehealth quality improvement initiative to support accessible, patient-centered care for veterans and strengthen clinicians’ communication and telehealth skills via video visits using ASPs. In this article, we describe clinician performance and experience during an ASP telehealth training program designed as both a needs assessment and an educational intervention.

Implementation of ASP visits and data collection occurred from June 2021 to April 2022 with postgraduate year (PGY) 1-3 residents and faculty clinicians practicing within the VA NYHHS Primary Care clinic.

A core team composed of 3 physicians and a research psychologist with more than 15 years of experience in medical education simulation designed 2 ASP cases that reflected the needs of the clinic’s patient population. In Case A, the ASP portrayed a 70-year-old male veteran with age-related hearing loss, social isolation, and poorly controlled hypertension. In Case B, the ASP portrayed a 60-year-old male veteran with poorly controlled hypertension, financial concerns, and depressive symptoms. To best manage the patient’s hypertension, clinicians were expected to access one of several options, including nursing visits and providing a referral to the VA Home Telehealth Clinic and VA My HealtheVet, a secure online portal that allows patients to communicate with clinicians.

Six ASPs (3 per case) received 6 hours of training from a research assistant and physician. Prior to the project, VA NYHHS primary care clinicians and internal medicine residents were informed that they would see one ASP televisit as part of a quality improvement project. Attaining Institutional Review Board (IRB) approval is not required for quality improvement projects per NYU Langone and VA NYHHS IRB policies.

Two behaviorally anchored checklists (one for each ASP case; identical except when noted) were developed to assess clinicians in 7 core competencies: 3 communication-related competencies (information gathering [4 items], relationship development [5 items], education and counseling [3 items], telehealth skills [5 items], patient activation [varies by case, refers to the patient’s internal readiness and capability to manage their health and health care¹⁹ ], patient satisfaction [4 items], and case-specific treatment and management [9 items in Case A, 8 items in Case B]; Table).⁹  Validity and reliability of checklists is supported by educational theory and has been demonstrated in simulation-based practice for more than 20 years.²⁰  Each item on the checklist was scored on a 3-point scale (not, partly, or well done, with behaviorally anchored descriptions). ASPs provided immediate verbal feedback at the end of the encounter.

Mastery scores were calculated as the mean percentage marked “well done” across items. Relationships between individual ASPs, cases, and physician types were assessed using a t test, and effect size was measured using Cohen’s d index.²¹  Individual data were shared with clinicians a week after their visit. Aggregate data on resident performance were shared with educational leadership to inform training methods. Following the project, clinicians completed a 12-item retrospective pre-post survey to assess self-perceived skills and their experience of the program using 4-point Likert scale response options and 3 open-ended questions (provided as online supplementary data).

From August 2021 to April 2022, 76 clinicians (60 residents [PGY-1-3] and 16 faculty) had an ASP visit (38 per case). Rate of participation for clinicians recruited was 100%. Median visit length was 34 minutes (range 18-64), and median wait time for the ASP was 16 minutes (range 0-47).

Aggregate analysis of ASP evaluations regarding communications skills indicates most clinicians were rated well done in 4 core competencies: information gathering (79% [60 of 76], range 0-100%), relationship development (67% [51 of 76], 20-100%), education and counseling (71% [54 of 76], 0-100%), and patient satisfaction (86% [65 of 76], 25-100%; Table). Performance was notably lower in telemedicine skills (38% [29 of 76], 0-100%), case-specific treatment and management (57% [43 of 76], 11-89%), and patient activation (48% [36 of 76], 0-100%). Scores were significantly different between Case A and Case B in information gathering (A=85%, B=74%; P=.046; SD=23%; effect size [Cohen’s d]=0.478), relationship development (A=76%, B=58%; P=.002; SD=28%; effect size [Cohen’s d]=0.643), and patient satisfaction (A=91%, B=80%; P=.015; SD=20%; effect size [Cohen’s d]=0.550). There were no significant differences in t test scores between type of clinician (resident or faculty), thus combined scores are shown in the Table.

Clinicians’ telemedicine skills were highly variable and less strong overall. Very few clinicians confirmed patient identifiers for these new patient cases (7%, 5 of 76), and few used nonverbal behavior to enrich communication on camera (38%, 29 of 76), maintained appropriate computer etiquette (39%, 30 of 76), or optimized technical aspects of the virtual encounter by requesting the ASP adjust their video camera to better see their face and environment (22%, 17 of 76). Most (79%, 60 of 76) exhibited comfort and confidence using the video interface.

Use of clinical resources varied by case. For Case A, 34% (13 of 38) of ASPs received a referral for mental health counseling and 79% (30 of 38) for audiology. For Case B, 63% (24 of 38) were referred to the department of social work. Regarding hypertension management, 92% (70 of 76) of clinicians in both cases offered to send a blood pressure monitor for home use and 79% (60 of 76) recommended a strategy to monitor blood pressure. Clinicians provided referrals to the home telehealth clinic for 75% (57 of 76) of all ASPs and to the My HealtheVet online system to 66% (25 of 38) of ASPs in Case A and 45% (17 of 38) in Case B.

Patient activation was notably low. Less than one-third of ASPs expressed confidence in the plan to treat hearing loss (29%, 11 of 38) in Case A or reduce the salt in their diet (32%, 12 of 38) in Case B. Only 37% (14 of 38) of ASPs felt that the treatment plan addressed their psychological needs, a gap which was reflected by low referral rates to mental health counseling. Most SPs (63%, 48 of 76) felt confident that they could manage their hypertension because of the encounter.

Ninety-nine percent of ASPs (75 of 76) reported they would recommend the clinician to a friend for the clinician’s professionalism and 96% (73 of 76) indicated they would recommend based on the clinician’s communication skills.

In their feedback following the encounter, most ASPs (86%, 65 of 76) commented positively on the sound and quality of the video. A few (11%, 8 of 76) noted minimal video interruptions and only 2 had to adjust the audio mechanism on their end. Several ASPs (36%, 27 of 76) suggested the clinician could improve on their ability to maintain eye contact on camera and 28% (21 of 76) indicated they would have benefitted from receiving more information on the VA’s telehealth resources, particularly the My HealtheVet portal. ASPs in Case A noted that the clinician often asked about financial hardship and depression but did not give them a referral to social services or counseling (provided as online supplementary data).

Sixty-eight of 76 clinicians responded to a feedback survey (89% response rate). Ninety-one percent (63 of 68) agreed the experience was well designed and engaging, and 88% (60 of 68) agreed it was an effective way to reinforce good habits in health care communication. Extensive commentary from clinicians indicated the program was useful and gave context for items to include in future training (provided as online supplementary data).

ASP ratings indicate that clinicians performed satisfactorily in several elements of the telemedicine and communication skills domains. However, the literature suggests that opportunities to further improve telehealth practices in primary care remain.²²  Until recently, most faculty and resident physicians had limited access to training for patient-centered care on video. Even 2 years into the pandemic, the consequence of this educational gap was evident in clinicians’ highly variable performance in telemedicine skills, as some were marked “well done” on all aspects of the televisit and others did not perform well on any individual competency. Qualitative data from ASPs on key components of the relationship development competency (acknowledging emotions appropriately and using nonverbal behavior to enrich communication) suggests that the virtual platform has had a negative effect on clinicians’ ability to build relationships with patients. As current and previous work demonstrates, telehealth requires distinct communication and assessment skills that necessitate dedicated pedagogical tools and training.²³ 

As part of a targeted, workplace-based assessment and training program, the ASP methodology offers a dynamic perspective on clinicians’ actual practice. Integrating clinician and ASP feedback revealed areas for further assessment and training in patient activation, counseling, and core telehealth skills. While patient satisfaction scores were high, patient activation scores were consistently low. It is unclear whether this disparity is related to the modality of the visit; more work is needed to understand patient activation in virtual care. In-person simulation assessment suggests performance in patient activation is often lower than in communication and satisfaction competencies, and patient activation and patient-clinician communication are correlated.^24,25 

Barriers to providing quality telemedicine care identified by VA clinicians (eg, issues maintaining eye contact, lack of personal connection, limited referral to existing VA telehealth resources) are prevalent across most health systems.^22,26,27  Much of the qualitative commentary provided by clinicians was similar to the recommendations for best practices (online supplementary data), indicating that our training is relevant to the broader medical community. Training clinicians to efficiently assess patients’ environments for safety hazards, home support systems, and complete medication reconciliation may be most effective for virtual care. In past studies, physicians have expressed that an advantage of televisits was seeing patients’ home environments because it enhances information gathering and improves rapport.^28,29  Additionally, brief trainings in or reminders about referral to VA home telehealth services, communication, and resources such as the My HealtheVet portal may reduce the burden of care follow-up on clinicians.^26,30 

As the health care landscape evolves, telehealth care will likely remain a permanent part of care management. Telehealth can eliminate barriers to accessing care, reduce hospitalizations, and improve continuity of care and clinical outcomes.^6,7,28,31,32  Educational programs must adapt to changing expectations and reach a diverse set of learners, particularly practicing clinicians. To extend the influence of our work, we developed a comprehensive training toolkit that can be used not only for improving uptake of new or emerging skills sets (ie, telehealth), but also for large-scale evaluation and standardization of common communication skills and clinic-specific resources. The toolkit includes feasibility information (costs, case information, and necessary materials) and is available as online supplementary data.

Literature on postgraduate interprofessional training indicates that announced workplace-based training such as our ASP program can reduce implementation costs, benefit local organizational learning, and increase the accessibility of training for clinicians.³³  Furthermore, in situ simulations are understood to have greater fidelity—the degree of similarity between true clinical practice and simulation—than simulation-based education that takes place outside the clinic setting.³⁴  Announced simulation can be less intimidating for learners and less logistically challenging than unannounced simulation, and several studies have found no notable difference in learning outcomes between the two. Future work will assess the effect of ASP training on improving clinical outcomes with clinician patient panels or clinical systems. We plan a subsequent iteration of this work to assess retention of skills and changes in practice with the same clinician participants. This curriculum has been integrated into standard resident training.

Limitations of this work include interrater variability, as different ASPs were trained for each case, and potential lack of generalizability, as all participants were associated with a single institution. A survey or focus group of clinicians following reception of their individual feedback reports might have provided additional insight into their experience of the ASP feedback.

A quality improvement project used ASPs to assess and provide detailed feedback on clinicians’ communication, telehealth, and clinical care skills within a VA Primary Care clinic. Faculty and resident clinicians responded well to this experiential workplace-based learning initiative.

The authors would like to thank Rita Kobb of the Office of Connected Care for her valuable input and our standardized patients at the New York Simulation Center for the Health Sciences (NYSIM), a Partnership of The City University of New York and NYU Langone Health.

The online version of this article contains a full project toolkit, including the surveys used in the study and further data, and representative themes and comments of announced standardized patient (ASP) feedback to clinician and clinician post-ASP reflections.

Funding: This project was made possible by funding from the Veterans Health Administration Office of Connected Care Grant OCC-21-01.

Conflict of interest: The authors declare they have no competing interests.

This work was previously presented at the Society for General Internal Medicine Annual Meeting, April 6-9, 2022, Orlando, FL, and the New York University Grossman School of Medicine Medical Education Conferences, October 2021 and June 2022, New York, NY.