Virtual Supervision in Graduate Medical Education: A Systematic Review

Supervision is a key component of graduate medical education (GME) for the delivery of safe and effective patient care as well as the acquisition of knowledge and skills by trainees.¹  The significant increase in telemedicine, largely driven by the COVID-19 pandemic, has fostered a growing interest in virtual supervision in GME.^2,3 

The Accreditation Council for Graduate Medical Education (ACGME) defines 2 forms of supervision: direct and indirect.⁴  In direct supervision, the supervising physician is colocated with the trainee and the patient. In indirect supervision, the supervising physician is immediately available by means of telephonic or electronic modalities but not physically present.⁵ 

Virtual supervision is defined as a form of hybrid supervision in which non-colocated supervisors and trainees interact via synchronous audio and/or video modalities while the patient is being seen.^6,7  Forms of virtual supervision include video teleconferencing, phone calls, text-based approaches, and mobile apps that facilitate virtual supervision in real time.⁷  Virtual supervision offers the opportunity to provide clinical supervision, feedback, and trainee learning and engagement, while also addressing some of the challenges associated with direct supervision, such as scheduling conflicts and geographic barriers.⁸ 

In July 2022, the ACGME published guidelines regarding the use of telecommunication technology for supervision in GME.⁹  These guidelines include one common program requirement that postgraduate year (PGY) 1 residents should start with direct supervision but may progress to supervision via telecommunication technology, depending on specialty-specific conditions set by review committees. However, the current review committee guidelines vary significantly by specialty, and some review committees have not yet determined specialty-specific guidelines. For example, anesthesiology, neurology, medical genetics, obstetrics and gynecology (OB/GYN), and urology permit the use of virtual supervision with history-taking, patient examination, assessment, and counseling, but not with invasive procedures such as the conduct of anesthesia and labor and delivery. Urology, ophthalmology, and OB/GYN permit the use of virtual supervision in ambulatory and acute care settings but not in the operating room. Other specialties such as family medicine, pathology, pediatrics, and plastic surgery permit virtual supervision but do not have any specific guidelines. The generally cautious approach to resident surgical education in many specialties is echoed in the United States Department of Veterans Affairs (VA) guidelines, which require the supervisor to be colocated for all operating room procedures.¹⁰ 

The current literature on virtual supervision in GME is not well described. To this end, we investigated the following research question: In ACGME-accredited specialties, what is the impact of virtual supervision on trainee education, patient care, and patient satisfaction? We hope this information can help national GME stakeholders such as the ACGME and VA refine their guidelines for virtual supervision and help program-level GME leaders formulate appropriate virtual supervision strategies for their faculty and trainees.

The reporting of this systematic review was guided by the standards of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement.¹¹  In consultation with a reference librarian, we searched 2 databases on December 19, 2022 for eligible studies: PubMed (January 1950 to December 19, 2022), and EMBASE (January 1950 to December 19, 2022). MEDLINE, included within PubMed, was also searched via OVID to ensure comprehensive coverage (January 1946 to December 19, 2022). Our search included database-specific thesaurus terms and keywords related to virtual supervision in GME (online supplementary data).

The inclusion criteria were full-text articles published in an English-language, peer-reviewed journal that reported the use of supervision through virtual modalities in GME, namely internship, residency, and fellowship programs in ACGME-accredited specialties and subspecialties. We included all study types, including randomized control trials, cohort studies, case reports, case series, cross-sectional studies, and quasi-experimental studies. The exclusion criteria were studies with in-person direct supervision as defined by ACGME, supervisors who were not credentialed physicians, and non-GME trainees such as medical students. We also excluded studies that focused on online learning, virtual simulation, telesurgery, and telehealth platforms for communication with patients, as well as studies that did not report empirical outcomes. In addition, we excluded abstracts, conference proceedings, non-peer-reviewed manuscripts, and non-English language studies without translations.

The authors deduplicated eligible studies in EndNote (Clarivate Analytics) and imported the studies into the systematic review software Covidence for screening, full-text review, and data extraction. The screening and selection process is displayed in a PRISMA flowchart (Figure). Two investigators (J.H., C.J.S.) independently conducted title/abstract screening, full-text review, and data extraction in Covidence following screening training and screening test sessions. Disagreements were resolved by the senior investigator (P.B.G.).

The authors developed a data template in Covidence to extract relevant information, including year of publication, location, study design, study timing, specialty, study setting, type of care performed, number of patients and trainees involved, and the postgraduate years of the trainees. We also collected information on the type of technology used for virtual supervision, as well as the objective and outcomes of the study. Two investigators (J.H., C.J.S.) independently appraised the methodological quality of the studies using the Mixed Methods Appraisal Tool (MMAT), and disagreements were resolved by the senior investigator (P.B.G.). The MMAT appraises studies based on 5 questions assessing the sampling strategy, outcome measurement, confounders, and statistical analysis of the study.¹²  Studies were scored on a scale of 1 to 5 based on the MMAT.

The initial database search yielded 5278 articles. Following duplicate removal, title and abstract screening, and full-text review, 26 articles were included in our review (Figure). The Cohen’s Kappa statistic for interrater reliability was 0.112 for title and abstract screening and 0.571 for full-text review.

Study design included quasi-experimental studies (35%, 9 of 26),^13-21  cohort studies (27%, 7 of 26),^22-28  cross-sectional studies (19%, 5 of 26),^29-33  case reports or case series (15%, 4 of 26),^34-37  and a randomized control trial (4%, 1 of 26)³⁸  (Table 1). The publication year of the articles ranged from 1999 to 2022. Most studies were conducted in the United States. Other studies were conducted in the United Kingdom,^21,27  Norway,²⁰  Italy,¹⁶  Nepal,²³  the Czech Republic,¹⁵  and Singapore.²²  Twenty studies were prospective (77%),^{13-24,26,28-33,38}  while 6 studies were retrospective in design (23%).^25,27,34-37  The objective of the studies ranged from comparing safety and quality of care between virtual supervision and direct supervision (31%, 8 of 26),^{15,17,18,20,25-27,38}  and determining the feasibility of using virtual supervision for remote consultations (23%, 6 of 26)^{13,16,21,23,24,34}  to such aims as describing the effectiveness of virtual supervision in trainee education (19%, 5 of 26),^{14,19,22,35,37}  characterizing resident acceptance of virtual supervision (15%, 4 of 26),^30-32,36  and understanding patient access and satisfaction levels with virtual supervision (12%, 3 of 26).^28,29,33 

A total of 1561 patients and 160 trainees were included. Specialty of focus included dermatology,^{14,16,18,26,28,33}  neurosurgery,^15,34,35  orthopedics,^17,22,37  radiology,^19,30  psychiatry,¹⁹  ophthalmology,^23,27  otolaryngology,^24,38  urology,^29,32  emergency medicine,¹³  pathology,²⁰  neurology,³¹  anesthesiology,²¹  and general surgery.³⁶  The supervisors and trainees were in the same specialty except for one study in which toxicology physicians supervised emergency medicine residents.¹³ 

Study location varied among the operating room (10 of 26, 38%),^{17,23,27,29,32,34-38}  inpatient settings (7 of 26, 27%),^{14,15,19,22,24,26,28}  outpatient clinics (4 of 26, 15%),^16,18,21,25  the emergency department (4 of 26, 15%),^13,30,31,33  and the autopsy room (1 of 26, 4%).²⁰  Virtual supervision was used for clinical examinations (14 of 26, 54%),^{13-16,18,19,21,22,24-26,28,31,33}  surgical procedures (11 of 26, 42%),^{17,22,23,27,29,32,34-38}  a diagnostic examination (1 of 26, 4%),³⁰  and autopsies (1 of 26, 4%).²⁰  For studies involving in-office procedures or surgeries in the operating room, virtual supervision was used to supervise arthroscopy,³⁷  repair of retinal detachments,²⁷  dermatology inpatient consultations,¹⁶  neuro-endovascular surgeries,³⁴  and endoscopic sinus surgeries.³⁸ 

Thirteen studies reported the PGY of the trainees, which included PGY-1 (6 of 13, 46%),^{19,20,24,26,28,30}  PGY-2 (7 of 13, 54%),^13,26,28-32  PGY-3 (6 of 13, 46%),^{16,17,25,26,28,31}  PGY-4 (3 of 13, 23%),^25,28,31  PGY-5 (2 of 13, 15%),^17,31  PGY-6 (1 of 13, 8%),³¹  and PGY-10 or 11 (1 of 13, 8%),²⁷  with some studies targeting multiple PGY levels (Table 1). For surgical procedures, studies used virtual supervision only for PGY-2 with at least 4 months of training in endourology suites, PGY-3, PGY-5, or fellows.^{17,23,29,32,34}  In another study, trainees had to have performed at least 4 endoscopic sinus surgeries in order to participate in virtual supervision.^27,38 

Synchronous communication between the supervisor and trainee was facilitated through various modalities such as videoconferencing software (15 of 26, 58%),^{17,19,20,23-25,29-34,36-38}  telephone (5 of 26, 19%),^{14,15,18,21,22}  and mobile apps (5 of 26, 19%)^{13,16,26,28,35}  (Table 2).

Virtual supervision offered innovative tools and techniques to enhance trainees’ understanding of complex procedures, but its implementation was not without technical challenges. For example, virtual supervision encouraged supervisors to be more specific with their feedback rather than using hand signals or gestures, which helped trainees better understand how to locate anatomical landmarks during flexible cystoscopies.²⁹  Other studies used videoconferencing software with augmented reality features with which supervisors could annotate the live surgical field,^34,38  virtually “touch” the tissues,¹⁷  or use a screen pointer³⁶  that enhanced the trainees’ ability to visualize and understand complex medical procedures. At the same time, technical difficulties, such as the lack of reliable internet connectivity,^14,23,28  low image quality,^14,23  low battery life of videoconferencing devices,³⁵  and inability to adjust the frame,^23,35  posed obstacles to effective teaching.

In 4 studies, virtual supervision provided flexibility in scheduling and location, allowing for increased patient volume in training and trainee autonomy. For example, a toxicology residency program used virtual supervision to work with distant institutions, decreasing the program’s reliance on poison control centers and increasing the number of bedside consultations for trainees.¹³  Virtual supervision was also useful for night call, during which supervising radiologists at their home work stations communicated with trainees.¹⁹  Another study explored the use of international teleradiology attending radiologist coverage (ITARC) for a residency program, where radiologists living abroad worked normal daytime hours to cover the night shifts of the home institution.³⁰  On UK bank holidays and weekends, an eye center utilized virtual supervision to supervise trainees performing retinal detachment repair, noting that virtual supervision facilitated trainees’ progression toward increased independence and confidence.²⁷ 

Virtual supervision also helped increase trainees’ exposure to patients in rural or underserved areas while eliminating the burden of travel. One psychiatry residency program used virtual supervision to add an elective during which trainees worked with veterans in rural Colorado.²⁵  The study found that virtual supervision improved patient access to care and may increase the recruitment of qualified psychiatrists to work with rural populations. Finally, 3 studies emphasized that virtual supervision provided additional educational opportunities by allowing trainees to rewatch video recordings taken during virtual supervision to review clinical examination skills and surgical techniques after the patient encounter.^23,24,29 

Four studies, which assessed educational outcomes based on self-reported questionnaires, found mixed opinions on the satisfaction of virtual supervision by supervisors and trainees.^30-32,37  Two studies noted a self-reported increase in competency levels of trainees post-virtual supervision and a decrease in stress and anxiety levels related to on-call shifts.^30,32  Another study indicated that neurology supervisors and their trainees preferred direct supervision to virtual supervision, expressing concerns that telephonic modalities were not sufficient for supervising the trainee’s acquisition of patient history and clinical skills.³¹  An additional study on the use of virtual supervision for teaching arthroscopy skills concluded that virtual supervision cannot replace direct supervision for basic surgical and decision-making skills.³⁷ 

Studies investigating the quality and effectiveness of patient care communication through virtual supervision reported mixed results (Table 2).^{13,15-18,21,22,24,34,38}  In some cases, virtual supervision allowed for quick and efficient communication between supervisors and trainees when clinical decisions needed to be made due to the minimal time needed to transmit images and videos.^16,24  Furthermore, real-time communication between supervisor and trainee improved the precision of the medical history obtained by the trainee, prompting the administration of an additional medication.¹³  Another study suggested that virtual supervision expedited patient care by allowing immediate evaluation and escalation of care, which is crucial for time-sensitive situations such as airway evaluations.²⁴  In other cases, virtual supervision limited the supervisor’s ability to observe physical cues and aspects of patient care that are not captured through a screen. For example, one study noted the inability of supervisors to use olfactory and tactile senses during remote autopsies.²⁰ 

Three studies assessed the agreement in diagnosis and treatment plan between supervisors and trainees when using virtual supervision, reporting a kappa coefficient ranging from 0.747 to 0.94.^16,24,26  In one study, 6% of discordant laryngoscopy examinations required changes in clinical management.²⁴  Another study reported a lower diagnosis concordance rate for complex skin conditions when using images to transmit clinical information.¹⁶  It is noteworthy that the absence of an independent grader²⁴  may have influenced the concordance rate, and technological limitations may have influenced the supervisor’s assessment.^16,24,26 

Only 3 studies examined patient care outcomes when comparing direct supervision and virtual supervision.^18,27,34  These studies noted no adverse events or significant difference in patient outcomes between direct supervision and virtual supervision when comparing the complication rates of retinal detachment repairs,²⁷  complication rates of complex neuro-endovascular procedures,³⁴  and differential diagnoses and disposition plans in dermatology.¹⁸ 

In some cases, there was an increase in time spent on patient care for virtual supervision compared to direct supervision.^15-17,38  Reasons included technical difficulties, set-up time, and increased didactic teaching when using virtual supervision.

Several studies surveyed patient satisfaction with care provided when virtual supervision was utilized^{13,21,23,28,29} ; however, none of these studies explicitly compared patient satisfaction between virtual supervision and in-person supervision. Most patients felt comfortable with the images and videos being taken during their clinical examinations and did not feel that their privacy was compromised.^13,29  Some patients felt that they were getting a more thorough and careful examination due to the supervision of a more experienced physician.²⁹ 

The MMAT was used to critically appraise the quality of the included studies. Limitations were noted in sampling strategy,^14,23,25,38  outcome measurement,^21,25,31,33  and inclusion of confounding factors.^{14-16,18,19,21,22,26,29,31,33,35,37,38}  Overall, 10 studies scored a 5 out of 5,^{13,17,20,24,27,28,30,32,34,36}  10 studies scored a 4 out of 5,^{15,16,18,19,22,23,26,29,35,37}  and 6 studies scored a 3 out of 5 (Table 2).^{14,21,25,31,33,38} 

We systematically reviewed 26 studies to assess the impact of virtual supervision on trainee education, patient care, and patient satisfaction in GME. Virtual supervision was used across several medical specialties and PGY levels for clinical examinations and surgical procedures, employing communication channels such as telephone, videoconferencing software, and mobile apps. The existing evidence base suggests that virtual supervision in GME has potential to promote communication, facilitate trainee autonomy, and expand educational opportunities by allowing collaboration between clinicians who are geographically distant. However, technical difficulties posed obstacles to effective patient care and communication, including limitations in conveying certain aspects of patient care (eg, autopsies that require olfactory and tactile senses and differentiating complex skin conditions) that necessitate physical presence.

The evidence base for virtual supervision in GME could be strengthened in several areas. First, many studies had methodological limitations such as lack of control groups, small sample sizes, and nonrandomized study designs, which made it difficult to generalize the results to larger populations. Studies often utilized surveys lacking robust validation and reported self-perceived educational outcomes of trainees, which can be unreliable and subjective.^39,40  Furthermore, most studies had a duration of less than 6 months and did not assess the long-term impact of virtual supervision on resident learning or patient outcomes. Finally, although our search strategy encompassed all GME specialties and subspecialties, we found a lack of primary studies for certain fields, including OB/GYN, pediatrics, physical medicine and rehabilitation, family medicine, and internal medicine. Longitudinal, randomized, and larger-scale studies can help ensure that the available evidence is more comprehensive and reliable.

In addition, we suggest that future research explore the following questions:

1. When is the use of virtual supervision not appropriate due to its potential risks to patient care? As highlighted in the arthroscopy study,¹³  this question is especially important in surgical education, and can inform the development of more detailed ACGME supervision criteria.

2. What are appropriate levels of clinical experience for a trainee for virtual supervision and how may this vary across specialties? Many studies did not report the PGY levels of the trainees, which render preliminary understanding of these questions challenging. Comparing the use of virtual supervision across various PGY levels can provide insight into how virtual supervision can be effectively incorporated into GME at various stages of training.

3. What is the relationship between varying amounts of virtual supervision and its impact on trainee education and patient outcomes? Achieving an appropriate balance between in-person and virtual supervision can ensure that trainees receive adequate clinical experience while also benefiting from the flexibility and efficiency of virtual supervision.

4. What is the cost-effectiveness of virtual supervision? This is important given the high cost of various teleconferencing devices and software systems.³⁸ 

The findings of this systematic review can assist ACGME review committees and the VA in updating their guidelines for virtual supervision in GME. Establishing minimum requirements for face-to-face interactions between supervisors and trainees before utilizing virtual supervision will promote supervisor-trainee relationships and establish clear expectations and goals for virtual supervision. In addition, virtual supervision may be better suited for more experienced trainees, allowing them to work more independently.^17,34  Hence, we recommend programs specify competencies that trainees must demonstrate via direct supervision before transitioning to virtual supervision. Additionally, comprehensive guidelines covering topics such as emergency procedures, device usage, troubleshooting, and effective communication techniques can also be useful, as supervisors and trainees often experience a learning curve when using virtual supervision.³⁸ 

This review has several limitations. First, we excluded articles published in languages other than English and gray literature, such as conference proceedings. Second, we did not hand-search references of included studies. Third, the reproducibility of MMAT ratings is limited by the authors’ judgements about the quality of the study design. Fourth, we did not explore databases such as PsychINFO, potentially overlooking relevant studies. Finally, the kappa coefficient for screening was low, indicating a possibility of missing relevant citations. However, we conducted a screening training and screening test session beforehand to mitigate bias and subjectivity.

This systematic review found that virtual supervision is generally technologically feasible and applicable in a wide range of specialties, trainee levels, and settings. Virtual supervision may have benefits such as increased training opportunities, trainee autonomy and communication, and can be particularly useful in the later stages of training. However, to ensure effective trainee education, future studies should employ large sample sizes, objective outcome measures, and rigorous methodology to expand the evidence base.

The authors would like to thank Andrew Creamer from Brown University for drafting and conducting the search strategy for this project.

The online supplementary data contains the search strategy used in the study.

Funding: The authors report no external funding source for this study.

Conflict of interest: Paul B. Greenberg, MD, MPH, was ex-officio member of the Accreditation Council for Graduate Medical Education (ACGME) Medically Underserved Areas and Population Advisory Group (MUA/P) in 2022-2023 and received accommodations for the MUA/P meeting, and is a Council Member, National Board of Medical Examiners (NBME), and received accommodations and meals for the NBME annual meeting.

Disclaimer: The views expressed here are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs or the US government.