A Systematic Review of Virtual Subinternships in Surgical Education: Impact on Curriculum Delivery and Learner Outcomes

The landscape of surgical education has undergone a pivotal transformation in response to the challenges posed by the COVID-19 pandemic.¹  Traditionally, subinternships have been vital for medical students across all surgical specialties to gain critical hands-on experience and make informed career decisions.²  Surgical subspecialties, already relying heavily on away rotations as part of the residency application process, were particularly invested in converting to virtual formats.^1-5  Innovative virtual curriculums have emerged to provide an immersive and comprehensive overview of surgical fields.^6-8  This transition to virtual subinternships (VSIs) is not just a stopgap but a progressive step in medical education, offering a range of benefits from removing geographical limitations to reducing costs and granting learning flexibility. However, this paradigm shift also introduces challenges, particularly in maintaining educational quality during virtual rotations.

This review presents a critical analysis of VSIs across diverse surgical fields, examining curriculum content, delivery methods, and evaluation techniques, and highlighting disparities in the existing literature. Given the pivotal role that subinternships play in both undergraduate medical education and graduate medical education for evaluating and selecting residency applicants, understanding the value and limitations of these virtual rotations is crucial. This synthesis is needed to inform program directors and educators whether VSIs can be successfully maintained or reimplemented, especially in contexts where in-person learning is not feasible. By assessing these varied contributions, this article provides a comprehensive perspective on the structure of VSIs in surgical education, emphasizing the importance of these pioneering studies and charting a course for future exploration and refinement in this evolving area of medical training.

Our research question examines how curriculum delivery, learner outcomes, and interpretation of program fit occur within the VSI model in surgical education, which emerged during the COVID-19 pandemic. To address this question, we conducted a systematic review of the literature, following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines to identify and synthesize studies that examined the implementation, content, and evaluation of VSIs across diverse surgical specialties (Figure 1). A systematic review was chosen to ensure a comprehensive and unbiased synthesis of available evidence, providing a structured evaluation of VSI effectiveness and limitations. This review was not registered.

A comprehensive literature search of English-language articles was performed across PubMed, Web of Science, Google Scholar, and Embase databases from January 2020 to June 2023. The search strategy included keywords and Medical Subject Headings (MeSH) terms related to “virtual sub-internships,” “surgical education,” “COVID-19,” and specific surgical specialties (see Box). The search strategy was developed iteratively, with multiple revisions to ensure comprehensiveness and accuracy. The process included refining search terms and incorporating feedback from preliminary searches to capture the broadest range of relevant studies.

Studies were included if they were original research articles focusing on VSIs across any surgical specialty, and described the curriculum content, delivery format, evaluation methodologies, and outcomes (Figure 1). Studies were excluded if they were non–original research such as reviews, editorials, or commentaries; if they did not relate to surgical specialties or VSIs; or if the article did not include a full description of the curriculum content and outcomes. The rationale for these criteria was to focus on original research that provides direct evidence about the effectiveness and structure of VSIs.

Two investigators independently screened titles and abstracts to identify eligible studies. Full-text reviews were then conducted to confirm eligibility. Discrepancies were resolved through discussion and consensus, with a third investigator consulted if necessary. The PRISMA flow diagram (Figure 1) illustrates the study selection process aimed to minimize bias and enhance the reliability of the study selection.

Data were extracted using a standardized form, capturing details such as authors, publication year, study design and duration, and a comprehensive description of each VSI, including learning objectives and educational outcomes (Table). The extracted data were independently reviewed by 2 researchers to ensure accuracy. Any disagreements were discussed and resolved through consensus meetings. Due to the heterogeneous nature of the articles and the lack of standardization, a meta-analysis would not be appropriate.

Themes were developed through a thematic analysis approach. Two researchers independently coded the data to identify recurring themes related to curriculum content, delivery methods, and evaluation techniques. Regular meetings were held to discuss and resolve any discrepancies, enhancing the reliability and strengthening the levels of content and construct validity evidence for the findings. The thematic analysis was iterative and recursive, ensuring a thorough understanding and interpretation of the data.

From the 1436 total articles identified using the specified search terms, 13 of the 57 assessed for eligibility after screening, met the criteria for inclusion in this review. The remaining 44 articles were excluded because they focused on in-person subinternships (n=4), pertained to virtual learning and lectures during residency training without discussion regarding subinternships (n=14), were single-day events (n=9), or were theoretical without outcomes listed (n=17). Of the 13 studies selected, 4 were published in otolaryngology, 3 in plastic surgery, 2 in urology, 2 in orthopedic surgery, 1 in vascular surgery, and 1 in neurosurgery.^5,9-21  Study size ranged from 6 to 61 participants and lasted on average 1 to 4 weeks across institutions. We evaluated the quality of the included studies using a thematic analysis approach, which highlighted variability in study design, sample size, and evaluation methods. The lack of standardized assessment tools across studies was a recurring limitation, which complicates direct comparisons between programs. Additionally, none of the included studies utilized questionnaires with established levels of validity evidence, underscoring the need for standardized methodologies in future VSI evaluations.

In our analysis of VSI studies, we identified 2 central themes in course objectives that are outlined in the Table. The majority of the studies, accounting for 77% (10 of 13), focused on a dual approach combining specialty-specific knowledge with an introduction to the unique aspects of their specialty or program. Studies by Byrnes et al, Laitman and Londino, and Lee et al deviated from this trend.^10,15,16  Objectives in these studies prioritized personal growth and alignment with the residency program’s culture over the explicit goal of knowledge acquisition.

Key patterns emerged in content and delivery methods and are outlined in online supplementary data Table 1. Interactive learning, a core component in all programs, included modalities like Q&A panels, group discussions, and case reviews as described in the flow diagram represented by Figure 2. These were often coupled with didactic sessions, which were integrated into existing departmental education events such as grand rounds or tumor board in 85% (11 of 13) of studies. Self-paced educational materials, which allow students to learn outside of scheduled hours, were highlighted in the curriculum of 62% (8 of 13) of programs. These materials ranged from prerecorded lectures to self-guided modules. Small group formats, used to facilitate active participation and teamwork, were emphasized in approximately 69% (9 of 13) of the studies. This approach was particularly valuable in programs with a larger number of students, as exemplified in the study by Hoffman et al, which grew from 11 to 27 participants over a 2-month period.¹⁴ 

Figure 3 highlights common curricular components across all studies. Forty-six percent (6 of 13) of programs offered virtual yet immersive insights into surgical procedures through operating room tours and live-streamed surgeries. Less frequently included content involved telehealth clinic exposure, featured in only 23% (3 of 13) of studies, with some programs omitting this due to HIPAA (Health Insurance Portability and Accountability Act) concerns. Hands-on simulations or workshops appeared in 31% (4 of 13) of studies, indicating diverse approaches to experiential learning. Interactive platforms like discussion boards (15%, 2 of 13) and social media (8%, 1 of 13) were utilized in a few studies, demonstrating innovative methods to enhance collaboration and engagement in the virtual learning environment.

An analysis of rotation durations revealed that 2-week rotations were the most common, representing 46% (6 of 13) of programs. This includes 2 of the 3 studies reported by Reghunathan et al. One-week rotations comprised 8% (1 of 13), while 4-week rotations accounted for 31% (4 of 13), including one study by Reghunathan et al. The remaining program had an unspecified duration. Most rotations (62%, 8 of 13) occurred during the summer (May to August), while 39% (5 of 13) took place in the fall (September to November). Notably, Reghunathan et al’s research encompassed 3 studies, each with distinct durations, contributing to both the 2-week and 4-week rotation categories.

Online supplementary data Table 2 outlines the assessment and curricular outcomes of each study. In over half of these studies (54%, 7 of 13), student knowledge was subjectively self-assessed versus objectively scored (15%, 2 of 13) or not specifically assessed (31%, 4 of 13). This primarily involved self-assessment tools and questions gauging students’ comfort and familiarity with content topics in the specific surgical specialty. The 2 studies utilizing objective tools to evaluate learning outcomes, referenced above, reported significant improvements in student scores following participation in the VSI.^11,21  However, in the remaining 4 studies, there was a noticeable shift in focus. These programs did not specify outcomes related to medical content. Instead, they placed a greater emphasis on ensuring that students understood program details and fit. Seventy-seven percent of programs (10 of 13) included a final presentation by each student as part of the curriculum. This component served as an opportunity for students to showcase their knowledge and experience, akin to an end-of-rotation presentation typically seen in in-person programs. This element of the VSIs not only provided a platform for students to demonstrate their learning but also facilitated a familiar evaluative process for faculty to assess student progress and understanding.

All of the programs reported high levels of student satisfaction or a willingness to recommend the VSI, reflecting positively on the structure, activities, and overall utility of these programs, especially considering the limitations imposed by the COVID-19 pandemic. The most highly rated aspects across studies were interactive and realistic learning opportunities, which included telehealth sessions, virtual operating room, consult sessions, case presentations, and online modules. These components were consistently appreciated by students for their engagement and relevance. While the study by Patel et al stood out for its inclusion of at-home suture stations for knot-tying practice, it was observed that student understanding of these techniques did not show significant improvement compared to the understanding of vascular concepts presented in a didactic format.¹⁸ 

Students universally felt they had gained sufficient insight into the dynamics of each individual residency program through the VSI. All studies reported student satisfaction with their faculty and resident interactions, indicating effective virtual networking and relationship-building opportunities. Four studies specifically reviewed students’ perceptions of the residency program following VSI participation.^9,11,13,19  These studies reported an increase in comfort and familiarity with the residency program, leading to an overall positive perception of the program. The study by Reghunathan et al, which encompassed 3 institutions, highlighted a notable shift in student perceptions post-VSI.²²  Students were more inclined to view the program’s residents favorably, less likely to consider research opportunities or location as drawbacks, and ranked the programs an average of 4.7 points higher post-rotation compared to pre-rotation. This outcome suggests a significant impact of VSIs on students’ perceptions and preferences regarding residency programs.

Analysis of faculty feedback from 54% (7 of 13) of VSI programs revealed mixed outcomes. Three studies reported faculty difficulties in evaluating virtual rotators’ personalities, interpersonal skills, and work ethic.^11,13,18  Studies by Dinis et al and Patel et al noted a significant challenge for faculty in gauging students’ suitability for their programs, a contrast to students’ more positive perceptions of gaining program insights from the VSI experience.^11,18  Conversely, studies by Egro et al and Hoffman et al indicated less difficulty in evaluating program fit through VSI.^12,14  In the study by Hoffman et al, 83% of faculty felt confident in determining student fit for their program, with 33% stating that their VSI experience would directly influence their residency ranking decisions.¹⁴  These diverse faculty experiences highlight the nuanced challenges and potential benefits of assessing student competencies and fit within virtual environments.

Our analysis of VSI programs revealed significant incorporation of social and mentorship elements. In terms of social interactions, 69% (9 of 13) of the programs integrated purely social virtual events into their curricula. These events varied in format, including weekly social networking opportunities with residents discussing life outside of the hospital in specific cities and meet-and-greet events. These social gatherings often involved both faculty and residents, fostering an informal and engaging environment for participants.

Mentorship was another key component in 54% (7 of 13) of the studies. The structure of mentorship varied: 3 programs assigned both faculty and resident mentors, 2 programs had only resident mentors, and 2 others organized specific sessions dedicated to mentorship without individual assignments. All 7 programs that included mentorship cited it as a valuable aspect of the curriculum. In the 3-institution analysis by Reghunathan et al, institutions with one-on-one mentorship saw a higher number of students achieving their mentorship goals with faculty, compared to the institution without a structured mentorship component.¹⁹  This suggests the effectiveness of direct mentorship in fulfilling students’ mentorship expectations in a virtual setting.

The integration of VSIs across surgical specialties during the COVID-19 pandemic maintained educational momentum while ensuring learner engagement. Essential aspects of traditional in-person rotations, including interactive learning sessions, clinical and surgical lectures, and resident and faculty engagement were replicated to maintain the integrity of medical education during tumultuous times.^19,23,24  High student satisfaction with VSIs was observed, though faculty reported challenges in assessing student skills and program fit in the virtual format.^20,21 

The various limitations on in-person away rotations may differentially affect applicant cohorts, potentially impacting diversity in residency programs.²⁵  VSIs offer enhanced accessibility in medical education by removing geographical and financial barriers.^26,27  Studies like those by Yellin et al and Reghunathan et al highlight how VSIs have allowed for broader participation, especially during the COVID-19 pandemic.^21,22  The ability for increased participation in virtual orthopedic rotations and the corresponding high rate of interview invitations to matriculation of participants reflect this outreach. Concerns surrounding potential lack of clinical exposure and consequently interview invitations for applicants after VSIs were also prevalent, as a study from Tawfik et al revealed.²⁸  Despite this, the transition to virtual learning curricula for away rotations may decrease the financial burden and allow students the continued opportunity to match at programs previously inaccessable.²⁹ 

The inclusion of diverse learning formats, such as asynchronous and synchronous content and a blend of case discussions and mentorship, caters to varied learning preferences and schedules. VSIs present an opportunity for medical students to engage in more away rotations than traditionally feasible, addressing the challenge of overlapping rotation schedules and graduation requirements at different institutions. This approach has the potential to attract a more diverse pool of medical students, as indicated in the works of Asaad et al and Radwanski et al.^26,30 

All included studies underscore the role of VSIs in communicating program culture and differences between residency experiences. Medical students in the 2020-2021 Match cycle struggled to evaluate program features, while virtual platforms for outreach and networking, introduced during the COVID-19 pandemic, significantly shaped the application process.^31,32  The predominance of 2-week rotations (46%, 6 of 13) suggests that this duration strikes a balance between sufficient exposure and accommodating more students. The majority of rotations (62%, 8 of 13) occurred in the summer, likely aligning with students’ schedules and allowing participation before residency applications. Despite variations in duration and timing, student satisfaction remained consistently high, indicating that VSIs can be effective across different formats.

Moreover, these virtual experiences aid faculty in assessing students’ fit for their residency programs, a critical aspect for residency placement where culture and fit are as crucial as technical knowledge. Haws et al highlighted the need for specific resident-only social events for more quality interaction and fit assessment.¹³  Studies like those by Hoffman et al and Raghunathan et al highlight the value of these interactions in reinforcing the importance of evaluating students’ compatibility with their future program.^14,19  This level of insight typically emerges during the residency application screening or interview stages, where time constraints limit the depth of evaluation for both parties. VSIs can present an opportunity to gather more comprehensive information aiding in more informed decisions during each phase of the Match process for more individuals than could be accommodated in-house.¹⁴ 

While VSIs offer many benefits, they come with inherent challenges and limitations. One notable drawback is the inability to fully replicate certain hands-on experiences, such as knot-tying in vascular surgery, which remains a crucial skill in surgical training.¹⁸  At-home suture labs attempted to bridge this gap but remain distinct from in-person experiences. Programs that considered live streaming or recording surgical procedures encountered technical and patient privacy issues, limiting their ability to fully replicate in-person subinternships.¹⁸ 

The virtual format poses challenges in building relationships between students and programs, as it often lacks the informal interactions crucial for interpersonal connection, despite facilitating virtual discussion.³³  Additionally, the variability in study designs, sample sizes, and evaluation methods across the included studies affects generalizability of these findings. The absence of standardized assessment tools across the studies limits internal validity and makes direct comparisons difficult. Similarly, limitations of our research process include reliance on published data and the absence of a standardized assessment tool across the reviewed studies.

Future research should evaluate the balance between educational quality and faculty workload. Smaller cohorts, increased faculty-to-student ratios, and a screening process for applicants may optimize resources.^18,21  Additionally, future studies should prioritize the use of standardized assessment tools to accurately measure VSI effectiveness on learner outcomes and program fit. Research on long-term outcomes of VSI participation, including impacts on residency performance and career satisfaction, would help estimate the extended value of virtual rotations. To further refine these virtual programs, feedback indicates a need for more opportunities for students to demonstrate their skills and competencies. This could be achieved through active workshops and graded assignments that challenge students’ motor skills or ability to work as a team, especially in surgical fields.

Several deviations from best practices were noted during the rapid implementation of VSIs. As mentioned, the absence of formal interrater reliability and standardized thematic analyses limited the ability to compare studies directly.

It is important to note that restricting the literature search to articles in English and using only the aforementioned databases may have resulted in the omission of other relevant studies, particularly those published in other languages or indexed in databases not included in our search strategy. Challenges illustrated include difficulty of maintaining methodological rigor under the constraints of an urgent and evolving global crisis.

VSIs have proven effective in delivering curriculum content and engaging learners across surgical specialties, with consistently high levels of student satisfaction. Despite this success, challenges remain in assessing student skills and evaluating residency program fit in a virtual setting.

The online supplementary data contains the curriculum components of virtual subinternship (VIS) studies and a VSI assessment and outcome summary.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.