The Accreditation Council for Graduate Medical Education (ACGME) uses accreditation and recognition site visits to assess Sponsoring Institution and program compliance with institutional, common, and specialty-specific requirements. All site visits are performed by Accreditation Field Representatives employed by the ACGME. During a site visit, Field Representatives review data and documents and collect information through interviews with key individuals in the Sponsoring Institution or program. After completing a site visit, they create site visit reports for the Review and Recognition Committees that provide essential information about the Sponsoring Institution or program not otherwise obtainable through the ACGME's Accreditation Data System (ADS) data.
Prior to March 2020, the role of a Field Representative was synonymous with a “Road Warrior”—someone who's work life consisted of movement and travel. Field Representatives met new people and experienced new places almost daily. Sponsoring Institution or program personnel greeted them at entrances to health care facilities and escorted them to reserved conference rooms for site visit activities. All site visits were performed in-person; motion and travel—boarding passes, suitcases, taxis, and hotel rooms—defined the Field Representative job.
Site visits had been synonymous with “in-person site visits.” But in March 2020, the ACGME suspended all site visits because of the COVID-19 pandemic, leaving Field Representatives wondering if they would be able to perform their jobs. For an ephemeral moment, their work appeared to be in jeopardy because not only had the ACGME postponed in-person site visits, but it also postponed the site visit process altogether. The uncertainty was brief; within several weeks, the Field Activities team began to develop a new method for performing site visits and began to design a process to re-tool and re-train the Field Representatives. Within 3 months of conception, Field Representatives shifted from being Road Warriors to “Zoom Masters.” In this article, we share the experience of the Field Activities team in developing the ACGME remote site visit.
Suspension of Site Visits
Postponing all site visits created space for Field Activities to review established practices and consider new ways to perform site visits. The Field Activities team had to develop remote site visit protocols to enable the ACGME to provide data needed for accreditation and recognition decisions, even amid the pandemic. As part of this objective, the remote site visit protocol needed to address feasibility (eg, training, technology) and acceptability to stakeholders. The Field Activities team formed the Remote Accreditation and Recognition Site Visit (RARSV) and the Leveraging Technology (LT) groups.
The LT group spearheaded all things technology; it tested the audiovisual software, created instructional videos and guidance, and trained all Field Representatives to use Zoom with the specific aim of performing remote site visit interviews. The LT group included Field Representatives, members of the Field Activities staff, and a member of the ACGME's Information Support Services team. This group provided over 150 hours of direct training. Field Representatives, whose experience with the ACGME spanned 2 months to more than 25 years, embraced the learning challenge and used their experiences working unconventionally as Road Warriors to inform the development of the remote site visit protocol. Field Representatives were no strangers to remote work, accessing information on their laptops from home or while traveling, communicating via phone, email, or text, and managing to work asynchronously in nontraditional work settings. Since all Field Representatives, regardless of their tenure in their positions, had to learn to work differently, there seemed to be an openness and shared trust that allowed for enhanced communication, creativity, and collaboration. The LT group trained Field Representatives to use more advanced features in Zoom and created weekly “Zoom Zingers” to allow them to become more facile and comfortable with the software. Additional training with OneDrive, a cloud-based document storage program developed by Microsoft, was also provided by the LT group.
The RARSV group designed the remote site visit protocol and program, including all steps in the process from scheduling site visits, communicating with Sponsoring Institutions and programs, conducting remote site visits, and preparing site visit reports. The RARSV group assisted in modifying all communication and published information to include the remote aspect of the site visit. For example, for in-person site visits there wasn't a need to collect an email address from each interview participant. For remote site visits, an email address for each interviewee is required to be able to send a Zoom invitation for the remote interview. The RARSV group also helped create Remote Site Visit Frequently Asked Questions on the ACGME website.1
The RARSV and LT groups met several times per week. The Field Activities team met weekly to discuss the direction the work was heading. Interdepartmental and organization-wide meetings augmented these Field Activities communications. The frequency of meetings expedited the development of the remote site visit protocol and ensured that the work was performed in a thoughtful, even if near-urgent, manner.
The First Remote Site Visit
As early as April 2020, Field Representatives were presented with several opportunities to implement the remote site visit plan on a small scale. Two previously scheduled remote ACGME-International (ACGME-I) site visits were performed. These site visits took place using Zoom with support from the ACGME-I staff, the ACGME audiovisual team, and the programs. Soon thereafter, Field Representatives performed remote COVID-19 investigations for domestic Sponsoring Institutions and programs.
Using a repeat iterative improvement process, the RARSV group gained several insights about the preliminary remote site visit protocol based on feedback from the Field Representatives who performed these first remote site visits. While the Field Representative for the ACGME-I site visits did not experience the jetlag associated with international travel, the Field Representative did perform the site visit between 10:00pm and 2:30am to accommodate the programs' time zones. A member of the ACGME's audiovisual team joined the Field Representative for the site visit in the middle of the night to provide essential backup support and troubleshoot any potential technical issues. With social distancing measures in place and individuals wearing masks, having more than one interview participant in the same conference room caused issues with audio and difficulty identifying who was speaking at any given time.
The RARSV group used this initial feedback from the ACGME-I remote site visits to rapidly adjust the remote site visit protocol to instruct interview participants to use their own personal devices to help give each person an identity, promote social distancing, and assist with any issues related to confusing audio reception. The Field Representative also suggested headphones might help to mitigate audio issues.
In April 2020, the ACGME announced a new policy in the wake of the COVID-19 pandemic related to potential noncompliance with ACGME COVID-19 Requirements. The policy provided a process for remote COVID-19 investigations at the associated Sponsoring Institution and program to provide information to the ACGME Review and Recognition Committees. Members of the RARSV group were some of the first Field Representatives to perform these investigations.
The COVID-19 investigations allowed Field Activities to further refine the remote site visit protocol, including the development of the Remote Site Visit Attestation Statement, signed by the program director and designated institutional official (DIO) for program site visits, and the DIO for Sponsoring Institution site visits. The purpose of the attestation is to ensure the privacy and confidentiality of the remote interview sessions by confirming that participants will not record any portions of the remote site visit communications. The Attestation Statement also states that the Sponsoring Institution and program must not retaliate against any resident or fellow, faculty, staff, or other person for participation in the site visit or for any statements made to the ACGME relating to this site visit. Field Activities conducted 15 investigations and used feedback from Field Representatives to further refine the remote site visit protocol. Field Representatives reported that they were able to collect the requested information through the remote site visit and began assessing their experiences with remote interviews.
The next step was to expand the remote site visit protocol to the full cadre of Field Representatives. In early May 2020, Field Activities scheduled a small number of application site visits for June and July 2020. The Field Activities scheduling team selected application site visits specifically because these site visits involve fewer interviewees (usually do not include resident/fellow interviews) and applications were deemed by the Review and Recognition Committees to be high priority site visits. All Field Representatives performed one application site visit in either June or July. The scheduling design included assignments during 2 specific weeks with the aim to schedule a single site visit each day. This assured that the ACGME's information and technology support team could manage assisting each site visit as necessary. Field Representatives provided further feedback and observations as part of an iterative process.
In August 2020, the Field Activities scheduling team continued to focus on scheduling high priority remote site visits, which included primarily applications and complaint site visits. The August remote site visits were assigned to teams of Field Representatives to continue training and protocol refinement. Starting in September, the scheduling team scheduled Field Representatives as solo site visitors, which had been the normal assignment for in-person site visits, to perform remote site visits and began to expand to all site visit types.
Studying the Remote Site Visit
Starting in March 2020, 2 groups of Field Representatives proposed research projects related to the remote site visit. Both research projects were awarded the ACGME Nathan K. Blank Fellowship, one each for 2020 and 2021. The projects assess several key aspects of remote site visits and were intended to provide rigorous review of and transparency for the remote site visit program.
The first study is designed to evaluate the efficacy, feasibility, and acceptability of remote site visits with stakeholders. The investigators collected data in 2 phases, the first for remote application site visits (June and July 2020), and the second for complex remote site visits (December 2020 and January 2021). In both phases, survey data were collected from key stakeholders, including program directors, faculty members, DIOs, chairs, residents and fellows, and Review and Recognition Committee members. Preliminary qualitative and quantitative stakeholder feedback data suggest remote site visits are feasible, effective, and reliable.
All site visits prior to the pandemic had been in-person and this modality was considered to be the gold standard, although a formal validation process had not previously been established.
To address this absence of baseline information, the second research project is a comparison of in-person and remote site visits. The study is using programs on initial accreditation that had their application site visit performed in-person, and their subsequent initial accreditation site visit performed remotely. The study is specifically comparing the 2 site visit modalities for programs with program directors who participated in both the in-person application site visit and the remote initial accreditation site visit. After results are examined and synthesized, both studies are expected to produce information that will be shared with the graduate medical education (GME) community.
Beginning in May 2020, the remote site visit program engaged in weekly project management support as the ACGME developed its Project Management Office. Senior leadership received monthly reports from the remote accreditation and recognition site visit project team and provided oversight, expertise, and feedback. Overall, the remote site visit project included more than 1200 hours of ACGME employee time to plan, do, study, and act in a progressive process to adapt to the COVID-19 pandemic and to assure that the ACGME could continue to support its mission. Project management involved establishing, monitoring, and reporting on metrics, including constituent feedback, site visit report quality, Field Activities productivity, timeliness, and budget.
Future research will include ongoing assessment, important considerations of constituent satisfaction, and cost of in-person and remote site visits. Field Activities plans to monitor feedback from Field Representatives, DIOs, program directors, and Review and Recognition Committee members to help measure the reliability and comparability of site visits and site visit reports for site visits completed using both in-person and remote modalities. Field Activities will continue to adapt its scheduling rubric and consideration of indications for in-person and remote site visits in an iterative process based on feedback from key stakeholders. The ACGME will aim to continue to provide high-quality, reliable, and valid site visits and site visit reports while serving as responsible stewards of the ACGME's and the GME community's resources.
Looking Back While Looking Forward
During 2020, Field Activities scheduled and completed 800 site visits, including 361 in-person and 439 remote site visits. This is compared to 1529 site visits in 2019 and projected approximately 1600 site visits in 2021 (see the Table and Figure). The last in-person site visit took place on March 6, 2020. Since then, and until the present, all site visits have been performed remotely using audiovisual technology (primarily Zoom). Remote site visits have been adopted fully for all types of site visits for Sponsoring Institutions and programs.
Like many organizations in 2021, Field Activities has begun to plan for a future that returns to a “new normal,” but will not simply return to performing only in-person site visits as it had been doing prior to the COVID-19 pandemic. Field Activities plans to incorporate several best practices learned from developing and implementing the remote site visit program into in-person site visits. Criteria are being constructed to determine the preferred modality for a given assigned site visit when both in-person and remote site visits are available. Field Representative, Sponsoring Institution, and program constituent safety will remain paramount when in-person site visits resume.
Several aspects of the remote site visit experience will be retained and woven into a new in-person site visit protocol. During the remote site visit protocol development, Field Activities identified secure methods for sharing large and sensitive documents prior to the remote site visit. Reviewing these documents prior to the site visit allowed Field Representatives to conduct a more thorough review, improved their overall familiarity with a Sponsoring Institution or program prior to the site visit, and ensured optimal use of interview time. Going forward, most documents will continue to be provided to and reviewed by Field Representatives prior to any type of site visit, in-person or remote.
Similarly, during the remote site visit development, Field Representatives added a Zoom test call with institutional or program leadership to ensure the Zoom technology was working properly in anticipation of a scheduled remote site visit. While this Zoom test call began as a necessity to test the technology (eg, internet connection), it has become an important opportunity for the Field Representatives and DIO, program director, and/or institutional or program coordinators to connect and begin to establish a working relationship prior to the remote site visit. It has also become a valued opportunity for institutional and program personnel to ask questions about the upcoming site visit. Going forward, Field Representatives will continue with a pre-site visit preparation/Zoom test call prior to either mode of site visit.
Finally, completion of the Sponsoring Institution and program Attestation Statement will be continued for all site visits, regardless of site visit modality. While the expectations delineated as protections and standards in the remote site visit Attestation Statement had always been expected during in-person site visits, the need to articulate these expectations more clearly and in writing became apparent during the development of the remote site visit. The Attestation Statement has been revised to assure applicability for all site visits.
Conclusions
Through development of the remote site visit, the ACGME Field Activities team was given the opportunity to create a new way of conducting work and attempted to do so through an evidence-based, transparent, and collaborative process. The remote site visit was designed to enable the ACGME to provide site visits and site visit reports for accreditation and recognition decisions, even amid the pandemic. The transformation from in-person to remote site visits during the previous 16 months reflects the ACGME's efforts to adapt with flexibility and creativity while also aiming to perform all core functions, including site visits, in an approximately equivalent manner to its pre-pandemic work. The Field Activities team developed and implemented a remote site visit protocol in a continuously sequential process, with data gathering, monitoring, assessment, and revisions performed at least monthly. The results from qualitative and quantitative measures and metrics, drawn from stakeholder feedback, have shown remote site visits to be effective and reliable with approximate equivalency to in-person site visits. Ongoing research will guide the ACGME's decisions for future use of remote and in-person site visits.
While many Field Representatives continue to embrace their Road Warrior selves, most have also adopted the Zoom Master identity to continue to support the ACGME's mission. The Field Activities team remains focused on its purpose to serve as the eyes and the ears, be it remotely or in-person, of the Review and Recognition Committees.
The authors would like to thank the Accreditation Field Representatives and Field Activities Staff for their tremendous contribution to the development of the remote site visit protocol, as well as Lynne Kirk, MD, for reviewing and editing the article.
