ABSTRACT
Prior to the COVID-19 pandemic, accreditation site visit interviews occurred in-person. In response to the pandemic, the Accreditation Council for Graduate Medical Education (ACGME) developed a remote site visit protocol.
To perform an early assessment of the remote accreditation site visits for programs applying for initial ACGME accreditation.
A cohort of residency and fellowship programs that had remote site visits was evaluated from June to August 2020. Surveys were sent to program personnel, ACGME accreditation field representatives, and executive directors following the site visits. Comparison of accreditation decisions (Initial Accreditation or Accreditation Withheld) was completed for matched residency or fellowship programs having in-person site visits in 2019.
Surveys were sent to all program personnel from the 58 residency and fellowship programs that had remote site visits for new program applications, as well as the accreditation field representatives who performed the remote visits. The survey response rate was 58% (352 of 607). Ninety-one percent of all respondents were extremely or very confident that remote site visits provided a thorough assessment of proposed residency or fellowship programs. Fifty-four programs having remote site visits were matched by specialty to programs having had in-person program application site visits in 2019. Forty-six programs that had remote site visits received Initial Accreditation, and 52 programs that had in-person site visits in 2019 received Initial Accreditation (P=.093, 95% CI 0.91-22.38).
Most program personnel and accreditation field representatives were confident that remote site visits conducted for program applications provided fair and thorough assessments of the program.
Introduction
The unprecedented events of the COVID-19 pandemic led many organizations to adjust their operations. To continue the accreditation process for sponsoring institutions and programs, the Accreditation Council for Graduate Medical Education (ACGME) adapted to the pandemic by implementing remote technology for site visits to mitigate the spread of COVID-19.1,2 Reluctance to perform remote interviews in qualitative research has been related to concerns including confidentiality, limited perception of non-verbal communication, and an inability to engage with or develop trust during an interview that may limit disclosure of sensitive information.3-7
The benefits of remote interviews in studies evaluating resident and fellow interviews for different medical specialties include cost-effectiveness, flexibility of scheduling that may potentially increase the qualified applicant pool, and effective time management.8-15
The purpose of this study is to obtain early assessment of the effectiveness of remote site visits for programs applications. Effectiveness is defined in this study as the degree to which remote site visits are perceived to be fair and provide a thorough assessment of the program application by program personnel and accreditation field representatives.
Methods
A consecutive cohort of residency and fellowship programs applying for ACGME initial accreditation completing remote site visits from June to August 2020 were evaluated by sending surveys to all program personnel and accreditation field representatives following their site visit.
Survey Design and Administration
Survey questions constructed by the authors were sent to selected ACGME accreditation field representatives, review committee executive directors, and review committee members for review. Survey questions were modified before the final surveys were distributed. Survey questions were constructed on a 5-point Likert agreement scale or binary (yes/no) questions, with the opportunity to provide narrative comments (refer to the online supplementary data). Formal content validity testing of the surveys was not performed.
Program personnel (program directors, faculty program coordinators, and designated institutional officials) and accreditation field representatives were invited by email to voluntarily participate in the study following completion of the site visit and before accreditation decisions were made by the review committees. A description of the study and a link to the survey instrument was provided. Identifying data were intentionally omitted on the surveys to ensure anonymity and confidentiality. Completing the survey required the consent of the participant. The survey results were stored in a secure electronic file and retrieved by the research team for subsequent data analysis.
Data Collection, Organization, and Analysis
Accreditation decisions by the respective ACGME review committees were documented for the programs in the study having completed remote site visits and matched by specialty to program applications having in-person site visits in 2019. Accreditation decisions were blinded until matching of the programs was completed.
Categorical data were analyzed by Fisher's exact test (2-tailed). Statistical significance was taken at P<.05. Institutional review board approval was obtained from the American Institutes for Research.
Results
Fifty-two residency and 6 fellowship programs (n=58) had remote program application site visits.
Survey Response Rates
A total of 607 surveys were sent and 352 completed surveys were completed. The overall response rate to surveys was 58%; individual group survey response rates are shown in the Table.
Program Personnel Survey Question Responses
After completing site visits for proposed residency or fellowship programs, 29% (102 of 353) of all respondents preferred an in-person site visit, 33% (118 of 353) of all respondents preferred a remote site visit, and 38% (133 of 353) had no preference.
Eighty-five percent (201 of 236) of program personnel were extremely or very confident that the remote site visit format provided a fair and reasonable assessment of their program, and 91% (277 of 303) were extremely or very confident that the remote site visit format provided a thorough assessment of their program.
Ninety-eight percent (223 of 228) of program personnel were extremely or quite satisfied completing the site visit remotely, 95% (273 of 287) of all respondents were extremely or quite satisfied that the site visit met their expectations, and 94% (217 of 230) were extremely or quite satisfied with the ability to communicate and describe their program or their role in the program.
Summary of Accreditation Field Representative Survey Question Responses
Eighty-eight percent (58 of 66) of accreditation field representatives were extremely or quite satisfied with their ability to connect (be attentive, focus, listen, eliminate distractions) with the program director during the remote site visit. Ninety-five percent (63 of 66) of accreditation field representatives were extremely or quite satisfied in their ability to consciously navigate time during each interview session, and 85% (56 of 66) of accreditation field representatives were extremely or quite satisfied in managing the physical environment (physical space, setting boundaries, competing technologies such as pagers and phones) during a remote site visit. Duplicate responses were noted for 8 of the programs returned by the accreditation field representatives.
Accreditation Decisions Following Remote Site Visits
Fifty of the program applications (86%) obtained Initial Accreditation following remote site visits. Eight program applications (14%) received Accreditation Withheld decisions.
Fifty-four programs that had remote site visits in 2020 could be matched by specialty to program applications completed in-person in 2019 (Figure). Forty-six programs having remote site visits received Initial Accreditation and 8 Accreditation Withheld decisions. Fifty-two programs that had in-person site visits received Initial Accreditation and 2 Accreditation Withheld decisions (P=.093, 95% CI 0.91-22.38).
Specialty Matched In-Person Site Visits From 2019 and Remote Site Visits From 2020 (P=.09)
Specialty Matched In-Person Site Visits From 2019 and Remote Site Visits From 2020 (P=.09)
Discussion
This study found that program personnel and accreditation field representatives were confident that remote site visits provided a fair, reasonable, and thorough assessment of their application programs and satisfied they could communicate and describe their program. In the matched cohort study comparing residency or fellowship programs having remote site visits (2020) and in-person site visits (2019), there was no statistical difference between programs obtaining Initial Accreditation or Accreditation Withheld status.
The majority of accreditation field representatives were extremely or quite satisfied with their ability to connect with the program director, navigate the time for each interview session, and manage their physical environment.
One of the benefits noted by program personnel and accreditation field representatives is that remote site visits offer simplified logistics and coordination, particularly for faculty members at distant geographic locations. Several studies evaluating the remote interview process for residency and fellowship selection described its benefits compared to traditional in-person interviews, including cost-effectiveness, time-efficiency, and increased scheduling flexibility.8,9,11-14 This circumstance is particularly relevant for rural-based training programs. Another benefit of remote site visits, noted by accreditation field representatives, is the option to conduct a site visit if bad weather and/or personal safety concerns may otherwise require rescheduling a site visit. An unintended benefit of the remote site visit was requesting and reviewing documents before the scheduled site visit that improves the efficiency of the time allotted for the site visit.
Potential drawbacks with remote site visits include the perceived lack of context not being present at the physical location. Unanticipated changes in the site visit agenda can be difficult to navigate during remote site visits. Videoconference or Zoom fatigue phenomena may occur during remote site visits that require more than 4 to 6 hours to complete.16
Limitations of this study include the lack of content validity testing of the surveys. Wording of survey questions or the direction of the Likert scale may have influenced how respondents interpreted the questions as intended. There was a large variability in survey response rates among respondents. As an early assessment of remote site visits, it is possible that survey results would shift over time as accreditation field representatives and program personnel gain experience with remote site visits. Studies of remote accreditation site visits involving residents and fellows, as key ACGME stakeholders, are planned to further assess their effectiveness and acceptance within the graduate medical education community.
Conclusions
This study demonstrates that most program personnel and accreditation field representatives were confident that remote site visits conducted for program applications provided fair and thorough assessments of their proposed residency or fellowship program.
References
Author notes
Editor's Note: The online version of this article contains open text comments from program personnel and accreditation field representatives and the surveys used in the study.
Funding: The authors report no external funding source for this study.
Competing Interests
Conflict of interest: The authors declare they have no competing interests.