Background

Prior to the COVID-19 pandemic, accreditation site visit interviews occurred in-person. In response to the pandemic, the Accreditation Council for Graduate Medical Education (ACGME) developed a remote site visit protocol.

Objective

To perform an early assessment of the remote accreditation site visits for programs applying for initial ACGME accreditation.

Methods

A cohort of residency and fellowship programs that had remote site visits was evaluated from June to August 2020. Surveys were sent to program personnel, ACGME accreditation field representatives, and executive directors following the site visits. Comparison of accreditation decisions (Initial Accreditation or Accreditation Withheld) was completed for matched residency or fellowship programs having in-person site visits in 2019.

Results

Surveys were sent to all program personnel from the 58 residency and fellowship programs that had remote site visits for new program applications, as well as the accreditation field representatives who performed the remote visits. The survey response rate was 58% (352 of 607). Ninety-one percent of all respondents were extremely or very confident that remote site visits provided a thorough assessment of proposed residency or fellowship programs. Fifty-four programs having remote site visits were matched by specialty to programs having had in-person program application site visits in 2019. Forty-six programs that had remote site visits received Initial Accreditation, and 52 programs that had in-person site visits in 2019 received Initial Accreditation (P=.093, 95% CI 0.91-22.38).

Conclusions

Most program personnel and accreditation field representatives were confident that remote site visits conducted for program applications provided fair and thorough assessments of the program.

The unprecedented events of the COVID-19 pandemic led many organizations to adjust their operations. To continue the accreditation process for sponsoring institutions and programs, the Accreditation Council for Graduate Medical Education (ACGME) adapted to the pandemic by implementing remote technology for site visits to mitigate the spread of COVID-19.1,2  Reluctance to perform remote interviews in qualitative research has been related to concerns including confidentiality, limited perception of non-verbal communication, and an inability to engage with or develop trust during an interview that may limit disclosure of sensitive information.3-7 

The benefits of remote interviews in studies evaluating resident and fellow interviews for different medical specialties include cost-effectiveness, flexibility of scheduling that may potentially increase the qualified applicant pool, and effective time management.8-15 

The purpose of this study is to obtain early assessment of the effectiveness of remote site visits for programs applications. Effectiveness is defined in this study as the degree to which remote site visits are perceived to be fair and provide a thorough assessment of the program application by program personnel and accreditation field representatives.

A consecutive cohort of residency and fellowship programs applying for ACGME initial accreditation completing remote site visits from June to August 2020 were evaluated by sending surveys to all program personnel and accreditation field representatives following their site visit.

Survey Design and Administration

Survey questions constructed by the authors were sent to selected ACGME accreditation field representatives, review committee executive directors, and review committee members for review. Survey questions were modified before the final surveys were distributed. Survey questions were constructed on a 5-point Likert agreement scale or binary (yes/no) questions, with the opportunity to provide narrative comments (refer to the online supplementary data). Formal content validity testing of the surveys was not performed.

Program personnel (program directors, faculty program coordinators, and designated institutional officials) and accreditation field representatives were invited by email to voluntarily participate in the study following completion of the site visit and before accreditation decisions were made by the review committees. A description of the study and a link to the survey instrument was provided. Identifying data were intentionally omitted on the surveys to ensure anonymity and confidentiality. Completing the survey required the consent of the participant. The survey results were stored in a secure electronic file and retrieved by the research team for subsequent data analysis.

Data Collection, Organization, and Analysis

Accreditation decisions by the respective ACGME review committees were documented for the programs in the study having completed remote site visits and matched by specialty to program applications having in-person site visits in 2019. Accreditation decisions were blinded until matching of the programs was completed.

Categorical data were analyzed by Fisher's exact test (2-tailed). Statistical significance was taken at P<.05. Institutional review board approval was obtained from the American Institutes for Research.

Fifty-two residency and 6 fellowship programs (n=58) had remote program application site visits.

Survey Response Rates

A total of 607 surveys were sent and 352 completed surveys were completed. The overall response rate to surveys was 58%; individual group survey response rates are shown in the Table.

Table

Survey Response Rates

Survey Response Rates
Survey Response Rates

Program Personnel Survey Question Responses

After completing site visits for proposed residency or fellowship programs, 29% (102 of 353) of all respondents preferred an in-person site visit, 33% (118 of 353) of all respondents preferred a remote site visit, and 38% (133 of 353) had no preference.

Eighty-five percent (201 of 236) of program personnel were extremely or very confident that the remote site visit format provided a fair and reasonable assessment of their program, and 91% (277 of 303) were extremely or very confident that the remote site visit format provided a thorough assessment of their program.

Ninety-eight percent (223 of 228) of program personnel were extremely or quite satisfied completing the site visit remotely, 95% (273 of 287) of all respondents were extremely or quite satisfied that the site visit met their expectations, and 94% (217 of 230) were extremely or quite satisfied with the ability to communicate and describe their program or their role in the program.

Summary of Accreditation Field Representative Survey Question Responses

Eighty-eight percent (58 of 66) of accreditation field representatives were extremely or quite satisfied with their ability to connect (be attentive, focus, listen, eliminate distractions) with the program director during the remote site visit. Ninety-five percent (63 of 66) of accreditation field representatives were extremely or quite satisfied in their ability to consciously navigate time during each interview session, and 85% (56 of 66) of accreditation field representatives were extremely or quite satisfied in managing the physical environment (physical space, setting boundaries, competing technologies such as pagers and phones) during a remote site visit. Duplicate responses were noted for 8 of the programs returned by the accreditation field representatives.

Accreditation Decisions Following Remote Site Visits

Fifty of the program applications (86%) obtained Initial Accreditation following remote site visits. Eight program applications (14%) received Accreditation Withheld decisions.

Fifty-four programs that had remote site visits in 2020 could be matched by specialty to program applications completed in-person in 2019 (Figure). Forty-six programs having remote site visits received Initial Accreditation and 8 Accreditation Withheld decisions. Fifty-two programs that had in-person site visits received Initial Accreditation and 2 Accreditation Withheld decisions (P=.093, 95% CI 0.91-22.38).

Figure

Specialty Matched In-Person Site Visits From 2019 and Remote Site Visits From 2020 (P=.09)

Figure

Specialty Matched In-Person Site Visits From 2019 and Remote Site Visits From 2020 (P=.09)

Close modal

This study found that program personnel and accreditation field representatives were confident that remote site visits provided a fair, reasonable, and thorough assessment of their application programs and satisfied they could communicate and describe their program. In the matched cohort study comparing residency or fellowship programs having remote site visits (2020) and in-person site visits (2019), there was no statistical difference between programs obtaining Initial Accreditation or Accreditation Withheld status.

The majority of accreditation field representatives were extremely or quite satisfied with their ability to connect with the program director, navigate the time for each interview session, and manage their physical environment.

One of the benefits noted by program personnel and accreditation field representatives is that remote site visits offer simplified logistics and coordination, particularly for faculty members at distant geographic locations. Several studies evaluating the remote interview process for residency and fellowship selection described its benefits compared to traditional in-person interviews, including cost-effectiveness, time-efficiency, and increased scheduling flexibility.8,9,11-14  This circumstance is particularly relevant for rural-based training programs. Another benefit of remote site visits, noted by accreditation field representatives, is the option to conduct a site visit if bad weather and/or personal safety concerns may otherwise require rescheduling a site visit. An unintended benefit of the remote site visit was requesting and reviewing documents before the scheduled site visit that improves the efficiency of the time allotted for the site visit.

Potential drawbacks with remote site visits include the perceived lack of context not being present at the physical location. Unanticipated changes in the site visit agenda can be difficult to navigate during remote site visits. Videoconference or Zoom fatigue phenomena may occur during remote site visits that require more than 4 to 6 hours to complete.16 

Limitations of this study include the lack of content validity testing of the surveys. Wording of survey questions or the direction of the Likert scale may have influenced how respondents interpreted the questions as intended. There was a large variability in survey response rates among respondents. As an early assessment of remote site visits, it is possible that survey results would shift over time as accreditation field representatives and program personnel gain experience with remote site visits. Studies of remote accreditation site visits involving residents and fellows, as key ACGME stakeholders, are planned to further assess their effectiveness and acceptance within the graduate medical education community.

This study demonstrates that most program personnel and accreditation field representatives were confident that remote site visits conducted for program applications provided fair and thorough assessments of their proposed residency or fellowship program.

1. 
Sohrabi
C,
Alsafi
Z,
O'Neill
N,
et al
World Health Organization declares global emergency: a review of the 2019 novel coronavirus (COVID-19)
.
Int J Surg
.
2020
;
76
:
71
-
76
.
2. 
Nasca
TJ.
ACGME's early adaptation to the COVID-19 pandemic: principles and lessons learned
.
J Grad Med Educ
.
2020
;
12
(3)
:
375
-
378
.
3. 
Deakin
H,
Wakefield
K.
Skype interviewing: reflections of two PhD researchers
.
Qual Res
.
2014
;
14
(5)
:
603
-
616
.
4. 
Irani
E.
The use of videoconferencing for qualitative interviewing: opportunities, challenges, and considerations
.
Clin Nurs Res
.
2019
;
28
(1)
:
3
-
8
.
5. 
Jenner
BM,
Myers
KC.
Intimacy, rapport, and exceptional disclosure: a comparison of in-person and mediated interview contexts
.
Int J Soc Res Methodol
.
2019
;
22
(2)
:
165
-
177
.
6. 
Krouwel
M,
Jolly
K,
Greenfield
S.
Comparing Skype (video calling) and in-person qualitative interview modes in a study of people with irritable bowel syndrome—an exploratory comparative analysis
.
BMC Med Res Methodol
.
2019
;
19
(1)
:
219
.
7. 
Sedgwick
M,
Spiers
J.
The use of videoconferencing as a medium for the qualitative interview
.
Int J Qual Method
.
2009
;
8
(1)
:
1
-
11
.
8. 
Daram
SR,
Wu
R,
Tang
SJ.
Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process
.
Am J Gastroenterol
.
2014
;
109
(2)
:
155
-
159
.
9. 
Edje
L,
Miller
C,
Kiefer
J,
Oram
D.
Using Skype as an alternative for residency selection interviews
.
J Grad Med Educ
.
2013
;
5
(3)
:
503
-
505
.
10. 
Hariton
E,
Bortoletto
P,
Ayogu
N.
Using video-conference interviews in the residency application process
.
Acad Med
.
2017
;
92
(6)
:
728
-
729
.
11. 
Healy
WL,
Bedair
H.
Videoconference interviews for an adult reconstruction fellowship: lessons learned
.
J Bone Joint Surg Am
.
2017
;
99
(21)
:
e114
.
12. 
Melendez
MM,
Dobryansky
M,
Alizadeh
K.
Live online video interviews dramatically improve the plastic surgery residency application process
.
Plast Reconstr Surg
.
2012
;
130
(1)
:
e240
-
e241
.
13. 
Pourmand
A,
Lee
H,
Fair
M,
Maloney
K,
Caggiula,
A.
Feasibility and usability of tele-interview for medical residency interview
.
West J Emerg Med
.
2018
;
19
(1)
:
80
-
86
.
14. 
Vadi
MG,
Malkin
MR,
Lenart
J,
Stier
GR,
Gatling
JW,
Applegate
RL
2nd.
Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study
.
Int J Med Educ
.
2016
;
7
:
102
-
108
.
15. 
Williams
K,
Kling
JM,
Labonte
HR,
Blair
JE.
Videoconference interviewing: tips for success
.
J Grad Med Educ
.
2015
;
7
(3)
:
331
-
333
.
16. 
Bennett
AA,
Campion
ED,
Keeler
KR,
Keener
SK.
Videoconference fatigue? Exploring changes in fatigue after videoconference meetings during COVID-19
.
J App Psychol
.
2021
;
106
(3)
:
330
-
344
.

Author notes

Editor's Note: The online version of this article contains open text comments from program personnel and accreditation field representatives and the surveys used in the study.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

Supplementary data