Reasons for Data-Prompted Site Visits: Field Staff Findings and Review Committee Decisions

An integral component of the Next Accreditation System (NAS) implemented by the Accreditation Council for Graduate Medical Education (ACGME) in 2013 is the annual review of programs' overall performance by review committees (RCs).¹  In September and October of each year, RCs are provided with data that permit them to render an annual accreditation status and identify programs with underperformance issues. Data reviewed by the RCs include program characteristics, participating teaching sites, changes in faculty and program leadership, resident and faculty attrition, faculty and resident scholarly activity, the annual ACGME Resident/Fellow and Faculty Surveys, resident clinical and case log experience, and program-level performance on the certification examinations of the American Board of Medical Specialties member boards.²  Upon annual review of these data, the RCs issue an updated accreditation status to each program. RCs may request a progress report if they identify potential problems or request a site visit. These data-prompted site visits (DPSVs) may be full, with review of all common and specialty-specific program requirements, or focused, with attention on specific areas noted by the RC. To date, the overall results of the DPSVs have not been reported to the graduate medical education community.

The aims of this study were to: (1) identify the reasons RCs requested DPSVs; (2) describe the findings by accreditation field representatives as reflected in their site visit reports for the DPSVs; and (3) summarize the accreditation decisions of RCs after the DPSVs.

All DPSVs requested by RCs from January 2015 to March 2020 were included in the study. Institutional DPSVs were not included in the study. Materials reviewed were the letters from the RCs to programs informing them of a DPSV, site visit reports, and letters from the RCs to programs with post–site visit accreditation decisions.

Data were recorded in Survey Monkey and reviewed for: (1) program specialty; (2) letter from the RC informing the program of a full or focused site visit; (3) reasons cited by the RC for the DPSV including concerns about the Resident/Fellow and Faculty Surveys, scholarly activity of residents and faculty, board pass rate, clinical experience/case log deficiencies, attrition of residents and faculty, progress on addressing citations, and other program-specific concerns; (4) pre–site visit accreditation status; (5) the site visit report for change in program director, program/institutional changes, notation of a special review by the Graduate Medical Education Committee (GMEC), views of interviewed stakeholders, confirmation of RC concerns, identification of new and potential noncompliance areas; (6) post–site visit letter from the RC with accreditation status, change in accreditation status, and number of extended and new citations; and (7) occurrence of additional DPSVs after the first site visit. The total number of DPSVs for each program was recorded, and the same data were noted for each additional DPSV. Programs were deidentified and grouped by specialty category as hospital-based, medical-based, and surgery-based. Data were aggregated as total numbers, frequencies, and percentages.

Institutional Review Board approval for the study was granted by the American Institutes for Research.

The RCs in all specialties requested 312 DPSVs for 59 hospital-based programs, 122 medical-based programs, and 131 surgery-based programs. Core residency programs in all specialties had at least one DPSV. The 59 hospital-based programs represent 5.1% of all ACGME-accredited hospital-based programs, the 122 medical-based programs represent 5.6% of all ACGME-accredited medical-based programs, and the 131 surgery-based programs represent 7.8% of all ACGME-accredited surgery-based programs. A single DPSV occurred in 214 of 312 programs (69%), as shown in table 1a-c. Repeat DPSVs were performed in 98 programs (31%), as shown in table 2a-c.

The RCs requested a full DPSV in 55% and a focused review in 45% of the 44 hospital-based programs. The medical-based programs had a full DPSV in 76% and a focused review in 24%. Full DPSVs were conducted in 70% of the surgery-based programs and focused reviews in 30%.

The RC's letter to the program listed concerns with the results of the ACGME Resident/Fellow Survey in 180 programs (84%) and as the only reason for the DPSV in 110 programs (51%). Other reasons given by RCs for a DPSV were graduates not meeting case log minimum requirements; first-time board pass rates below accepted thresholds; results of the Faculty Survey; insufficient resident, fellow, and/or faculty scholarly activity; resident, fellow, and/or faculty attrition; absence of subspecialty faculty in critical areas; and failure to take actions to resolve citations. Table 3 shows the percentage of noncompliant domains when the Resident/Fellow Survey was cited by RCs. All programs had noncompliance in more than one domain, with duty hours problems noted less often in the hospital-based programs in contrast to the medical- and surgery-based specialties. Noncompliance within domains common to all specialties included faculty supervision, feedback after assignments, opportunities for scholarly activities, and ability to raise concerns without fear of intimidation or retaliation. Patient safety and teamwork concerns were rarely noted to be of concern.

RCs identified concerns with results of the Faculty Survey in 42 programs distributed among all specialties in the following domains: (1) Supervision and Teaching, in the areas of not being satisfied with performance feedback, residents/fellows not seeking supervisory guidance, and poor interest of the program director; (2) Educational Content, in the areas of not working with residents/fellows on scholarly projects and concerns about graduates' effectiveness; (3) Resources, in the areas of not being satisfied with the process to deal with concerns and not being satisfied with faculty development; (4) Patient Safety, in the areas of information being lost during shift changes and not participating with residents/fellows on quality improvement or patient safety projects; and (5) Teamwork, in the areas of trainees not communicating effectively when transferring care and not being effective in teaching teamwork skills.

Deficiencies in meeting case log minimum requirements occurred in 30% of the surgery-based programs, 5% of medical-based programs, and 4% of hospital-based programs. In the majority of programs this concern was noted by the RC for programs with more than 1 year of graduates not meeting case log requirements.

Failure of graduates to meet thresholds for first-time board pass rates was noted by RCs for 28% of hospital-based, 16% of medical-based, and 13% of surgery-based programs. Inadequate resident/fellow scholarly activity was listed by RCs in 30% of hospital-based, 13% of medical-based, and 28% of surgery-based programs. Faculty scholarly activity was cited by RCs less commonly: 8% hospital-based, 7% medical-based, and 17% surgery-based programs.

A change in the program director was made in 14 of 214 programs (7%) after the RC's letter to the program and prior to the DPSV. Site visit reports indicated that special reviews were conducted by the Sponsoring Institution in 39 programs (18%), 19 site visit reports noted that the programs did not have a special review, and 156 site visit reports contained no mention of a special review. The most common reasons for the special review were declining compliance on the annual Resident/Fellow Survey and notification by the RC of an impending DPSV.

Site visit reports noted that 78% of programs made changes prior to the ACGME review, most frequently to address noncompliant areas on the Resident/Fellow Survey, addition of board review sessions for programs with suboptimal board pass rates, and initiation of plans to correct deficiencies noted by the RC.

Accreditation field representatives confirmed the concerns of RCs in the majority of programs: 77% in hospital-based, 69% in medical-based, and 67% in surgery-based programs. Identification of new areas of potential noncompliance with common and/or specialty requirements was noted in 34% of site visit reports, evenly distributed among the 3 specialty categories.

In the hospital-based category, the RCs maintained continued accreditation after the DPSVs in 31 of 40 programs, issued continued accreditation with warning in 8 programs, and withdrew accreditation in 1 program. For the 3 hospital-based programs on continued accreditation with warning, 2 were given continued accreditation and 1 remained on continued accreditation with warning. The single hospital-based program on probationary accreditation prior to the DPSV was awarded continued accreditation. For the 78 medical-based programs with prior continued accreditation, 70 maintained this status, 7 received continued accreditation with warning, and 1 was issued probationary accreditation. Three of 5 medical-based programs on prior continued accreditation with warning were given continued accreditation and 2 had no change in their accreditation status. In the 65 surgery-based programs with prior continued accreditation, 33 maintained this status, 28 were given continued accreditation with warning, 3 probationary accreditation, and 1 had accreditation withdrawn. For the 5 programs on continued accreditation without outcomes, 2 were converted to continued accreditation, 2 had no change in status, and 1 was given continued accreditation with warning. Eight of 18 surgery-based programs with prior continued accreditation with warning received continued accreditation, 8 had no change in status, and 2 were placed on probationary accreditation. Of the programs that had prior continued accreditation, 64% of the surgery-based programs received continued accreditation with warning, in contrast to 20% of the hospital-based and 9% of the medical-based programs.

RCs issued extended citations for 23% of hospital-based, 22% of medical-based, and 41% of surgery-based programs. New citations were issued in 61% of hospital-based, 71% of medical-based, and 64% of surgery-based programs. The accreditation status as of March 2020 for the 214 programs was continued accreditation in 166 programs, continued accreditation without outcomes in 4, continued accreditation with warning in 37, probationary accreditation in 3, voluntary withdrawal in 3, and accreditation withdrawn by the RC in 1 program.

All programs in this group had at least 2 DPSVs. A third DPSV occurred in 22 programs: 3 hospital-based programs (20%), 7 medical-based programs (18%), and 12 surgery-based programs (27%). Twenty-three of the 98 programs had favorable accreditation status after their first DPSV, and the RCs requested second and/or third DPSVs after concerning issues were noted in subsequent and/or nonsequential annual reviews.

For the initial DPSV, full reviews were requested by RCs in 87% of hospital-based, 74% of medical-based, and 87% of surgery-based programs, with the remaining programs having reviews focused on specific issues. All second and third DPSVs were full reviews.

The RC's letter to the program listed concerns with the results of the Resident/Fellow Survey in 96 programs (98%) and as the only reason for the DPSV in 44 programs (45%). The Resident/Fellow Survey was listed along with the same additional concerns noted in the single DPSV group: graduates not meeting case log minimum requirements; first-time board pass rates below accepted thresholds; results of the Faculty Survey; insufficient resident, fellow, and/or faculty scholarly activity; and resident, fellow, and/or faculty attrition. Table 4 shows the frequency of noncompliant domains on the Resident/Fellow Survey cited by RCs as a reason for the DPSV. In comparison to programs with a single DPSV, issues with duty hours, teamwork/patient safety, and a negative/very negative overall evaluation of the program were more common for programs in the repeat DPSV group.

Fifteen programs distributed among the 3 specialty categories were noted by the RCs to have noncompliance in the Faculty Survey. The domains of concern were the same as described for the single DPSV group, except for a higher frequency of faculty dissatisfaction with the process to deal with problems and effectiveness of graduates.

Following notification by the RC of their first DPSV, 28 of the 98 programs (29%) made a change in program director. The site visit report noted that a special review was conducted by the GMEC in 3 programs before the first DPSV, while lack of a special review was stated in 40 site visit reports and not mentioned in 55 reports. Changes to address noncompliant domains on the Resident/Fellow Survey and other RC concerns were initiated by 75% of the programs prior to the review.

During the first DPSVs, accreditation field representatives confirmed RC concerns in 100% of hospital-based, 92% of medical-based, and 91% of surgery-based programs. New issues of potential noncompliance with common and/or specialty requirements were described in 55 programs (56%). In 91 programs (92%), accreditation field representatives identified ongoing challenges with correcting chronic problems.

Of the 79 programs with continued accreditation prior to the first DPSV, 74% received either continued accreditation with warning or probationary accreditation, and 26% maintained their previous status. The 2 programs on continued accreditation without outcomes and the single program on initial accreditation had no change in status. Five of the 13 programs on prior continued accreditation with warning maintained that status, and 8 programs were placed on probationary accreditation. Of the 3 programs on probationary accreditation, 1 received continued accreditation, 1 continued accreditation with warning, and 1 had no change in status. RCs issued extended citations in 51 programs (54%) and new citations in 69 programs (70%).

Prior to their second DPSV, 22 programs (22%) appointed a new program director. Accreditation field representatives indicated that a special review was performed in 20 programs (20%), with lack of a special review noted in 39 site visit reports and not mentioned in 39 reports. The special reviews took place in 16 programs with either continued accreditation with warning or probationary accreditation, and in 4 programs with continued noncompliance on the Resident/Fellow Survey. Accreditation field representatives confirmed the concerns of the RCs in 100% of programs at the second DPSV.

Nineteen of the 29 programs on continued accreditation prior to the second DPSV maintained their status, 7 received continued accreditation with warning, 1 program was placed on probationary accreditation, and 1 program had accreditation withdrawn. Of the 30 programs on continued accreditation with warning, 17 achieved continued accreditation. For the 37 programs with probationary accreditation, 16 were granted continued accreditation, 18 had a status of continued accreditation with warning, 1 remained on probationary accreditation, and 2 had their accreditation withdrawn. RCs issued extended citations in 64 programs (65%) and new citations in 70 programs (71%).

In the 22 programs that had a third DPSV, 6 (27%) made a change in program director and 8 (36%) had a special review after receiving a status of continued accreditation with warning or probationary accreditation following their second review. All these programs achieved a status of continued accreditation after the third DPSV. Two programs received probationary accreditation and 1 program had accreditation withdrawn by the RC. The site visit reports for these programs noted major improvements in areas of concern and high morale among stakeholders. Thirteen programs (59%) received extended citations and 11 programs (50%) were given new citations. As of March 2020, 70% of the 98 programs had a status of continued accreditation, 19% in continued accreditation with warning, and a small percentage of programs on probationary accreditation (2 programs) or had accreditation withdrawn by the RC (5 programs).

Our study provides the first comprehensive review of the annual assessment of program performance by RCs at each critical stage of the accreditation process: (1) identification of underperformance by RCs; (2) accreditation field representative findings reflected in their site visit reports; and (3) RCs post–site visit accreditation decisions. A significant finding is that RCs requested DPSVs for a small number, 312 of over 12 000 ACGME-accredited programs following their annual evaluation of overall performance during the 6 years of the study. The majority of the 312 programs selected for site visits, 184 in the single DPSV group and 79 in the repeat DPSV group, had an accreditation status of continued accreditation at the time they were identified by RCs for underperformance.

Noncompliance on the annual Resident/Fellow Survey was the most frequent reason identified by RCs for requesting a DPSV—the only reason in 51% of programs in the single DPSV group and in 45% of programs in the repeat DPSV group. Of particular concern to RCs was noncompliance with duty hours, noted in one-third of the medical and almost half of the surgery-based programs in the single DPSV group. In the repeat DPSV group, noncompliance with duty hours was reported in one-third of hospital-based and half of medical-based and surgery-based programs. Noncompliance in the faculty, educational content, evaluation, and resources domains was similar in the 3 specialty categories for the single and repeat DPSV groups. Noncompliance in the teamwork/patient safety domain and more negative overall program evaluation were more frequent in all specialty categories in the repeat DPSV group.

While RCs cited the annual Faculty Survey in only 42 programs in the single DPSV group and 15 programs in the repeat DPSV group, the concerns of faculty were noted to include inadequate faculty development, insufficient opportunities to work with residents/fellows on their scholarly, quality improvement, and patient safety projects, and loss of information during transitions of patient care. For programs with dissatisfaction on the Faculty Survey, the corresponding Resident/Fellow survey also had significant noncompliance in multiple domains.

We found that RCs were more likely to request repeat DPSVs for programs with the following characteristics: (1) annual Resident/Fellow Surveys in successive years with multiple noncompliant domains and negative overall evaluations; (2) board pass rates below thresholds for successive years; (3) graduates in surgery-based programs did not meet minimum case requirements in successive years; (4) programs failed to resolve chronic problems and/or recurrent issues; (5) absence of subspecialty faculty in critical areas; and (6) chronic attrition of faculty and/or program leadership.

While notification of a DPSV resulted in a change in program director in only 14 of the 214 programs in the single DPSV group, more than half of the programs in the repeat DPSV group made this significant transition in leadership. Reasons for the change in program leadership, as described in the site visit reports, included a collaborative decision between the department chair and the designated institutional official, a desire for a leadership change by residents, a recommendation following a special review, and personal choice by the program director.

An interesting finding of our study is that site visit reports indicated the occurrence of a special review in less than one-quarter of all programs and explicitly stated that no special review was conducted in one-third of programs. We anticipated that more site visit reports would describe the occurrence of special reviews in most of these programs that had been identified as underperforming by their respective RCs. For the 39 programs in the single DPSV group that had a special review, 29 maintained their pre–site visit accreditation status of continued accreditation, one remained on continued accreditation without outcomes, and one gained continued accreditation after being on warning. The remaining programs received continued accreditation with warning (7 programs) and probationary accreditation (1 program). For the programs in the repeat DPSV group that had a special review prior to the second or third site visit, the majority achieved a more favorable accreditation status, either continued accreditation with warning to continued accreditation, or transitioning from probationary accreditation to continued accreditation (5 programs) or continued accreditation with warning (6 programs). While we are unable to conclusively show that a special review is of benefit to underperforming programs for avoiding unfavorable accreditation decisions, the majority of programs that had special reviews either maintained a favorable accreditation status or gained an improved status.

Our findings also highlight that surgery-based programs in both the single and repeat DPSV groups were more likely to receive an unfavorable accreditation status. Of 87 surgery-based programs in the single DPSV group, 37 (43%) were given continued accreditation with warning, 5 (6%) were placed on probationary accreditation, and 1 had accreditation withdrawn. In the repeat DPSV group, the surgery-based programs had a higher percentage of continued accreditation with warning and probationary accreditation than the other specialty categories after their first and second site visits.

The majority of programs achieved continued accreditation in both the single DPSV and repeat DPSV groups. This finding provides support for an important goal of NAS—timely identification of underperformance by the RC and programs responding with successful efforts for program improvement.

A limitation of this study is the lack of precise information about the process used by respective RCs to determine which programs should have DPSVs instead of simple reviews of submitted progress reports or observation of performance for another annual review. A second limitation is that the occurrence of a special review was not mentioned in almost three-quarters of site visit reports. It is possible that more special reviews were performed in programs in both the single and repeat DPSV groups.

From January 2015 to March 2020 a total of 312 DPSVs were performed at the request of RCs in all core programs and multiple subspecialties. A single DPSV occurred in 214 programs and repeat DPSVs in 98 programs. Noncompliance on the annual Resident/Fellow Survey was the most frequent reason RCs requested a DPSV. Notification of a DPSV prompted a change in program director in 7% of programs with a single DPSV and in 57% of programs in the repeat DPSV. The majority of the reviewed programs in both the single DPSV group and in the repeat DPSV group had a status of continued accreditation as of March 2020.

The authors would like to thank Andrea Chow, MA, and Margarita Perez for their assistance in completion of this project.