Understanding ACGME Standards for Simulation: A Document Analysis of Institutional and Program Requirements

Numerous studies have documented decreased errors and improved clinical outcomes following simulation-based training.^1-6  Several studies have also documented a substantial return on investment to health care systems as a result of simulation training by decreasing costs associated with complications.^7-9  Health care educators recognize the value of simulation, and simulation centers have become widespread.^10-13  However, the use of simulation in graduate medical education (GME) varies among specialties, and information is lacking about how institutions support simulation for their GME programs.^14-27 

The utility of GME simulation has been reported for a single program type or a specific competency domain; several studies have referenced individual Accreditation Council for Graduate Medical Education (ACGME) simulation standards.^21-27  However, no publications have examined ACGME simulation standards at an institutional level. In 2018, University of Texas Southwestern Medical Center (UTSW) opened its new campus-wide Simulation Center. Realizing the importance of simulation within GME, our institutional GME Committee established a Simulation Subcommittee charged with providing oversight of GME simulation activities and increasing awareness in our GME community regarding the utility of simulation-based training and assessment. Since that time, our center has seen substantial growth in the number of GME simulation activities (Figure 1). However, within the 21 UTSW clinical departments, most programs design, develop, and implement simulation curricula with minimal collaboration with other programs. While all departments are required by our institutional financial model to provide monetary support to cover GME simulation activities, their participation varies substantially. Currently, 2 departments do not use the center for any GME simulation activities (Figure 2). Additionally, the current financial model does not cover all the costs of operating the center. Therefore, partnering with clinical learning sites is being explored to further support the center’s operations.

The UTSW Simulation Center and GME leadership have identified 3 priorities based on this current state: (1) increasing adoption by all departments; (2) garnering additional health care system financial support; and (3) sharing simulation curricula across programs at an institutional level. Our academic leadership recently charged the Simulation Subcommittee with developing a core GME simulation program to design and implement simulation curricula that address competencies relevant to all specialties. In accordance with Kern’s 6-Step Approach to Curriculum Development, we are conducting a general needs assessment to identify the current and ideal approaches.^28,29  Some studies describe simulation curricula that have been implemented across 4 to 8 residency programs,^26,27  but none have involved all GME programs at an institution. Given the lack of published information, we decided to examine the ACGME requirements for simulation as a source of extant data to use as part of this needs assessment. A better understanding of these requirements will also inform GME simulation efforts at other institutions.

The purpose of this study was to perform a structured review of ACGME simulation standards to inform our efforts to increase department adoption, garner additional financial support, and create a core GME simulation curriculum. A document analysis process was used to answer the following research questions: (1) Do the ACGME requirements support simulation usage by all departments? (2) Do the ACGME requirements indicate a need for institutional financial support for simulation? (3) Do the ACGME requirements support shared simulation curricula?

This study was conducted from May 2023 to June 2024 by our investigator group, which has extensive expertise in both GME administration and simulation. During the study the investigator team engaged in ongoing reflection activities, including journaling, questioning, and debriefing, to minimize bias and optimize transparency.^30,31  A document analysis was chosen based on its advantages of being straightforward, efficient, cost-effective, and manageable. Document analysis is a systematic procedure for evaluating documents and requires that extant data be examined and interpreted in order to elicit meaning, gain understanding, and develop empirical knowledge.^32-34  For this study, Morgan’s Conceptual Framework for Qualitative Document Analysis was used to provide a structured approach to finding, selecting, appraising, and synthesizing data. This framework outlined the process for reflexive thematic coding of the data extracted from the selected documents.^30-32 

To maximize rigor, investigator group consensus was obtained iteratively in accordance with the thematic analysis method outlined in Morgan’s Conceptual Framework for Qualitative Document Analysis.³²  The versatility of Morgan’s approach to thematic analysis allowed researchers to select the project design that aligned with their interests and areas of expertise.³²  A shared drive was used to house all materials and facilitate investigator group collaboration. The ACGME glossary of terms was used to standardize terminology.³⁵ 

Following Morgan’s framework, the primary reviewers (principal investigator [PI], A.E.S., and senior author [SA], D.J.S.) selected the documents planned for analysis after examining 4 factors, including authenticity, credibility, representativeness, and meaning.³²  Since the ACGME is well established in the United States as the primary accrediting body for residency and fellowship training, all these factors were easily met. The documents were publicly accessible on the ACGME website.^36-38  The Institutional and Common Program Requirements (Residency and Fellowship) were selected as they were deemed to be broadly applicable to our institution and programs. The selection process was further tailored by identifying the Specialty-Specific Program Requirements for the primary specialty sponsored by each of the 21 departments that financially support our center (Figure 2 and Table). For cardiovascular and thoracic surgery (CVTS), the Independent Specialty-Specific Program Requirements, which apply to fellowships, were selected since this is the training format at our institution. A PDF file for each of the selected requirements was downloaded to a shared folder by the PI and verified for correctness by the SA.

The primary reviewers familiarized themselves with all documents, including their context, organization, and content. The entirety of each document was then systematically searched for any content that might be related to our research questions. First, terms (Box 1) were identified that were associated with any sections potentially relevant to simulation by reading sample documents and conducting a preliminary search until no new relevant terms were found. From these terms, a subset of words and word stems (Box 1) were generated and used to perform a comprehensive search of each PDF file. The text from the entire paragraph containing each occurrence was copied to a spreadsheet. Information was organized according to the source document from which it originated. Multiple examinations of each document were performed by 2 independent reviewers (PI, SA) to ensure adequate searching.³⁴  A full search was conducted a second time after substantially broadening the search terms in response to reviewer feedback from our initial manuscript submission; the subsequent steps of analysis were also repeated. The final set of terms is listed in Box 1.

A thorough examination was performed for each captured text paragraph. Through iterative review, consensus was achieved among all investigators to identify content that was relevant to this study. Any captured paragraph that was not relevant to simulation (eg, verification of skills in a clinical setting) was deleted; only content deemed relevant to simulation was retained for coding. Using reflexive thematic analysis, the coding process evolved in an iterative fashion.³²  Each paragraph was discussed by the PI and SA to determine the meaning of the captured text. Each paragraph was then assigned a category on the coding spreadsheet in a cell adjacent to the text. Since the categories were not available prior to the analysis, category codes were split, consolidated, or modified as deemed appropriate by consensus among 3 investigators until a final set of categories was created and assigned. The final category codes were reviewed by all investigators to mitigate investigator biases.^30,31 

Each requirement was further categorized regarding the extent to which it was a mandate. Using ACGME definitions, codes were assigned as “must” if the text indicated a mandatory requirement; otherwise, codes were assigned as “should” (Table).³⁵  Additionally, each requirement was coded as simulation-specific (simulation was a specific part of the requirement) versus simulation-optional (simulation was listed as 1 of at least 2 options, such as “real and/or simulated”). The coding results were used to interpret evidence relevant to our 3 research questions. The entire investigator team approved all assigned coding, having reached consensus following multiple discussions regarding the interpretation of this evidence as presented in this article.

For Question 1, all 12 categories were relevant to simulation usage and were mapped to this question. The categories were analyzed separately for the Institutional and Common Program Requirements (Residency and Fellowship) and for each specialty’s Program Requirements. Evidence was considered strong for any document that included at least one simulation-specific “must” requirement, moderate for any document with at least one simulation-specific “should” requirement, weak for any document with at least one simulation-optional “should” requirement, or nonexistent for any document with no simulation-specific requirement.

For Questions 2 and 3, categories that were identified as relevant by our investigator team were mapped to each question and analyzed across all documents. Categories that addressed criteria requiring financial resources were mapped to Question 2. Categories that addressed curriculum requirements across all specialties were mapped to Question 3. Evidence for Questions 2 and 3 was considered strong if at least 75% of documents included the mapped category, moderate if at least 25% of documents included the mapped category and the majority of these included “must” or were simulation-specific, or weak if fewer than 25% of documents included the mapped category. The results were then analyzed across all mapped categories for each question.

Institutional review board approval for this study was not sought given the document analysis methodology of publicly available accreditation documents and in accordance with the UTSW definition of human subjects research.

Twenty-four documents including ACGME Institutional Requirements, Common Program Requirements (Residency and Fellowship), and 21 Specialty-Specific Program Requirements were selected, searched, and analyzed. A total of 120 requirements relevant to simulation were identified and assigned to 12 categories. Seventy of 120 (58%) were coded as “must” versus 50 (42%) as “should,” and 48 of 120 (40%) were coded as simulation-specific versus 72 (60%) as simulation-optional (Table).

No relevant requirements were identified in the Institutional Requirements, 4 in the Common Program Requirements (Residency), 2 in the Common Program Requirements (Fellowship), and 114 (average 5.4 per specialty) in the Specialty-Specific Program Requirements. The CVTS fellowship had 2 simulation-related requirements whereas all residency programs had at least 4 and up to 12 per specialty.

Regarding this question, evidence was nonexistent for the Institutional Requirements. Evidence was moderate for the Common Program Requirements (Residency and Fellowship) since both of these documents included a “should” requirement for one simulation-specific category (Core Faculty). According to the Specialty-Specific Program Requirements, 11 (52%) of 21 specialties demonstrated strong evidence for simulation usage requirements, 10 (48%) of 21 specialties demonstrated moderate evidence, and none demonstrated weak or nonexistent evidence (Table).

For the 11 specialties demonstrating strong evidence, 7 had standards for access to simulation resources, though the specific requirements varied among specialties. For example, internal medicine requires providing “access to training using simulation,” defined broadly as “learning about patient care in settings that do not include actual patients.”³⁹  Similarly, Obstetrics and Gynecology (OB/GYN) Program Requirements state “There must be … access to simulation resources.”⁴⁰  Radiation oncology includes specific requirements regarding computed-tomography (CT) simulations. Ophthalmology, orthopedic surgery, plastic surgery, and surgery all stipulate requirements for the availability of skills laboratories for the purpose of practicing skills outside of the operating room. Six specialties have standards for specialty-specific simulation curricula, which range from minimum numbers of content-specific simulations (anesthesiology, radiation oncology) to practicing relevant nontechnical (surgery) and technical (ophthalmology, otolaryngology, pediatrics, surgery) skills. Two specialties (emergency medicine and radiation oncology) have case log requirements stipulating that a record must document simulated procedures. Individual specialties have additional mandatory requirements. Simulation-specific assessment is required by ophthalmology by using “structured, hands-on, simulated surgical skills activities to assess resident performance”⁴¹  and by internal medicine which requires faculty participation in “designing and implementing simulation and/or standardized patients for teaching and assessment.”³⁹  OB/GYN requires protected time for learners and surgery requires administrative and personnel support for simulation.

For the 10 specialties demonstrating moderate evidence, similar to the Common Program Requirements (Residency and Fellowship), all included a “should” requirement for the Core Faculty category. This requirement states that: “Core faculty members may also be selected for their specific expertise” and “participate in nonclinical activities…[which] include…simulation exercises.”³⁷  Thus, the standard supports, but does not mandate, programs to have faculty with expertise in simulation.

Of the 24 documents analyzed, 11 (46%) provided strong evidence affirming that the ACGME requirements support simulation usage by all departments, 12 (50%) provided moderate evidence, and one (4%) did not provide any evidence.

Categories 1, 4, 5, 7, 8, 9, 11, and 12 addressed criteria that require financial resources and were mapped to this question. These categories involved resources, protected time for faculty and learners, and specifications for simulation curricula (Table). Strong evidence was present for 4 categories: 4 (23 of 24, 96% of documents), 5 (23 of 24, 96% of documents), 7 (21 of 24, 88% of documents), and 9 (21 of 24, 88% of documents). Moderate evidence was present for 2 categories: 1 (7 of 24, 29% of documents) and 12 (6 of 24, 25%, of documents). Weak evidence was present for 2 categories: 8 and 11, which were each present in only 1 of 24 (4.2%) documents.

Of the 8 categories mapped to this question, 4 (50%) provided strong evidence, 2 (25%) provided moderate evidence, and 2 (25%) provided weak evidence affirming that the ACGME requirements indicate a need for institutional financial support for simulation.

Categories 5 and 7 addressed curriculum requirements across all specialties and were mapped to this question. The Interprofessional Patient Safety and QI standard (Category 5) states that: “Residents must participate as team members in real and/or simulated interprofessional clinical patient safety and quality improvement activities, such as root cause analyses or other activities that include analysis, as well as formulation and implementation of actions.”³⁶  This standard was coded as a “must” statement and as simulation-optional since simulation is listed as 1 of 2 options. The Institutional Requirements did not address this category, but the remaining 23 of 24 total documents (96%) did. Thus, this criterion was considered strong evidence. The Patient Safety and Disclosure standard (Category 7) states that: “Residents must demonstrate competence in using tools and techniques that promote patient safety and disclosure of patient safety events (real or simulated).”³⁶  Similarly, this standard was coded as a “must” statement and as simulation-optional since simulation is listed as 1 of 2 options. The Institutional, Common (Fellowship), and CVTS Program Requirements did not address this category but the remaining 21 of 24 total documents (88%) did. Thus, this criterion was also considered strong evidence.

Even though only 2 categories mapped to this question, both provided strong evidence affirming that the ACGME requirements support shared simulation curricula.

In the 24 documents selected, we found a high number (120) of simulation-related standards, the majority (69, 58%) of which were mandatory. This observation, combined with the finding that both the Common Program Requirements (Residency and Fellowship) and all Specialty Program Requirements included simulation-specific criteria, suggests that the ACGME strongly values simulation as an important educational tool for residents and fellows. Surprisingly, there were no standards in the Institutional Requirements that address simulation, or the resources required.

The information gathered in this study will be useful as we develop our institutional core GME simulation program and may also encourage other institutions to identify similar opportunities. Our findings justify involving all departments in GME simulation and developing shared curricula. While only 6 programs had mandatory specialty-specific curriculum requirements, all 21 Program and both Common Program (Residency and Fellowship) Requirements had mandatory criteria for Interprofessional Patient Safety and QI, which focused on patient safety event training. This standard also specified the use of interprofessional education, which is considered critical to developing effective teamwork and collaboration skills and a best practice in simulation-based medical education.^29,42  Twenty Specialty-Specific Program and Common Program (Residency) Requirements had mandatory criteria addressing Patient Safety and Disclosure, which emphasized disclosure of patient safety events. Simulation has been shown to improve patient safety and is well suited for these curricula.^1-10,21-27  However, the resources required to host such important simulation activities may be substantial.^10,11,24,25  Our results indicated a need for institutional financial support, recognizing that individual programs may not be able to provide these resources on their own. For example, the Internal Medicine Program Requirements state that, while programs must “provide residents with access to training using simulation,” it also states that, “the Review Committee does not expect each program to own a simulator or to have a simulation center.”³⁹  Orthopedic Surgery and Plastic Surgery Program Requirements state that “the program, in partnership with its sponsoring institution, must ensure the availability of adequate resources for resident education”^43,44  and that these resources must include “a dedicated space to facilitate basic surgical skills training”⁴³  or “skills laboratories.”⁴⁴ 

Our findings also identify opportunities for the ACGME to revise its simulation standards (Box 2). It would be very helpful for the ACGME simulation requirements to be consistently labeled as “simulation,” such that programs and institutions could more easily identify, understand, and comply with these requirements. Acquisition of skills during live patient care may introduce variability in training, provide fewer opportunities for debriefing and feedback, and pose challenges to both psychological safety and deliberate practice.^{2,21-25,29,45,46}  Given the proven educational value and patient safety benefits of simulation,^1-10,21-27  the ACGME should encourage more widespread adoption by making more requirements mandatory, simulation-specific, and consistent across specialties. In light of the persistent threat of clinical demands compromising education, mandating protected faculty and learner time for simulation would be helpful for all specialties, but it is only mandatory currently for 2 specialties. For unknown reasons, protected simulation time is referenced only for residency programs and not for fellowships. Alignment between Specialty-Specific, Common, and Institutional Requirements would provide a clearer emphasis on the importance of simulation. For example, the Common Program Requirements do not contain any mandatory simulation-specific standards, even though most specialty requirements do. Moreover, the Institutional Requirements do not contain any simulation standards. By creating simulation requirements at the institutional level, the ACGME could help institutions garner the resources required to build, maintain, and operate simulation centers, as well as to support simulation personnel, faculty, learners, and curricula. Integrating simulation into other ACGME processes would further enhance GME education. Incorporating questions about simulation on the annual Resident/Fellow Survey would help the ACGME monitor outcomes and provide valuable information to guide simulation efforts for initiatives like the Harmonized Milestones.^47-49  Dissemination of successfully implemented curricula would also help programs overcome barriers related to curriculum design.^10,21,23 

We realize that the results of our study may not be completely generalizable to other institutions, as we intentionally analyzed the requirements for the 21 primary specialties that represent our GME simulation community. The makeup of specialties at other institutions may vary. Further, we did not include Program Requirements for subspecialties, which would have added 162 additional documents, as it was deemed outside the scope of our study. However, in keeping with Morgan’s framework, after thoroughly analyzing the 24 selected documents, we did not feel that analyzing more data would lead to the development of any new themes.³²  Additionally, it is likely that the specialties we included are representative of the majority of stakeholders that participate in GME-level simulation at most academic institutions. Our study also did not address the lack of longitudinal competency tracking from the Association of American Medical Colleges Entrustable Professional Activities to the ACGME Milestones.⁵⁰  Simulation-based education and assessment may help address the continuum between undergraduate medical education and GME learning, and this is an area that may allow for future consensus.

This document analysis of ACGME requirements identified a large number of simulation requirements, including mandatory patient safety curricula requirements, for all 21 clinical departments that support our simulation center.

The authors would like to acknowledge support provided by the UT Southwestern Simulation Center and the UT Southwestern Office of Graduate Medical Education. The authors wish to thank the JGME editors and reviewers for their expert feedback and guidance.

The online supplementary data contains a visual abstract.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.