CLER and COVID-19: A Formative Assessment of Clinical Learning Environments During the Pandemic

The COVID-19 pandemic has had a significant impact on health care, resulting in major shifts in care delivery and varied organizational responses that are compounded by social, economic, and financial stressors. Since early 2020, many clinical sites that serve as CLEs for GME have had to adapt and change in response to the pandemic. The CLER program, by way of its structured formative evaluation design, has the potential to inform key stakeholders as they navigate the unknown. In this context, the CLER Program developed and implemented a special site visit protocol to assess the impact of COVID-19 on the major CLEs of ACGME-accredited Sponsoring Institutions to provide the executive and GME leadership of these sites with information to inform decision-making aimed at optimizing patient care and learning. Additionally, the findings in aggregate can inform the ACGME and GME community at large of the pandemic's potential impact on GME for the near future.

With 8 years of experience in developing and conducting formative assessments of CLEs, the CLER Program was positioned to expeditiously design, test, and implement this special assessment—the CLER COVID-19 protocol—across a range of Sponsoring Institutions in an 18-month time frame. From June 2020 to January 2021, the CLER Program designed the protocol and conducted numerous rounds of pilot testing.

In designing the protocol, the CLER Program utilized a mixed methods approach to data gathering and analysis (ie, both quantitative and qualitative inquiry) to assess the impact of COVID-19 as it relates to the 6 CLER Focus Areas of Patient Safety, Health Care Quality (including health care disparities), Teaming, Supervision, Well-Being, and Professionalism. For this protocol, the CLER Program also explores issues around health disparities and other factors impacting the CLE (eg, the rapid rise of telemedicine, the establishment of new COVID-19 care units, COVID-19 testing and quarantining of clinical care team members, and organizational financial impact).

Recognizing that the COVID-19 pandemic has placed significant stress on CLEs and the ACGME, this protocol is designed to minimize the level of burden associated with the site visit. In scheduling the site visits, the CLER team avoids contacting any Sponsoring Institution that is actively in the ACGME emergency categorization.¹ 

The visits are limited to no more than 7 hours of total time for each site and are conducted remotely to protect the safety of both the CLEs and the CLER Field Representatives. The protocol includes interviews with the clinical site's executive and GME leadership, resident and fellow physicians, program directors, and the CLE's leaders of patient safety, quality, health disparities, and health care disparities. Many of the protocol questions are designed to understand how the CLE and GME community at that clinical site envisions the impact of the pandemic in the near future (ie, 2-year horizon).

The CLER COVID-19 protocol completed the design and testing phase and launched in February 2021, assessing the CLEs for a stratified random sample of approximately 300 Sponsoring Institutions with anticipated completion in February 2022.

The findings from this protocol will be available to a number of stakeholders. Each Sponsoring Institution visited receives an individualized written report. In addition, the ACGME Board of Directors receives interim reports of aggregate, de-identified information throughout the 1-year time frame of the protocol. These interim reports also provide the basis for periodic updates to the ACGME's internal community (eg, CLER Evaluation Committee, Institutional Review Committee, and Residency Review Committees) and the external GME community at large. The CLER Program will also publish a national report of aggregate data at the conclusion of the protocol.

As noted above, the CLER Program's plan for disseminating the aggregate findings from this protocol differs from other CLER cycles in that it includes several interim looks at key points in the 1-year time frame. The dissemination plan involves presenting early impressions after approximately one-third of the visits are completed, selecting interim findings around the midpoint of the protocol's cycle, and providing a comprehensive report of findings in 2022 at the protocol's completion.

The following are some of the highlights of the early impressions informed by the visits conducted between October 19, 2020 and April 9, 2021 to the major CLEs of 84 ACGME-accredited Sponsoring Institutions. Twenty-five of these sites had been on ACGME COVID-19 emergency status.

The early impressions were formed from debriefing sessions with the CLER Field Representatives and reflect their overall gestalt of the learning from the visits early in the cycle. Quantitative and qualitative results will follow later in the year (interim findings) and at the conclusion of the protocol.

The impressions are presented in 2 groupings—the first are findings in areas that are outside the CLER Focus Areas and the second reflect findings that are within each of the 6 CLER Focus Areas.

In general, CLE leadership described the overall impact of the pandemic experience to date as one that they have adequately managed with many lessons learned about their organizational capacity, flexibility, and crisis management ability.

• ▪

  In general, the CLE leadership described manageable financial impact. In a number of CLEs, they indicated that additional government assistance (eg, the Coronavirus Aid, Relief, and Economic Security Act) allowed them to achieve neutral to positive financial short-term outcomes from 2020 to 2021.

• ▪

  While each CLE had a different approach to crisis management, CLE leadership depended on workforce flexibility and often workforce capacity management (eg, furloughs and layoffs) to manage financial impact early in the pandemic. In many CLEs, executive leadership also noted acute needs to employ select temporary clinical staff—specifically travelling nurses. They noted that these experiences may have residual impact on workforce planning in the future, primarily for nonphysician members of the CLE workforce.

• ▪

  When asked about major changes in business and clinical operations resulting from COVID-19 that will likely persist over the next 2 years, some CLE leadership emphasized different aspects of business and clinical operations based on the timing of when they entered and exited their most acute phase of surge in patients with COVID-19 during the pandemic (eg, NYC/East Coast, Midwest, rural).

• ▪

  CLEs varied in their plans for managing their major capital investments in the next 2 years. Some deferred or delayed decisions to proceed with pre-pandemic plans (eg, implementing new electronic health records). Others redirected funds to invest in build-out of floors or units (eg, negative pressure rooms, additional ICUs) that could better address infectious diseases both now and into the future (eg, to address the next virus or potential pandemic). Others indicated that the pandemic has not affected their current business operations or future plans for capital investment.

• ▪

  In many CLEs, executive leadership noted that lower patient volumes (emergency department [ED] and inpatient) experienced during the acute phases of the pandemic have not rebounded to pre-pandemic levels and are anticipated to continue for the next several years. This caused a shift in emphasis on enhancing outpatient treatment modalities, including expansion of ambulatory care sites (and rural services), telemedicine, hospital at home programs, remote monitoring technologies (eg, home heart, blood pressure, pulse oximetry monitoring), and new clinics for patients with prolonged COVID-19 symptoms and/or sequelae of infection. CLE and GME leaders also expressed concern that, over the next several years, patients presenting to the ED and inpatient settings will be of higher acuity due to deferred care.

• ▪

  Nearly all CLEs quickly and dramatically implemented or increased use of telemedicine during the acute phases of the pandemic. This served a critical need during the acute phases, but it also raised many questions and concerns for the future, including:

  • ○

    Amplification of remote-based care due to COVID-19 created new patient safety risks related to the increased complexity of diagnosing and monitoring conditions during these visits without the ability to perform a physical examination (eg, many CLEs indicated telemedicine visits were delivered via telephone only with no access to video).

  • ○

    Challenges in connectivity, making diagnoses, and managing therapeutics via telemedicine led to increases in deferrals of needed care and increases in patients presenting to the ED with late-stage disease.

  • ○

    Residents, fellows, and program directors noted increased risk for patient safety when patients were not forthcoming on calls due to lack of privacy in the home setting.

  • ○

    Residents, fellows, and program directors identified patient safety vulnerabilities associated with a lack of well-defined guidance/protocols that outline criteria for appropriateness for use of telemedicine versus in-person visits.

  • ○

    Residents, fellows, and program directors noted numerous challenges and disparities in patient access to and capabilities of navigating various devices and technology platforms for conducting telemedicine visits.

  • ○

    Questions arose as to how to train residents, fellows, faculty members, and patients in using telemedicine approaches to patient care and how to address resident and fellow supervision.

  • ○

    Questions arose on the future of reimbursement for telemedicine (eg, Centers for Medicare & Medicaid Services and other payors).

• ▪

  In many CLEs, executive leadership indicated they were experiencing challenges in maintaining adequate workforce, especially nursing staff, and anticipated this will continue for the next several years.

  • ○

    In the short term, this caused leadership to divert resources to temporary and traveling agencies. Integrated health care systems also balanced/redeployed staffing resources (physicians and nurses) across their various hospitals.

  • ○

    For the long term, some CLEs are considering new approaches to staffing models utilizing advanced practice providers, nurse technicians, and student nurses with increased expectations and efforts to cross-train existing and incoming staff.

• ▪

  In many CLEs, CLE and GME leaders noted lasting delays in patient throughput, both in and out of their inpatient and ambulatory surgical facilities, due to the need for COVID-19 testing and infection prevention measures. They expressed uncertainty as to the impact of these delays over the next several years (eg, delays in admitting to floors pending COVID-19 test results, delays transferring to acute rehab and skilled nursing facilities due to policies on testing and acceptance of COVID-19 patients) and recognized the need to address capacity for high volume rapid testing.

• ▪

  In many CLEs, CLE and GME leaders identified changes to both provider and patient processes for communication that abruptly changed during the acute phases of the pandemic and are expected to continue as safety precautions for the next several years. For example:

  • ○

    Restrictions on number of providers on rounds and in patient rooms

  • ○

    Restrictions on patient visitors

  • ○

    Increased use of computer technologies (eg, tablets, virtual platforms) for provider, patient, and visitor/family conferencing

These changes were viewed as having both positive and negative attributes (eg, ease of connectivity results in greater, more timely participation; lack of in-person interactions negatively affects well-being of patients and providers, including residents and fellows).

• ▪

  GME leaders expressed concern over resident and fellow gaps in training and experience resulting from the pandemic and the negative impact on preparedness for independent practice.

• ▪

  GME leaders expressed concern over transition from undergraduate medical education to GME and the need to provide additional training over the next several years to address gaps in training caused by the pandemic.

• ▪

  In general, across CLEs the number of patient safety event reports appeared to be the same or lower, the number of patient safety event investigations appeared to be lower, and resident and fellow involvement in patient safety event investigations seemed to have decreased.

• ▪

  Few residents, fellows, and program directors indicated that they were involved in patient safety event investigations related to COVID-19.

• ▪

  Few CLEs appeared to have goals for involving residents and fellows in patient safety event investigations unless they were part of the event.

• ▪

  CLE and GME leaders varied widely in describing patient safety event investigations—from brief departmental huddles, to morbidity and mortality conferences, to formal root cause analyses.

• ▪

  CLE leadership indicated COVID-19 largely confirmed what they already knew about health and health care disparities in their patient populations. This has resulted in several new actions:

  • ○

    Mandatory collection of COVID-19-related data by subpopulations (eg, race, gender, age)

  • ○

    Deepening of existing or development of new community partnerships to address COVID-19 (testing, access to care, vaccinations)

• ▪

  A limited number of CLEs were applying learning from COVID-19 to identify and address other health and health care disparities.

• ▪

  Many residents and fellows appeared to recognize compliance with or participation in CLE efforts to address quality improvement in day-to-day patient care (eg, compliance with clinical guidelines/checklists/bundles); however, they appeared to do so absent an understanding of the CLE's overarching quality improvement framework, monitoring, and progress.

• ▪

  Few CLE leaders noted issues with resident or fellow supervision resulting from the pandemic that will likely persist for the next 2 years. In general, patient safety and quality leaders appeared to delegate issues of supervision to GME.

• ▪

  Across CLEs, some program directors identified challenges in resident supervision resulting from the pandemic; few indicated they expect these challenges to persist for the next several years.

• ▪

  CLE leaders varied in their descriptions of the challenges associated with supervising residents throughout the rapid deployment of telemedicine. The degree to which these challenges impacted the training program appeared to be specialty-specific—with significant effects noted by the primary care specialties. Many noted challenges such as:

  • ○

    Lack of standardized guidance for determining when a visit is to be conducted remotely or in person (such as when telephone only is acceptable)

  • ○

    Lack of standardized processes for coordinating care across the team while on a remote visit (ordering tests, follow-up appointments, or arranging for interpreter services as part of the visit)

  • ○

    Inability of faculty members to supervise multiple residents scheduled for simultaneous remote visits

  • ○

    Inability of faculty members to supervise when resident and faculty members are in different locations (limitations with devices, software, etc)

  • ○

    Challenges with supervision of remote visits when patients experience difficulties connecting to video conferencing or when videoconferencing platforms do not allow for more than 2 simultaneous participants

• ▪

  Across CLEs, executive leadership recognized the pandemic as a major stressor to their workforce with a negative impact on provider well-being.

• ▪

  The majority of CLE well-being efforts are focused on providing resources for an individual's acute needs and resiliency; few CLEs were addressing system-level factors that negatively impact well-being.

• ▪

  Many CLEs appeared to focus on addressing individual behaviors rather than measuring or assessing the overall culture of professionalism.

At the time of this publication, the CLER COVID-19 protocol is actively ongoing, while across the United States many CLEs are experiencing an upsurge in patients with COVID-19 due to the Delta variant. More than half of the sample of Sponsoring Institutions' visits have been completed, and the CLER Program is in the process of reviewing the data to identify emerging themes that will inform the next interim report.

Although early in the process of completing the 1-year protocol, the CLER Program's findings to date notably highlight the abilities of the CLE and GME communities to withstand the major impact of this pandemic. The nation's CLEs appear to be learning a number of important lessons, including how GME is adapting to the changing environment.

Some of the key areas to watch include attention to new and/or evolving models of health care—such as home health and telemedicine—and the pandemic's impact on the clinical care team, including the stressors associated with workforce challenges in recruiting and retaining nurses, respiratory therapists, and other members of the clinical care team. Changes impacting the clinical care team will directly affect how physicians, including residents and fellows, care for patients. Another area to watch is the cumulative effect these health system changes are having on GME faculty, program directors, and designated institutional officials—as they may find themselves at a nexus of various stressors induced by the pandemic. In addition, the residents and fellows have identified numerous opportunities and challenges resulting from changes in how care is being delivered and how they have had to adapt to new approaches of working and learning together. As the CLER COVID-19 protocol progresses, it will be important to understand how the early impressions noted above evolve over time.

Overall, the COVID-19 pandemic has created new challenges and opportunities in the way that health care and GME are delivered. In the coming months, the CLER COVID-19 protocol will continue to explore how CLEs are adapting and learning how to best emerge from this major public health emergency.