Background Internal medicine (IM) resident physicians spend a considerable amount of time managing their inbox as part of their longitudinal continuity clinic experience. There are no standardized guidelines for how programs should train, monitor, or supervise residents in this type of patient care.
Objective To understand how IM residency programs educate, monitor, and supervise resident electronic health record (EHR) inbox management as part of their longitudinal continuity clinic and determine whether patient safety events have occurred due to EHR inbox-related patient care decisions made by unsupervised resident physicians.
Methods In August 2021, 439 program directors at accredited US IM residency programs who were members of the Association of Program Directors in Internal Medicine (APDIM) were asked 12 questions developed by the study authors and APDIM survey committee members regarding resident EHR inbox management as part of the annual APDIM survey.
Results Two hundred and sixty-seven (61%) PDs responded. The majority (224 of 267, 84%) of programs provided guidelines for expected message response times; less than half (115, 43%) monitored timeliness metrics. Only half (135; 51%) of programs required faculty supervision of inbox messages for all residents; 28% (76) did not require supervision for any residents. Twenty-one percent of PDs (56) reported awareness of a patient safety event occurring due to an unsupervised resident inbox-related patient care decision.
Conclusions Substantial variability exists in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. Program directors are aware of patient safety events resulting from unsupervised resident inbox management.
Introduction
Physicians spend more time interacting with the electronic health record (EHR) than providing face-to-face patient care. For every hour of direct patient care, physicians spend 2 hours completing EHR and desk-related tasks.1,2 An increasing amount of time is spent in the EHR inbox, including the review and documentation of test results, management of refill requests, and communication with patients and other health care team members. This has implications for well-being, as the amount of time spent in the inbox is associated with job dissatisfaction and burnout.3
Primary care physicians spend nearly one hour per day managing their inbox, with 37% of this time occurring outside of work hours.3-7 Studies examining EHR practices among resident physicians demonstrate a similar amount of time is spent in the EHR and inbox while in the inpatient setting.8,9 A comparable amount of inbox work can be assumed for internal medicine (IM) resident physicians who manage a panel of patients as part of their “longitudinal continuity experience,” as defined by the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education (GME) in IM.10 Due to the hybrid inpatient-outpatient nature of IM residency training, residents may find it challenging to complete outpatient continuity clinic inbox tasks during inpatient rotations.11
Considering that an extensive amount of care is conducted through inbox-related communications, one might expect all residency programs to have guidelines on the training and supervision of residents in this non-face-to-face form of patient care. This seems particularly relevant given the increased volume of inbox messages secondary to the COVID-19 pandemic.12 However, neither the ACGME’s program requirements nor the 2014 Clinical Learning Environment Review (CLER) Pathways to Excellence13 address what training, monitoring, or supervision should look like for this type of patient care. Therefore, individual programs are responsible for determining appropriate levels of EHR inbox training, coverage models, and supervision. With a lack of standardized guidelines for resident supervision, it is possible that EHR inbox-related patient care decisions are being made by unsupervised residents, resulting in possible adverse patient outcomes.
Through a national survey of IM residency program directors (PDs), our primary aim was to understand how programs educate, monitor, and supervise resident EHR inbox management as part of their longitudinal continuity clinic experience. Our secondary aim was to understand whether patient safety events occurred secondary to EHR inbox-related patient care decisions made by unsupervised resident physicians. Findings from this survey could aid graduate medical education programs in recognizing internal opportunities for improvement in their EHR inbox management training and support.
KEY POINTS
Internal medicine residents must manage their electronic health record (EHR) inboxes on their continuity patients, but training and supervision are not standardized for this task.
This survey of internal medicine program directors revealed that there are a broad range of strategies for, and levels of, education and supervision of the EHR for residents. Some were aware of patient safety events resulting from unsupervised inbox management.
This study suggests a need for intentional structure around learning to manage an EHR inbox independently.
Methods
Study Setting and Participants
The Association of Program Directors in Internal Medicine (APDIM) is a founding organization of the Alliance for Academic Internal Medicine (AAIM), a professional association that represents more than 10 000 IM educators and administrators. The APDIM Survey and Scholarship Committee oversees the development of an annual research survey of IM residency PDs to collect essential trend data and to study issues central to GME training. In addition to a section about residency program characteristics, the survey includes a limited number of thematic sections that vary annually.
The 2021 Annual Survey, which included questions about EHR inbox management, was disseminated to PDs from all 439 APDIM member residency programs that were of “initial” or “continued” accreditation status by the ACGME prior to July 1, 2020. Five email reminder messages were sent to nonrespondents. The survey landing page served as the study’s informed consent page, with participation being voluntary and no completion incentives offered. Only AAIM Surveys staff (M.K.) had access to the survey platform and dataset/contacts during fielding.
Survey Instrument
In April 2019, a call for survey topics was disseminated online to APDIM physician-members. The APDIM Survey Committee blind-reviewed question section proposals, scored them on merit and relevance, and selected 3 for inclusion in the survey. In February 2021, the 15-member committee appointed section development lead authors and co-contributors based on relevant experience. Question revisions, pretesting of the complete instrument, and further content revisions occurred from March through June 2021, during which time AAIM Surveys staff programmed the instrument in the Qualtrics Surveys XM platform.14 From late June through mid-July, the web survey was pilot tested for content validity by the survey committee and by 5 members of AAIM’s Research Committee, which consisted of experts in GME that were blinded to the survey committee. The final instrument included the thematic section “Management of the Electronic Health Record Inbox: Education, Monitoring, and Supervision for Internal Medicine Residents” and consisted of 13 questions related to IM resident inbox training and supervision (online supplementary data).
Statistical Analysis
Descriptive statistics are reported as mean (standard deviation) or count (percent of respondents) as appropriate. Program director awareness of patient safety events are categorized as “Yes” versus any other response (“No,” “Unsure,” or “New PD”) for associative analyses. Bivariate associations with safety event(s) are assessed using chi-square tests for nominal valued responses (coverage type, supervision, training, guidelines, monitoring, program type) or Mann-Whitney-Wilcoxon tests for interval valued responses (PD tenure, program size). A significance threshold of α=.01 was used to account for multiple comparisons, with all analyses performed using SAS statistical software (Version 9.4, SAS Institute Inc).
This study was deemed exempt by Pearl IRB.
Results
At the time of the study, APDIM member programs represented 80% of residency programs with ACGME accreditation prior to that date. The survey response rate was 61% (267 of 439 survey-eligible PDs). There was no statistical association between respondents and nonrespondents based on characteristics that closely defined the complete survey population (online supplementary data).
Basic demographic characteristics of program directors are reported in the Table. Formal training on EHR inbox management was reported by 232 of 267 (87%) programs, with only 149 (56%) PDs “very” or “somewhat” satisfied with that training. Most programs (224, 84%) provided guidelines for expected message response times, but less than half (115, 43%) monitored timeliness metrics. Program directors reported a variety of inbox coverage models when a resident was on an inpatient rotation or away—most commonly a faculty member, alternate resident, resident pool, or nursing pool (Figure 1). Additional coverage models are listed in the Box, and were dependent on program size, rotation type, and rotation location.
Residents grouped in trios or quads with ambulatory resident expected to cover coresidents on inpatient rotation (4 + 1 or 6 + 2 models)
Residents grouped in pods and expected to cover each other at all times (always logged into coresident inbox)
Ambulatory resident in clinic assigned “doc of the day” and handles all urgent messages for inpatient or absent residents
Ambulatory Chief Resident covers all residents when inpatient or away
Backup/float pool of residents covers all residents when inpatient or away
Only half (135, 51%) of programs required faculty supervision of inbox messages for all residents, 28% (76) did not require supervision for any of their residents, and 21% (56) required supervision for some residents. Supervision structures varied, but common examples are included in Figure 2. Among the 21% of PDs who reported that some of their residents required supervision, supervision was determined by year of training, licensure status, or milestone/competency based. Only 26% (69) of PDs reported monitoring the amount of time that residents spend in the inbox. More than 1 in 5 PDs (53, 20%) were aware of patient safety events resulting from an unsupervised resident inbox-related patient care decision.
There was a significant association between PD awareness of patient safety events and having residents responsible for selecting another resident to cover their inbox while away (P=.007). There were no other associations between coverage groups and PD awareness of patient safety events (all P>.19; Figure 3). There was no association between faculty supervision of the resident inbox and PD awareness of patient safety events (P=.66). There was no association between method of supervision, formal inbox training, inbox response guidelines, monitoring of inbox timeliness metrics, PD tenure, program size, or program type with PD awareness of patient safety events (all P>.11).
Discussion
Our study shows that there is substantial variability in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. These varying models likely exist due to local team resources, individual residency program characteristics, and continuity clinic staffing models. Delays in patient care and missed diagnoses occur from mismanagement of EHR inbox messages, often secondary to excessive message burden, poor usability, and gaps in communication among team members.15-18 Studies have demonstrated that trainees are less likely than practicing physicians to respond appropriately to inbox alerts.19-20 The ACGME Common Program Requirements allow residency programs flexibility with respect to supervision of residents in general, stating that, “Faculty members must provide appropriate levels of supervision to promote patient safety.” They acknowledge that, “Supervision is contextual due to tremendous diversity of resident patient interactions, education and training locations, and resident skills and abilities even at the same level of the educational program.”10 Interestingly, the ACGME requirements include no guidelines with respect to supervision of nonvisit care activities such as management of the inbox.
Our study found that a considerable number of IM PDs reported a patient safety event secondary to unsupervised resident inbox decision making. Given that a minority of programs monitor inbox timeliness metrics, there may be additional unrecognized care delays and errors. Furthermore, we suspect that only the most egregious patient safety events rise to the level of program director awareness. We anticipate both factors may explain why there was no statistical association between faculty supervision of the resident inbox and PD awareness of patient safety events. We hypothesize that supervising clinical faculty and other members of the health care team such as nurses may be more aware of the frequency of patient safety events. It may be helpful to survey these stakeholders in the future for a more accurate understanding of the magnitude of patient safety events occurring secondary to an unsupervised resident inbox-related decision.
Interestingly, we found a significant association between PD awareness of patient safety events when it was left to the resident to delegate an alternate resident to cover their inbox when they were away. We suspect that inbox delegation may not occur when left to an individual resident, leaving an inbox unmonitored. Inbox coverage structures that do not depend on the individual resident to assign coverage (eg, program assigns resident coverage) or involve other team members (eg, covering nurse pool) may help reduce patient safety events related to inbox-related management.
With respect to EHR training, a recent systematic review found few high-quality interventions to improve EHR-related knowledge and skills among resident physicians.21 Residency programs should consider developing robust EHR curricula to optimize resident handling of inbox messages. Ideally, this curriculum would exist outside of the general EHR onboarding training and be tailored to unique program-based needs. Furthermore, it should be delivered longitudinally, as prior studies have shown improvement in resident self-assessed EHR competence with ongoing and cumulative training opportunities.22 Studies have shown that simulation has been effective at improving resident satisfaction with and navigation of the EHR, and perhaps this would be a modality to explore for inbox training.23
The ACGME requires that residency programs monitor and adhere to work hours, which should include time spent in nonvisit care tasks such as inbox management.10 Studies conducted on primary care physicians have shown that approximately 90 minutes are spent in the EHR after hours, with a substantial amount of this time in the inbox.4 Given that only one-quarter of programs monitor time spent in the inbox, work hour violations may be occurring due to after-hours inbox work.24 There is a known relationship between increased volume of inox messages and burnout in physicians, with downstream consequences, including decreased job satisfaction and increased likelihood of reducing clinical work time.25 A study examining the impact of after-hours EHR work on burnout found a direct relationship between time spent on EHRs and burnout in primary care residents and faculty.26 Residency programs could leverage EHR platforms to track the time of day residents are in the inbox to identify EHR work being done outside of traditional work hours. This information may help identify residents who are struggling with inbox management and who are at risk for burnout.
There is an inherent tension between autonomy and supervision in residency training. This fueled the emergence of competency-based medical education, where residents “earn” autonomy by graduated progression through milestone-based assessment of different competencies.27-30 Direct clinical observations or mini-clinical evaluation exercises are helpful tools that faculty utilize to determine the appropriate amount of autonomy to grant a resident for different entrustable professional activities.31 There is no standardized equivalent to assess resident competence for nonvisit care tasks, including inbox management. Perhaps a comparable graduated entrustment for inbox work would ensure residents were granted appropriate autonomy while still prioritizing patient safety. Potential ways to accomplish this could include direct observation of resident inbox work or review of clinical records to ensure task completion. Lastly, our study only captured the perspective of the PD, who may have limited knowledge of “boots on the ground” EHR coverage types and supervision structures. Further work is needed to explore resident perception of the EHR burden, faculty inbox support, and the overall impact on well-being.
Conclusions
Substantial variability exists in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. One in 5 PDs were aware of at least one patient safety event resulting from unsupervised resident inbox management.
The authors would like to thank the Alliance for Academic Internal Medicine (AAIM) Surveys staff for their technical and administrative support of this project; the members of the APDIM Survey and Scholarship Committee who pretested and live pilot-tested the survey; and they extend their appreciation to the IM residency PDs who completed the survey.
References
Editor’s Note
The online version of this article contains the survey used in the study and essential characteristics of responding and nonresponding internal medicine residency programs from the 2021 APDIM Annual Survey of Internal Medicine Residency Program Directors.
Author Notes
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
This work was previously presented at the Association of Program Directors in Internal Medicine (APDIM) Fall Meeting, September 22-24, 2022, San Diego, California, USA.