Background Internal medicine (IM) resident physicians spend a considerable amount of time managing their inbox as part of their longitudinal continuity clinic experience. There are no standardized guidelines for how programs should train, monitor, or supervise residents in this type of patient care.

Objective To understand how IM residency programs educate, monitor, and supervise resident electronic health record (EHR) inbox management as part of their longitudinal continuity clinic and determine whether patient safety events have occurred due to EHR inbox-related patient care decisions made by unsupervised resident physicians.

Methods In August 2021, 439 program directors at accredited US IM residency programs who were members of the Association of Program Directors in Internal Medicine (APDIM) were asked 12 questions developed by the study authors and APDIM survey committee members regarding resident EHR inbox management as part of the annual APDIM survey.

Results Two hundred and sixty-seven (61%) PDs responded. The majority (224 of 267, 84%) of programs provided guidelines for expected message response times; less than half (115, 43%) monitored timeliness metrics. Only half (135; 51%) of programs required faculty supervision of inbox messages for all residents; 28% (76) did not require supervision for any residents. Twenty-one percent of PDs (56) reported awareness of a patient safety event occurring due to an unsupervised resident inbox-related patient care decision.

Conclusions Substantial variability exists in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. Program directors are aware of patient safety events resulting from unsupervised resident inbox management.

Physicians spend more time interacting with the electronic health record (EHR) than providing face-to-face patient care. For every hour of direct patient care, physicians spend 2 hours completing EHR and desk-related tasks.1,2  An increasing amount of time is spent in the EHR inbox, including the review and documentation of test results, management of refill requests, and communication with patients and other health care team members. This has implications for well-being, as the amount of time spent in the inbox is associated with job dissatisfaction and burnout.3 

Primary care physicians spend nearly one hour per day managing their inbox, with 37% of this time occurring outside of work hours.3-7  Studies examining EHR practices among resident physicians demonstrate a similar amount of time is spent in the EHR and inbox while in the inpatient setting.8,9  A comparable amount of inbox work can be assumed for internal medicine (IM) resident physicians who manage a panel of patients as part of their “longitudinal continuity experience,” as defined by the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education (GME) in IM.10  Due to the hybrid inpatient-outpatient nature of IM residency training, residents may find it challenging to complete outpatient continuity clinic inbox tasks during inpatient rotations.11 

Considering that an extensive amount of care is conducted through inbox-related communications, one might expect all residency programs to have guidelines on the training and supervision of residents in this non-face-to-face form of patient care. This seems particularly relevant given the increased volume of inbox messages secondary to the COVID-19 pandemic.12  However, neither the ACGME’s program requirements nor the 2014 Clinical Learning Environment Review (CLER) Pathways to Excellence13  address what training, monitoring, or supervision should look like for this type of patient care. Therefore, individual programs are responsible for determining appropriate levels of EHR inbox training, coverage models, and supervision. With a lack of standardized guidelines for resident supervision, it is possible that EHR inbox-related patient care decisions are being made by unsupervised residents, resulting in possible adverse patient outcomes.

Through a national survey of IM residency program directors (PDs), our primary aim was to understand how programs educate, monitor, and supervise resident EHR inbox management as part of their longitudinal continuity clinic experience. Our secondary aim was to understand whether patient safety events occurred secondary to EHR inbox-related patient care decisions made by unsupervised resident physicians. Findings from this survey could aid graduate medical education programs in recognizing internal opportunities for improvement in their EHR inbox management training and support.

What Is Known

Internal medicine residents must manage their electronic health record (EHR) inboxes on their continuity patients, but training and supervision are not standardized for this task.

What Is New

This survey of internal medicine program directors revealed that there are a broad range of strategies for, and levels of, education and supervision of the EHR for residents. Some were aware of patient safety events resulting from unsupervised inbox management.

Bottom Line

This study suggests a need for intentional structure around learning to manage an EHR inbox independently.

Study Setting and Participants

The Association of Program Directors in Internal Medicine (APDIM) is a founding organization of the Alliance for Academic Internal Medicine (AAIM), a professional association that represents more than 10 000 IM educators and administrators. The APDIM Survey and Scholarship Committee oversees the development of an annual research survey of IM residency PDs to collect essential trend data and to study issues central to GME training. In addition to a section about residency program characteristics, the survey includes a limited number of thematic sections that vary annually.

The 2021 Annual Survey, which included questions about EHR inbox management, was disseminated to PDs from all 439 APDIM member residency programs that were of “initial” or “continued” accreditation status by the ACGME prior to July 1, 2020. Five email reminder messages were sent to nonrespondents. The survey landing page served as the study’s informed consent page, with participation being voluntary and no completion incentives offered. Only AAIM Surveys staff (M.K.) had access to the survey platform and dataset/contacts during fielding.

Survey Instrument

In April 2019, a call for survey topics was disseminated online to APDIM physician-members. The APDIM Survey Committee blind-reviewed question section proposals, scored them on merit and relevance, and selected 3 for inclusion in the survey. In February 2021, the 15-member committee appointed section development lead authors and co-contributors based on relevant experience. Question revisions, pretesting of the complete instrument, and further content revisions occurred from March through June 2021, during which time AAIM Surveys staff programmed the instrument in the Qualtrics Surveys XM platform.14  From late June through mid-July, the web survey was pilot tested for content validity by the survey committee and by 5 members of AAIM’s Research Committee, which consisted of experts in GME that were blinded to the survey committee. The final instrument included the thematic section “Management of the Electronic Health Record Inbox: Education, Monitoring, and Supervision for Internal Medicine Residents” and consisted of 13 questions related to IM resident inbox training and supervision (online supplementary data).

Statistical Analysis

Descriptive statistics are reported as mean (standard deviation) or count (percent of respondents) as appropriate. Program director awareness of patient safety events are categorized as “Yes” versus any other response (“No,” “Unsure,” or “New PD”) for associative analyses. Bivariate associations with safety event(s) are assessed using chi-square tests for nominal valued responses (coverage type, supervision, training, guidelines, monitoring, program type) or Mann-Whitney-Wilcoxon tests for interval valued responses (PD tenure, program size). A significance threshold of α=.01 was used to account for multiple comparisons, with all analyses performed using SAS statistical software (Version 9.4, SAS Institute Inc).

This study was deemed exempt by Pearl IRB.

At the time of the study, APDIM member programs represented 80% of residency programs with ACGME accreditation prior to that date. The survey response rate was 61% (267 of 439 survey-eligible PDs). There was no statistical association between respondents and nonrespondents based on characteristics that closely defined the complete survey population (online supplementary data).

Basic demographic characteristics of program directors are reported in the Table. Formal training on EHR inbox management was reported by 232 of 267 (87%) programs, with only 149 (56%) PDs “very” or “somewhat” satisfied with that training. Most programs (224, 84%) provided guidelines for expected message response times, but less than half (115, 43%) monitored timeliness metrics. Program directors reported a variety of inbox coverage models when a resident was on an inpatient rotation or away—most commonly a faculty member, alternate resident, resident pool, or nursing pool (Figure 1). Additional coverage models are listed in the Box, and were dependent on program size, rotation type, and rotation location.

Table

Demographic Data of 267 Association of Program Directors in Internal Medicine Member Program Directors From 2021 Survey

Demographic Data of 267 Association of Program Directors in Internal Medicine Member Program Directors From 2021 Survey
Demographic Data of 267 Association of Program Directors in Internal Medicine Member Program Directors From 2021 Survey
Figure 1

Inbox Coverage Types When Resident Was on an Inpatient Rotation (N=231) or Away (N=267)

Figure 1

Inbox Coverage Types When Resident Was on an Inpatient Rotation (N=231) or Away (N=267)

Close modal
Box Additional Coverage Models Outlined by Responders

Residents grouped in trios or quads with ambulatory resident expected to cover coresidents on inpatient rotation (4 + 1 or 6 + 2 models)

Residents grouped in pods and expected to cover each other at all times (always logged into coresident inbox)

Ambulatory resident in clinic assigned “doc of the day” and handles all urgent messages for inpatient or absent residents

Ambulatory Chief Resident covers all residents when inpatient or away

Backup/float pool of residents covers all residents when inpatient or away

Only half (135, 51%) of programs required faculty supervision of inbox messages for all residents, 28% (76) did not require supervision for any of their residents, and 21% (56) required supervision for some residents. Supervision structures varied, but common examples are included in Figure 2. Among the 21% of PDs who reported that some of their residents required supervision, supervision was determined by year of training, licensure status, or milestone/competency based. Only 26% (69) of PDs reported monitoring the amount of time that residents spend in the inbox. More than 1 in 5 PDs (53, 20%) were aware of patient safety events resulting from an unsupervised resident inbox-related patient care decision.

Figure 2

Supervision Structures for Resident Inbox Messages (N=191)

Figure 2

Supervision Structures for Resident Inbox Messages (N=191)

Close modal

There was a significant association between PD awareness of patient safety events and having residents responsible for selecting another resident to cover their inbox while away (P=.007). There were no other associations between coverage groups and PD awareness of patient safety events (all P>.19; Figure 3). There was no association between faculty supervision of the resident inbox and PD awareness of patient safety events (P=.66). There was no association between method of supervision, formal inbox training, inbox response guidelines, monitoring of inbox timeliness metrics, PD tenure, program size, or program type with PD awareness of patient safety events (all P>.11).

Figure 3

Program Director Awareness of Patient Safety Events by Inbox Coverage Type When Resident Was Away (N=267)

Figure 3

Program Director Awareness of Patient Safety Events by Inbox Coverage Type When Resident Was Away (N=267)

Close modal

Our study shows that there is substantial variability in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. These varying models likely exist due to local team resources, individual residency program characteristics, and continuity clinic staffing models. Delays in patient care and missed diagnoses occur from mismanagement of EHR inbox messages, often secondary to excessive message burden, poor usability, and gaps in communication among team members.15-18  Studies have demonstrated that trainees are less likely than practicing physicians to respond appropriately to inbox alerts.19-20  The ACGME Common Program Requirements allow residency programs flexibility with respect to supervision of residents in general, stating that, “Faculty members must provide appropriate levels of supervision to promote patient safety.” They acknowledge that, “Supervision is contextual due to tremendous diversity of resident patient interactions, education and training locations, and resident skills and abilities even at the same level of the educational program.”10  Interestingly, the ACGME requirements include no guidelines with respect to supervision of nonvisit care activities such as management of the inbox.

Our study found that a considerable number of IM PDs reported a patient safety event secondary to unsupervised resident inbox decision making. Given that a minority of programs monitor inbox timeliness metrics, there may be additional unrecognized care delays and errors. Furthermore, we suspect that only the most egregious patient safety events rise to the level of program director awareness. We anticipate both factors may explain why there was no statistical association between faculty supervision of the resident inbox and PD awareness of patient safety events. We hypothesize that supervising clinical faculty and other members of the health care team such as nurses may be more aware of the frequency of patient safety events. It may be helpful to survey these stakeholders in the future for a more accurate understanding of the magnitude of patient safety events occurring secondary to an unsupervised resident inbox-related decision.

Interestingly, we found a significant association between PD awareness of patient safety events when it was left to the resident to delegate an alternate resident to cover their inbox when they were away. We suspect that inbox delegation may not occur when left to an individual resident, leaving an inbox unmonitored. Inbox coverage structures that do not depend on the individual resident to assign coverage (eg, program assigns resident coverage) or involve other team members (eg, covering nurse pool) may help reduce patient safety events related to inbox-related management.

With respect to EHR training, a recent systematic review found few high-quality interventions to improve EHR-related knowledge and skills among resident physicians.21  Residency programs should consider developing robust EHR curricula to optimize resident handling of inbox messages. Ideally, this curriculum would exist outside of the general EHR onboarding training and be tailored to unique program-based needs. Furthermore, it should be delivered longitudinally, as prior studies have shown improvement in resident self-assessed EHR competence with ongoing and cumulative training opportunities.22  Studies have shown that simulation has been effective at improving resident satisfaction with and navigation of the EHR, and perhaps this would be a modality to explore for inbox training.23 

The ACGME requires that residency programs monitor and adhere to work hours, which should include time spent in nonvisit care tasks such as inbox management.10  Studies conducted on primary care physicians have shown that approximately 90 minutes are spent in the EHR after hours, with a substantial amount of this time in the inbox.4  Given that only one-quarter of programs monitor time spent in the inbox, work hour violations may be occurring due to after-hours inbox work.24  There is a known relationship between increased volume of inox messages and burnout in physicians, with downstream consequences, including decreased job satisfaction and increased likelihood of reducing clinical work time.25  A study examining the impact of after-hours EHR work on burnout found a direct relationship between time spent on EHRs and burnout in primary care residents and faculty.26  Residency programs could leverage EHR platforms to track the time of day residents are in the inbox to identify EHR work being done outside of traditional work hours. This information may help identify residents who are struggling with inbox management and who are at risk for burnout.

There is an inherent tension between autonomy and supervision in residency training. This fueled the emergence of competency-based medical education, where residents “earn” autonomy by graduated progression through milestone-based assessment of different competencies.27-30  Direct clinical observations or mini-clinical evaluation exercises are helpful tools that faculty utilize to determine the appropriate amount of autonomy to grant a resident for different entrustable professional activities.31  There is no standardized equivalent to assess resident competence for nonvisit care tasks, including inbox management. Perhaps a comparable graduated entrustment for inbox work would ensure residents were granted appropriate autonomy while still prioritizing patient safety. Potential ways to accomplish this could include direct observation of resident inbox work or review of clinical records to ensure task completion. Lastly, our study only captured the perspective of the PD, who may have limited knowledge of “boots on the ground” EHR coverage types and supervision structures. Further work is needed to explore resident perception of the EHR burden, faculty inbox support, and the overall impact on well-being.

Substantial variability exists in how IM residency programs train, monitor, supervise, and provide coverage for resident inbox work. One in 5 PDs were aware of at least one patient safety event resulting from unsupervised resident inbox management.

The authors would like to thank the Alliance for Academic Internal Medicine (AAIM) Surveys staff for their technical and administrative support of this project; the members of the APDIM Survey and Scholarship Committee who pretested and live pilot-tested the survey; and they extend their appreciation to the IM residency PDs who completed the survey.

1. 
Sinsky
CA,
Dyrbye
LN,
West
CP,
Satele
D,
Tutty
M,
Shanafelt
TD.
Professional satisfaction and the career plans of US physicians
.
Mayo Clin Proc
.
2017
;
92
(11)
:
1625
-
1635
.
2. 
Young
RA,
Burge
SK,
Kumar
KA,
Wilson
JM,
Ortiz
DF.
A time-motion study of primary care physicians’ work in the electronic health record era
.
Fam Med
.
2018
;
50
(2)
:
91
-
99
.
3. 
Murphy
DR,
Satterly
T,
Giardina
TD,
Sittig
DF,
Singh
H.
Practicing clinicians’ recommendations to reduce burden from the electronic health record inbox: a mixed-methods study
.
J Gen Intern Med
.
2019
;
34
(9)
:
1825
-
1832
.
4. 
Arndt
BG,
Beasley
JW,
Watkinson
MD,
et al.
Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations
.
Ann Fam Med
.
2017
;
15
(5)
:
419
-
426
.
5. 
Murphy
DR,
Meyer
AN,
Russo
E,
Sittig
DF,
Wei
L,
Singh
H.
The burden of inbox notifications in commercial electronic health records
.
JAMA Intern Med
.
2016
;
176
(4)
:
559
-
560
.
6. 
Murphy
DR,
Reis
B,
Sittig
DF,
Singh
H.
Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis
.
Am J Med
.
2012
;
125
(2)
:
209.e1
-
e7
.
7. 
Akbar
F,
Mark
G,
Warton
EM,
et al.
Physicians’ electronic inbox work patterns and factors associated with high inbox work duration
.
J Am Med Inform Assoc
.
2021
;
28
(5)
:
923
-
930
.
8. 
Chaiyachati
KH,
Shea
JA,
Asch
DA,
et al.
Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations
.
JAMA Intern Med
.
2019
;
179
(6)
:
760
-
767
.
9. 
Alromaihi
D,
Godfrey
A,
Dimoski
T,
Gunnels
P,
Scher
E,
Baker-Genaw
K.
Internal medicine residents’ time study: paperwork versus patient care
.
J Grad Med Educ
.
2011
;
3
(4)
:
550
-
553
.
10. 
Accreditation Council for Graduate Medical Education
.
ACGME Program Requirements for Graduate Medical Education in Internal Medicine
.
11. 
Jagannath
AD,
Nabors
C,
Southern
W,
Schlair
S,
Conigliaro
R.
Resident inbox task completion is improved with a single electronic health record (EHR) system
.
J Gen Intern Med
.
2021
;
36
(3)
:
815
-
817
.
12. 
Matulis
JC,
McCoy
R,
Liu
SK.
Moving away from chaos: intentional and adaptive management of the non-visit care river
.
J Gen Intern Med
.
2023
;
38
(3)
:
784
-
788
.
13. 
Accreditation Council for Graduate Medical Education
.
Clinical Learning Environment Review (CLER) Program
.
14. 
StataCorp.
Stata Statistical Software: Release 16
.
Accessed September 26, 2023. https://www.stata.com/
15. 
Barbieri
AL,
Fadare
O,
Fan
L,
Singh
H,
Parkash
V.
Challenges in communication from referring clinicians to pathologists in the electronic health record era
.
J Pathol Inform
.
2018
;
9
:
8
.
16. 
Singh
H,
Thomas
EJ,
Sittig
DF,
et al.
Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain?
Am J Med
.
2010
;
123
(3)
:
238
-
244
.
17. 
Hysong
SJ,
Sawhney
MK,
Wilson
L,
et al.
Understanding the management of electronic test result notifications in the outpatient setting
.
BMC Med Inform Decis Mak
.
2011
;
11
:
22
.
18. 
Mazur
LM,
Mosaly
PR,
Moore
C,
Marks
L.
Association of the usability of electronic health records with cognitive workload and performance levels among physicians
.
JAMA Netw Open
.
2019
;
2
(4)
:
e191709
.
19. 
Singh
H,
Thomas
EJ,
Mani
S,
et al.
Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?
Arch Intern Med
.
2009
;
169
(17)
:
1578
-
1586
.
20. 
Menon
S,
Smith
MW,
Sittig
DF,
et al.
How context affects electronic health record-based test results follow-up: a mixed-methods evaluation
.
BMJ Open
.
2014
;
4
(11)
:
e005985
.
21. 
Rajaram
A,
Hickey
Z,
Patel
N,
Newbigging
J,
Wolfrom
B.
Training medical students and residents in the use of electronic health records: a systematic review of the literature
.
J Am Med Inform Assoc
.
2020
;
27
(1)
:
175
-
180
.
22. 
Kim
JG,
Rodriguez
HP,
Estlin
KA,
Morris
CG.
Impact of longitudinal electronic health record training for residents preparing for practice in patient-centered medical homes
.
Perm J
.
2017
;
21
:
16
-
122
.
23. 
Miller
ME,
Scholl
G,
Corby
S,
Mohan
V,
Gold
JA.
The impact of electronic health record-based simulation during intern boot camp: interventional study
.
JMIR Med Educ
.
2021
;
7
(1)
:
e25828
.
24. 
Deano
RC,
DeKosky
A,
Appannagari
A,
et al.
Resident time spent in clinical and educational activities at home: implications for duty hours
.
Arch Intern Med
.
2011
;
171
(11)
:
1038
-
1039
.
25. 
Tai-Seale
M,
Dillon
EC,
Yang
Y,
et al.
Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records
.
Health Aff (Millwood)
.
2019
;
38
(7)
:
1073
-
1078
.
26. 
Robertson
SL,
Robinson
MD,
Reid
A.
Electronic health record effects on work-life balance and burnout within the I(3) Population Collaborative
.
J Grad Med Educ
.
2017
;
9
(4)
:
479
-
484
.
27. 
Green
ML,
Aagaard
EM,
Caverzagie
KJ,
et al.
Charting the road to competence: developmental milestones for internal medicine residency training
.
J Grad Med Educ
.
2009
;
1
(1)
:
5
-
20
.
28. 
ten Cate
O.
Nuts and bolts of entrustable professional activities
.
J Grad Med Educ
.
2013
;
5
(1)
:
157
-
158
.
29. 
Sterkenburg
A,
Barach
P,
Kalkman
C,
Gielen
M,
ten Cate
O.
When do supervising physicians decide to entrust residents with unsupervised tasks?
Acad Med
.
2010
;
85
(9)
:
1408
-
1417
.
30. 
Rekman
J,
Gofton
W,
Dudek
N,
Gofton
T,
Hamstra
SJ.
Entrustability scales: outlining their usefulness for competency-based clinical assessment
.
Acad Med
.
2016
;
91
(2)
:
186
-
190
.
31. 
Durning
SJ,
Cation
LJ,
Markert
RJ,
Pangaro
LN.
Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training
.
Acad Med
.
2002
;
77
(9)
:
900
-
904
.

The online version of this article contains the survey used in the study and essential characteristics of responding and nonresponding internal medicine residency programs from the 2021 APDIM Annual Survey of Internal Medicine Residency Program Directors.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

This work was previously presented at the Association of Program Directors in Internal Medicine (APDIM) Fall Meeting, September 22-24, 2022, San Diego, California, USA.

Supplementary data