Abstract

Background

Graduating residents transition their continuity clinic patients to junior colleagues every year, creating a vulnerable transition period for about 1 million patients nationally.

Objective

We examined a standardized, electronic template for handing off high-risk ambulatory patients by outgoing residents from 7 residencies within a large health care system, and compared handoff quantity and provider satisfaction for handoffs with and without that template.

Methods

Residents graduating in 2011 from 5 internal medicine, 1 family medicine, and 1 internal medicine-pediatrics residency programs in 1 health care system were randomized to a new electronic handoff process with a standardized intervention template or a free-text handoff. Expert reviewers independently evaluated all handoff notes, and providers were surveyed after follow-up appointments regarding use, helpfulness, and overall satisfaction with the handoffs.

Results

Fifty-two of 79 residents (66%) participated, performing 278 handoffs. Eighty-four patients (30%, 17 of 57) failed to follow up within the study period. For patients who followed up, providers read 61% (101 of 165) of the handoffs at the time of the visit. No significant difference existed between groups in the satisfaction of the follow up provider or the quality measure of the handoffs in our process. Expert agreement on which features make the handoff “helpful” was fair (κ  =  0.34).

Conclusions

A standardized template did not improve handoff quantity or satisfaction compared with a free-text handoff. Practical handoff programs can be instituted into diverse residencies within a short time frame, with most residents taking part in creating the handoff formats.

What was known

A system for end-of-training handoffs may improve continuity of care for 1 million patients annually.

What is new

A standardized, electronic template for handing off high-risk ambulatory patients transferred by graduating residents to other providers did not improve provider satisfaction.

Limitations

Nonequivalence of groups despite randomization may limit generalizability. Low expert interrater agreement and agreement between respondents and expert raters on what makes a handoff useful were additional limitations.

Bottom line

The protocol for end-of-residency handoffs is practical and widely used. Practical handoff programs may not require a specialized format and can be readily instituted.

Editor's Note: The online version of this article includes handoff instructions to the residents.

Introduction

The Joint Commission and the World Health Organization have targeted patient handoffs as an area for improving patient safety for all health care settings.1 Research on ambulatory handoff communication from graduating residents to subsequent providers has been emerging in multiple fields after their identification as a safety concern.2,3 One survey identified that pediatrics residents want written handoffs of complex patients and that the absence of that documentation creates excessive work.4 Internal medicine studies confirmed that high-risk patients are less likely to follow up once their resident graduates,5,6 that residents can identify high-risk patients to hand off without supervision,5 and that written and verbal handoffs improved completion rates of assigned clinical tasks.6 Prior identification and identifying high-risk ambulatory patients in psychiatry led to fewer hospitalizations.7 Finally, we found that an e-mail reminder and use of an electronic health record (EHR) template increased the number of handoffs performed by graduating residents.8 

The new Clinical Learning Environment Review program of the Accreditation Council for Graduate Medical Education (ACGME) requires residency curricula to formally include handover processes,9 but many programs do not have formal systems in place to hand off clinic patients at the end of residency. In a survey of combined internal medicine-pediatrics program directors, we found that although 46% of programs have formal outpatient handoff systems10 and even fewer of their attached categorical programs did, most program directors feel that those systems are important for patient care.11 

Based on the low percentage of programs with handoff systems, we pursued a practical way to facilitate and improve outpatient handoffs. Using a standardized handoff tool within an EHR, we undertook a study across multiple residencies with continuity clinics in a single health care system. We included programs with and without existing end-of-residency handoff structures, hypothesizing that using a standardized EHR template would increase the number of handoffs performed and improve handoff quality as defined by the satisfaction of the next provider, relative to an unstructured handoff. We adapted our prototype EHR handoff template8 and designed this study to document the number of handoffs performed, their quality as characterized by follow-up provider satisfaction, and the timeliness of patient follow-up. Residents were randomized to either the intervention (instructions for the standardized template) or control (instructions for free-text) conditions.

Methods

We recruited residents graduating in 2011 from 7 of the 8 internal medicine, family medicine, and internal medicine-pediatrics residency programs in the MedStar Health system, a regional health care organization in the Baltimore–Washington DC area. All continuity clinics use 1 EHR (Centricity, General Electric, Fairfield, CT).

The Georgetown University and MedStar Health Institutional Review boards approved the study, and all participating providers gave informed consent.

Of 79 eligible graduating residents in the 7 programs, 52 (66%) volunteered to participate in “a study about handoffs.” The e-mail originated from 1 of the authors (M.J.D.), who was known only to residents in 2 programs. Residents were randomized, irrespective of program, into control and intervention groups using an Internet-based randomizer (http://www.randomization.com) to generate a randomly permuted 8-by-10 block design.

All study participants received an e-mail reviewing the professional importance of handoffs, requesting handoffs for patients “requiring some continuity of care,” stating expectations for writing several handoffs, instructing participants how to place the note in the EHR, and lastly, asking them not to disclose the e-mail's contents with anyone else (instructions provided as online supplemental material). E-mail instructions sent to the intervention group included extra instructions on how to open an EHR template to facilitate and standardize their handoffs. We sent an identical reminder e-mail 1 week later to all residents in the study. Subjects had at least 21 days to complete their handoff notes.

Three residency programs had formal handoff systems in place: 2 used EHRs, and 1 used oral and e-mail handoffs. Study instructions only addressed the EHR component to avoid affecting existing handoff systems. Our protocol did not provide protected time for participants to complete handoffs.

Two independent reviewers (M.J.D. and J.M.C.) identified all the EHR handoff data for study participants from 2 weeks before the date of randomization until 6 months after the end of residency. These reviewers searched the EHR monthly to determine when each patient followed up, recording the provider and visit date. We then surveyed follow-up providers (who were blind to the intervention) about each first follow-up visit with a 6-item survey that we developed. These surveys were completed as soon as the same day up to as long as 3 months later, and they assessed the following: whether the provider read the handoff during or before the visit, handoff features that were helpful, the provider's characterization of patient complexity, whether the patient followed up when the graduating resident asked them to, and survey respondents' satisfaction with the handoff. Satisfaction was represented on a 7-point scale ranging from 0 (not at all satisfied/dissatisfied) to 7 (extremely satisfied). We analyzed this variable as both an ordinal variable and dichotomized ratings of 0 to 4 as “less satisfied” and 5 to 7 as “satisfied.”

Finally, 2 coauthors (M.J.D. and J.M.C.) evaluated almost every handoff for patients having a follow-up visit during the study period. Those evaluations were independent, using a structured evaluation form based on our earlier handoff study.8 No validated evaluation tool exists, to our knowledge, to assess outpatient handoffs. Our prior work identified 2 features that any useful handoff must have: value for subsequent care (was this helpful or not) and an identifiable reason (why or why not). Reviewers decided whether each handoff was useful, assigned a reason why or why not from a preidentified list, and then recorded whether the handoff used the standardized template. A κ score was calculated to evaluate interrater reliability of the 2 reviewers.

The data were analyzed using SPSS Version 20.0 (IBM Inc, Chicago, IL).

Results

During the study, 52 residents performed 278 end-of-training handoffs (range, 0–25 handoffs each). The number of handoffs performed by either group did not differ between the intervention and the control group, but residents in the intervention group were more likely to write at least 1 handoff (χ2  =  9.67, P  =  .002). Eighty-four patients (30%, 17 of 57) did not have an appointment in the 6-month follow-up period.

Eighty-nine different follow-up providers completed 165 surveys (range, 1–6; mean, 1.87) after seeing a patient for follow-up. Interns completed 48 surveys (29%), residents completed 109 (66%), and attendings completed 8 (4.8%). Most handoffs (101 of 165, 61%) were read during or before the patient visit. We analyzed satisfaction ratings only for that subset, excluding 3 surveys because physicians rated handoffs although they reported they had not read them.

For the 101 surveys completed by follow-up providers who read either type of handoff, there was no difference in satisfaction ratings between the intervention and the control group (χ2  =  0.122; P  =  .06; table 1).

TABLE 1

Handoff Survey Responses

Handoff Survey Responses
Handoff Survey Responses

Two experts (M.J.D. and J.M.C.) each rated 164 handoffs for helpfulness. Interrater concordance (68.3%) was modest (κ  =  0.34). Although reviewer 1 found no difference between intervention and control handoffs (χ2  =  0.02, P  =  .88), reviewer 2 detected a significant difference between the groups (χ2  =  4.364, P  =  .042). The most common reason for discord between the reviewers (45 of 52 discordant handoffs [87%]) in their assessment of the helpfulness of the intervention was that 1 reviewer judged that information in the handoff could not easily be found in the EHR, whereas the other reviewer concluded that all information could easily be found.

Because of disagreement between the reviewers, few handoffs (3 of 101 [2.9%], all in the intervention group) were deemed by respondents and expert reviewers to be unhelpful. Reasons for a handoff being deemed unhelpful by providers included 2 respondents in the control group who reported the handoff information was unnecessary because the patient returned for an urgent care visit, and 2 respondents in the intervention group stated they needed more detailed information regarding active issues.

We examined whether satisfaction with the handoff was related to patient complexity because of the finding that significantly more “complex” (table 1) patients were in the intervention group than were in the control group. When stratified by patient complexity (table 2), survey respondents showed no preference between control or intervention handoffs at the levels of patient complexity (χ2  =  3.14, P  =  .08). Further, there was no relationship between expert reviewer handoff helpfulness and level of patient complexity. We also found the number of handoffs performed by each resident was not associated with the quality of the handoff in satisfaction or helpfulness ratings.

TABLE 2

Handoff Ratings Stratified by Patient Complexity

Handoff Ratings Stratified by Patient Complexity
Handoff Ratings Stratified by Patient Complexity

Discussion

This is the first study, to our knowledge, to examine the end-of-residency handoff process in multiple residencies within a health care system. We implemented a standardized and feasible transfer-of-care program into diverse residencies within 1 to 2 months. Our simple approach is consistent with the ACGME Clinical Learning Environment Review program, by formally introducing residents to the handoff process and by placing a correctly labeled handoff function into the EHR. However, our intervention failed to show that a standardized handoff versus a free-text handoff changed the quantity performed, the follow-up provider satisfaction, or our handoff quality measure. We also found that 2 expert reviewers disagreed with both the survey respondents and each other on which handoffs were helpful.

Although the number of handoffs performed per resident parallels those of other studies, our handoff system differed from others in several respects. Face-to-face handoffs have been employed by several programs,57 but our study suggests that because many handoffs (61%) were read before the first follow-up visit, the EHR effectively provided the information. We designed the EHR function to improve the relevant patient information the follow-up provider received—possibly prompting them to follow up more actively. Our findings corroborate previously reported no-show rates among high-risk patients3,6,12; those no-show rates of nearly 40% are likely a significant safety threat for those patients.

In contrast to focusing on a to-do list,6 we left handoff content to the discretion of the resident by providing a general outline of important topics possibly needing communication. We were concerned that interns responding to our survey (29%) would not be able to assess handoffs and would rate any handoff highly. Ultimately, having all pertinent data in 1 document may be more important than the format (template versus open-text), especially for an intern who may not be familiar with the EHR. Our low rate of handoffs (2.9%) deemed by respondents and both experts to be unhelpful may be due to recipients perceiving that any handoff is better than none. However, handoffs that miss important information could lead to a false sense of security. Because residents are trained to be independent professional practitioners, it is important to raise their level of awareness of, and skill in providing, helpful handoff notes.

One program had extremely close attending management, consistent with our findings that 52% of program directors without handoff systems in their residencies believed that those handoffs are not helpful because of attending comanagement.11 That model likely improves patient safety but may not confer appropriate responsibility to the resident.

Our study has several limitations. Two attributes were unbalanced across groups: Providers read more control handoffs before the visit, and patients in the intervention group were more complex than those in the control group. This was likely due to chance. There was low expert interrater agreement or agreement between respondents and expert raters on what makes the information in a given handoff useful.

The only fair interrater agreement by our expert raters related to perceptions of the ease with which handoff information could be located in the EHR. Much of that information related to psychosocial matters (family or social issues), which may have a larger long-term impact on compliance and health. That suggests that future evaluations of handoffs should assess immediate or long-term utility for the patient. Finally, if experts cannot agree on what makes a handoff useful, residency directors will have difficulty teaching what to include.

A surprising finding was that handoffs using a standardized format were considered no more satisfactory than those using unstructured free text. However, our prior research suggested that straightforward scoring systems sometimes fail to capture the true worth of a handoff8; perhaps, an improved template or blended free text and template format would be perceived as more effective.

Conclusion

Our assessment of a standardized handoff template did not produce uniform perceptions of helpfulness or satisfaction compared with a free-text handoff form, although our protocol for end-of-residency handoffs is practical and widely used. Patient complexity was associated with the providers' perceived level of satisfaction, regardless of the format of the EHR note.

References

References
1
Joint Commission International
.
WHO Collaborating Centre for Patient Safety Solutions
. .
2
Young
JQ
,
Wachter
RM
.
Academic year-end transfers of outpatients from outgoing to incoming residents: an unaddressed patient safety issue
.
JAMA
.
2009
;
302
(
12
):
1327
1329
.
3
Young
JQ
,
Eisendrath
SJ
.
Enhancing patient safety and resident education during the academic year-end transfer of outpatients: lessons from the suicide of a psychiatric patient
.
Acad Psychiatry
.
2011
;
35
(
1
):
54
57
.
4
Lerner
CF
.,
Hamilton
LJ
,
Klitzner
TS
.
Improving year-end transfers of care in academic ambulatory clinics: a survey of pediatric resident physician perceptions [published online ahead of print May 22, 2012]
.
Med Educ Online
.
doi:10.3402/meo.v17i0.17815
.
5
Pincavage
AT
.,
Ratner
S
,
Prochaska
ML
,
Prochaska
M
,
Oyler
J
,
Davis
AM
,
et al.
Outcomes for resident-identified high-risk patients and resident perspectives of year-end continuity clinic handoffs
.
J Gen Intern Med
.
2012
;
27
(
11
):
1438
1444
.
6
Garment
AR
.,
Lee
WW
,
Harris
C
,
Phillips-Caesar
E
.
Development of a structured year-end sign-out program in an outpatient continuity practice [published online ahead of print September 19, 2012]
.
J Gen Intern Med
.
2013
;
28
(
1
):
114
120
.
doi:10.1007/s11606-012-2206-2
.
7
Young
JQ
.,
Pringle
Z
,
Wachter
RM
.
Improving follow-up of high-risk psychiatry outpatients at resident year-end transfer
.
Jt Comm J Qual Patient Saf
.
2011
;
37
(
7
):
300
308
.
8
Donnelly
MJ
.,
Clauser
JM
,
Weissman
NJ
.
An intervention to improve ambulatory care handoffs at the end of residency
.
J Grad Med Educ
.
2012
;
4
(
3
):
381
384
.
9
Weiss
KB
.,
Wagner
R
,
Nasca
TJ
.
Development, testing, and implementation of the ACGME Clinical Learning Environment Review (CLER) program
.
J Grad Med Educ
.
2012
;
4
(
3
):
396
398
.
10
Donnelly
MJ
.,
Clauser
JM
,
Tractenberg
RE
.
Current practice in end-of-residency handoffs: a survey of internal medicine-pediatrics program directors
.
J Grad Med Educ
.
2013
;
5
(
1
):
93
97
.
11
Donnelly
MJ
.,
Clauser
JM
,
Tractenberg
RE
.
Systematic training in internal medicine-pediatrics end-of-residency clinic handoffs: residency director attitudes and perceived barriers
.
Teach Learn Med
.
2014
;
26
(
1
):
17
26
.
12
Caines
LC
.,
Brockmeyer
DM
,
Tess
AV
,
Kim
H
,
Kriegel
G
,
Bates
CK
.
The revolving door of resident continuity practice: identifying gaps in transitions of care
.
J Gen Intern Med
.
2011
;
26
(
9
):
995
998
.

Author notes

Michael J. Donnelly, MD, is Associate Professor, Department of Medicine and Pediatrics, Georgetown University Hospital; at the time of the study, Janelle M. Clauser, MD, was Assistant Professor, Department of Medicine and Pediatrics, Georgetown University Hospital, and is now Attending Physician, Department of Internal Medicine, Providence Sacred Heart Medical Center; and Rochelle E. Tractenberg, PhD, MPH, PStat, is Associate Professor, Departments of Neurology, Psychiatry, and Biostatistics, Biomathematics, and Bioinformatics, Georgetown University Medical Center.

Funding: The authors report no external funding source for this study.

The authors would like to thank the MedStar Health Ambulatory Resident Sign-out Investigators: David Weisman, DO, and Rosemary Maraj, MD, Department of Internal Medicine, Good Samaritan Hospital; Deborah Topol, MD, Department of Medicine, Washington Hospital Center; Netra Thakur, MD, Department of Family Medicine, Georgetown University Hospital. Stephanie Detterline, MD, and Radhika Vij, MD, Department of Internal Medicine, Union Memorial Hospital; Sheenu Sheela, MBBS, Department of Internal Medicine, Harbor Hospital; and Stephanie Davis, BA, and Mai Al-Eskolar, BA, Georgetown University.

Supplementary data