Background 

Following up on patients' clinical courses after hospital discharge may enhance physicians' learning and care of future patients. Barriers to this practice for residents include time constraints, discontinuous training environments, and difficulty accessing patient information.

Objective 

We designed an educational intervention facilitating informed self-assessment and reflection through structured postdischarge follow-up of patients' longitudinal clinical courses. We then examined the experience of interns who received this intervention in a mixed methods study.

Methods 

Internal medicine interns on a 4-week patient safety rotation received lists of hospitalized patients they had cared for earlier in the year. They selected patients for chart review and completed a guided reflection worksheet for each patient reviewed. Interns then discussed lessons learned in a faculty-led group debrief session.

Results 

Of 62 eligible interns, 62 (100%) participated in this intervention and completed 293 reflection worksheets. We analyzed worksheets and transcripts from 6 debrief sessions. Interns reported that postdischarge patient follow-up was valuable for their professional development, and helped them understand the natural history of disease and patients' illness experiences. After reviewing their patients' clinical courses, interns stated that they would advocate for earlier end-of-life counseling, improve care transitions, and adjust their clinical decision-making for similar patients in the future.

Conclusions 

Our educational intervention created the time, space, and structure for postdischarge patient follow-up. It was well received by participants, and is an opportunity for experiential learning.

What was known and gap

Following patients' clinical courses after hospital discharge may facilitate learning and practice changes in residents.

What is new

An intervention created space and structure for residents to follow patients postdischarge, by identifying records of prior patients and prompting residents to review them.

Limitations

Single institution study reduces generalizability; outcomes limited to self-reported learning gains.

Bottom line

The intervention was well received, with residents reporting learning related to end-of-life counseling, care transitions, and clinical decision-making.

Reflecting on one's own clinical practice is a crucial step in experiential learning.1  Physicians favor learning from the patient care problems they face at work above all other sources of knowledge.2,3  However, to learn from their own practice, physicians cannot rely on unstructured contemplation about their patients. They must compare their work against an external data source (eg, patient outcomes).4  This informed self-assessment approach may help identify strengths and weaknesses in knowledge and skills, and improve physicians' future performance for similar patients and situations.5 

The Accreditation Council for Graduate Medical Education competency of practice-based learning and improvement asks residents to investigate, evaluate, and improve their patient care practices.6,7  Creating the time and space for residents to systematically analyze their personal practice patterns is challenging for training programs. Residents who wish to participate in this type of informed self-assessment face multiple barriers, including time limitations, discontinuities of care, and lack of electronic health record (EHR) interoperability.8  While some studies highlight feedback to residents on specific aspects of their care, including procedure proficiency,9  quality metrics,1012  and supervisor corrections to their work,13  few curricular interventions have addressed the question that is most fundamental to a resident's professional judgment: How did my patient ultimately fare?

To address this gap, we developed an educational intervention that incorporated structured reflection to facilitate internal medicine interns' ability to follow up on patients they cared for during earlier inpatient medicine rotations. We then examined the experience of interns who received this intervention in a mixed methods study.

Our intervention spanned portions of 2 academic years (2014–2015 [AY-1] and 2015–2016 [AY-2]). We used purposive sampling14  to identify all interns on a 4-week patient safety rotation who had previously completed at least 1 month of inpatient general medicine wards at our university hospital. These interns were eligible to participate in our intervention (N = 62: n = 21 in AY-1 and n = 41 in AY-2).

Patient List Development

Using Standard Query Language (a domain-specific language used in programming and designed for database management), we probed the EHR for any clinical notes (admission, progress, or discharge summary) written by internal medicine interns for patients discharged from the medicine service at our university hospital. Interns received personalized lists of patients for whom they had written at least 1 clinical note. Lists contained basic demographic and clinical data (ie, name, medical record number, age, sex, admission/discharge date, discharge diagnosis, and death status).

Chart Review Reflection Worksheet Development

We developed a worksheet to guide chart review and reflection (provided as online supplemental material). The worksheet asked interns (1) what medical questions or concerns they had at the time of patient discharge; (2) what lessons they learned from the review of the patient's postdischarge clinical course; (3) about the patient's readmissions and outpatient care needs; and (4) how they intended to change their future clinical practice as a result of this review. Prompts were based on behaviors and reasoning practices associated with reflective practice in medicine.15  Worksheets were stored in a secure online research platform, and patient names and medical record numbers were not recorded.16 

Program Structure

Each month, 3 to 6 interns rotated on a 4-week patient safety rotation. These interns were provided their patient list, instructed to review the EHRs of as many of their patients as they chose, and asked to complete a reflection worksheet for each patient reviewed (figure). We suggested they devote 3 to 4 hours on this exercise during the rotation. During the last week of the rotation interns participated in a 1-hour, faculty-facilitated group debrief session. Two investigators with experience running focus groups led the sessions, with 1 clinician (either S.N. or A.R.) and 1 qualitative research scientist (J.D.H.). To ensure consistency, debriefs were guided by a series of questions to evaluate the lessons interns learned from postdischarge chart review, expected changes to or reinforcement of clinical practice, perceived value of the exercise, and barriers to sustainability of this practice. When necessary, we asked interns clarifying questions. We obtained written consent from all interns who participated.

Figure

Postdischarge Follow-Up Intervention Overview and Structure

Figure

Postdischarge Follow-Up Intervention Overview and Structure

Close modal

The University of California, San Francisco, Institutional Review Board approved this study.

Program Evaluation

Data Collection

Participating interns in AY-2 (n = 41) were asked to complete surveys to assess their baseline follow-up habits prior to this intervention. Debrief sessions in AY-2 were audio-recorded and professionally transcribed (n = 6).

Data Analysis

Responses to surveys were summarized using descriptive statistics. We analyzed the reflection worksheets (n = 293) and the transcripts from the debrief sessions (n = 6) using content analysis17  to describe lessons learned, clinical management changes, and educational value and sustainability of the intervention. Two authors (S.N. and J.D.H.) with experience in coding independently performed open coding using a data-driven (inductive) approach,17,18  which allowed us to identify new and emerging patterns from the data. Authors met throughout the analysis process to determine coding categories. Coding disparities were discussed and resolved by negotiated consensus.19  Coding categories were then grouped into 2 higher-order categories (themes): patient-related factors and health system factors. We coded all reflection worksheets and quantified how many times each code appeared. All AY-2 debrief sessions were analyzed, and data saturation was reached such that no new coding categories emerged.20  We used Dedoose version 7.0.23 (SocioCultural Research Consultants LLC, Los Angeles, California) to conduct the qualitative analysis.

Of 62 eligible interns in AY-1 and AY-2, 62 (100%) received patient lists and completed 293 reflection worksheets. Of the 41 interns from AY-2, 24 interns (59%) participated in recorded debrief sessions. The remaining 17 interns from AY-2 (41%) could not participate due to scheduling conflicts. On average, each intern's list contained 45 patients (range, 11–88 patients, SD = 18). Each intern completed an average of 4.7 reflection worksheets, for a total of 293 worksheets across the 62 interns. We did not measure the amount of time devoted to each review. Faculty spent approximately 2.5 hours monthly extracting the lists, securely delivering the lists to the individual interns, and running the debrief session.

Intern Baseline Follow-Up Habits

Of 41 interns, 39 (95%) from AY-2 completed baseline surveys. Most interns (79%, 31 of 39) reported following up on less than 40% of patients they had cared for in the hospital. They reported time constraints, a multi-hospital training program, and difficulty tracking patients through the EHR as major barriers to postdischarge follow-up (Table 1).

Table 1

Interns' Preintervention Responses to Questions About Barriers to and Value of Postdischarge Patient Follow-Upa

Interns' Preintervention Responses to Questions About Barriers to and Value of Postdischarge Patient Follow-Upa
Interns' Preintervention Responses to Questions About Barriers to and Value of Postdischarge Patient Follow-Upa
Table 2

Content Analysis of Reflection Worksheet Questions and Debrief Session Prompts

2a: “What Lessons Did You Learn From This Patient's Postdischarge Clinical Course?”

Content Analysis of Reflection Worksheet Questions and Debrief Session Prompts
Content Analysis of Reflection Worksheet Questions and Debrief Session Prompts

Lessons Learned Through Postdischarge Follow-Up and Reflection

In reflection worksheets, interns reported being curious at the time of discharge about patients' clinical outcomes, their postdischarge management, pending test results, final diagnoses, and patients' adherence to treatment plans. They revealed in debrief sessions that they mostly reviewed those patients for whom they had a clinical question or concern, or with whom they had a memorable relationship.

Content analysis of reflection worksheets and debrief session transcripts revealed that interns learned multiple lessons about clinical management and health systems (Table 2a). They also identified areas where they would change their future practice (Table 2b).

Interns reflected on ways their current clinical practice was reinforced. In 6% (19 of 293) of reflection worksheets, they highlighted cases where their intended discharge plans for patients succeeded. In 19% (57 of 293) of worksheets, interns noted instances when they would not change their inpatient clinical management after reviewing patients' postdischarge clinical courses.

Interns characterized the effect of this exercise along 4 themes:

  • 1. 

    Review catalyzes change in clinical practice. Interns identified ways to change their clinical practice for similar patients in the future.

“I had never treated hypernatremia before: Did I pick the right rate of fluids? Did it work? Did [the sodium] drop too quickly? Those are the types of things that I want to know—whether my clinical judgment was correct, and what happened to the patient afterward.”

“Even if you find out that your diagnosis was incorrect, it improves your confidence because [then you'll think], ‘Oh next time, I'll make sure to think about this, and this.'”

  • 2. 

    Review fosters the development of clinical instincts. Longitudinal patient follow-up reinforced management choices and fortified existing clinical intuition.

“It helps develop the ‘clinical gut'… it's not stuff you can get by reading, because when you read on “Up to Date” that a prognosis is 6 weeks to 6 months it means very little. In my brain I put them into several categories that don't really solidify until I've seen several patients fall into that trend; at least for me I remember it a lot better [that way].”

“It's helpful to reinforce the illness script, from prognosis to whether you managed the patient correctly.”

“Did anyone, during this exercise, have any moments of vindication like, ‘I totally called it'? I had 2 of those and I was like, ‘Yes!'”

  • 3. 

    Review develops a more comprehensive picture of a patient's life and illness experience. By evaluating a longitudinal clinical course, interns were prompted to think about how their inpatient interaction was a small part of their patient's illness journey.

“It lengthens the amount time that you're ‘caring for someone' or ‘thinking about someone,' so you get a more complete and comprehensive picture of them.”

“This allows you to contextualize hospitalizations—it is part of a bigger picture of a patient's life and the patient's interaction with the health care system.”

  • 4. 

    Review demonstrates health system vulnerabilities. Interns specifically noted how they might contribute to and improve care transition challenges.

“It was helpful to reflect upon my discharge summaries and [see] how confusing they were even for me to read going back.”

“It showed me [that] even if you do everything perfectly with your discharge planning, there are always errors that [can] happen. I need to be even more vigilant, and recognize that our system makes it really difficult for things to go seamlessly.”

Sustainability of Structured Postdischarge Follow-Up and Reflection

Interns were supportive of this program and valued learning about their patients' clinical courses.

“With short hospitalizations I think there is a loss of learning opportunities, and this [exercise] closes the gap.”

“We don't have a lot of a feedback on how things have gone … it would be something that would be nice to shift in the culture.”

Some interns stated that receiving patient lists and reflective prompts would be enough motivation to independently and regularly review patient charts. However, others felt that a structure to provide the time and guidance to complete this exercise was imperative to nurture a habit of regular patient follow-up.

“There has to be dedicated time for [this exercise] if it's going to be important.. . . If you just make it part of, ‘Hey, are you a self-motivated physician? You should be doing this on your own time instead of watching your favorite TV show,' it's not going to happen.”

Some interns noted that their ability to reflect through this intervention was limited by the lack of follow-up data if a patient was discharged to a nursing facility or to a health care system that did not use our institution's EHR. While few interns did telephone patients or contact an outside hospital, this was not a required part of the intervention.

Interns valued structured reflection on patients' postdischarge clinical courses, which may facilitate informed self-assessment by capitalizing on physicians' most revered source of learning: their patients.

Two published reports (1 in emergency medicine21  and 1 in neurology22) highlighted residents who rigorously tracked and reflected on patients they evaluated. A recent study showed that the practice of following patient outcomes through the EHR is common among medical students.23  The goal of our intervention was to develop lifelong learners and professionals who are “proactive in seeking feedback because they want to improve, rather than being reactive in responding to feedback because others would like them to.”24 

This program is distinct from published postdischarge telephone25,26  and resident clinic27  follow-up programs, because the time since discharge (often weeks to months) provides more opportunity to gain perspective on the course of illness and to reach definitive conclusions. Our intervention is also unique in that interns drew their own insights from chart review without faculty review and correction, as is done in chart-stimulated recall.28  Our evaluation shows that those insights are sophisticated, and they span a wide variety of medical practice components, from disease-specific knowledge to goals of care and care transitions.

A training program may be able to adopt this intervention if it has the capacity to generate patient lists from the EHR, has space in the curriculum to allow interns to spend several hours reviewing their patients' records, and has faculty who can facilitate postdischarge feedback and reflection.

There are ways the intervention can be modified. We intentionally did not specify types or numbers of patients for review. Instructing interns to follow up on patients with specific outcomes (eg, patients who died or who had a change in diagnosis after discharge) might promote more deliberate reflection on specific areas of their practice. Similarly, combining this intervention with quality metrics (eg, readmission rates) could further direct their self-assessment. We purposefully instructed interns to review the EHR because of the low barrier to access this sufficient, albeit incomplete, data source. Interns could be given additional time to seek other sources of information (eg, outside hospital records, phone conversations with patients or physicians, etc) to create a more complete picture of a patient's postdischarge course. To increase the number of patients formally reviewed, interns could be given additional time to review patient charts, be instructed to spend less time reviewing each chart, or be offered a briefer reflection worksheet.

Finally, accessing EHRs after patient encounters solely for learning purposes has raised legal and ethical concerns,29  which should be discussed with interns in advance of the exercise.

This intervention has several limitations. It was conducted at a single institution, and we did not measure improvement in clinical performance or patient outcomes. It also is unclear whether this intervention will catalyze a long-term change in practice-based learning.

We are planning a multi-year longitudinal curriculum that extends to other rotations, such as night float, to determine whether the habit of postencounter patient follow-up can be instilled during residency.

We demonstrated that an educational intervention that creates the time, space, and structure for reflection on postdischarge patient follow-up is well received and may be an opportunity for experiential learning.

1
Rudolph
JW,
Simon
R,
Rivard
P,
et al.
Debriefing with good judgment: combining rigorous feedback with genuine inquiry
.
Anesthesiol Clin
.
2007
;
25
(
2
):
361
376
.
2
Campbell
C,
Parboosingh
J,
Gondocz
T,
et al.
Study of the factors influencing the stimulus to learning recorded by physicians keeping a learning portfolio
.
J Cont Educ Health Prof
.
1999
;
19
(
1
):
16
24
.
3
Kassirer
JP.
Teaching clinical reasoning: case-based and coached
.
Acad Med
.
2010
;
85
(
7
):
1118
1124
.
4
Sargeant
J,
Armson
H,
Chesulk
B,
et al.
The process and dimensions of informed self-assessment: a conceptual model
.
Acad Med
.
2010
;
85
(
7
):
1212
1220
.
5
Croskerry
P.
The feedback sanction
.
Acad Emerg Med
.
2000
;
7
(
11
):
1232
1238
.
6
Burke
AE,
Benson
B,
Englander
R,
et al.
Domain of competence: practice-based learning and improvement
.
Acad Pediatr
.
2014
;
14
(
suppl 2
):
38
54
.
7
Accreditation Council for Graduate Medical Education
.
Competency definitions and recommended practice performance tools
. ,
2017
.
8
Schiff
GD.
Minimizing diagnostic error: the importance of follow-up and feedback
.
Am J Med
.
2008
;
121
(
suppl 5
):
38
42
.
9
Barsuk
JH,
Cohen
ER,
Vozenilek
JA,
et al.
Simulation-based education with mastery learning improves paracentesis skills
.
J Grad Med Educ
.
2012
;
4
(
1
):
23
27
.
10
Phillips
LS,
Ziemer
DC,
Doyle
JP,
et al.
An endocrinologist-supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD) 7
.
Diabetes Care
.
2005
;
28
(
10
):
2352
2360
.
11
Ehrenfeld
JM,
McEvoy
MD,
Furman
WR,
et al.
Automated near-real-time clinical performance feedback for anesthesiology residents: one piece of the milestones puzzle
.
Anesthesiology
.
2014
;
120
(
1
):
172
184
.
12
Hildebrand
C,
Trowbridge
E,
Roach
MA,
et al.
Resident self-assessment and self-reflection: University of Wisconsin-Madison's five year study
.
J Gen Intern Med
.
2009
;
24
(
3
):
361
365
.
13
Sharpe
RE,
Surrey
D,
Gorniak
RJT,
et al.
Radiology report comparator: a novel method to augment resident education
.
J Digit Imaging
.
2012
;
25
:
330
336
.
14
Berg
BL.
Qualitative Research Methods for the Social Sciences. 6th ed
.
Boston, MA
:
Pearson Education Inc;
2007
.
15
Mamede
S,
Schmidt
HG,
Rikers
R.
Diagnostic errors and reflective practice in medicine
.
J Eval Clin Pract
.
2007
;
13
(
1
):
138
145
.
16
Harris
PA,
Taylor
R,
Thielke
S,
et al.
Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support
.
J Biomed Inform
.
2009
;
42
(
2
):
377
381
.
17
Schreier
M.
Qualitative Content Analysis in Practice
.
Los Angeles, CA
:
SAGE Publications Ltd;
2012
.
18
Elo
S,
Kyngas
H.
The qualitative content analysis
.
J Adv Nurs
.
2008
;
62
(
1
):
107
115
.
19
Bradley
EH,
Curry
LA,
Devers
KJ.
Qualitative data analysis for health services research: developing taxonomy, themes and theory
.
Health Serv Res
.
2007
;
42
(
4
):
1758
1772
.
20
Morse
JM,
Barrett
M,
Mayan
M,
et al.
Verification strategies for establishing reliability and validity in qualitative research
.
Int J Qual Methods
.
2002
;
1
(
2
):
13
22
.
21
Smith
KA.
To keep an incessant watch
.
Acad Emerg Med
.
2011
;
18
(
5
):
545
548
.
22
Dhand
A.
Right brain: the case library as a tool to enhance clinical observation
.
Neurology
.
2012
;
78
(
7
):
512
513
.
23
Brisson
GE,
Tyler
PD.
Medical student use of electronic health records to track former patients
.
JAMA Intern Med
.
2016
;
176
(
9
):
1395
1397
.
24
Hayden
SR,
Dufel
S,
Shih
R.
Definitions and competencies for practice-based learning and improvement
.
Acad Emerg Med
.
2002
;
9
(
11
):
1242
1248
.
25
Record
JD,
Niranjan-Azadi
A,
Christmas
C,
et al.
Telephone calls to patients after discharge from the hospital: an important part of transitions of care
.
Med Educ Online
.
2015
;
20
:
26701
.
26
Saba
GW,
Chou
CL,
Satterfield
J,
et al.
Teaching patient-centered communication skills: a telephone follow-up curriculum for medical students
.
Med Educ Online
.
2014
;
19
:
22522
.
27
Booth
KA,
Vinci
LM,
Oyler
JL,
et al.
Using a resident discharge clinic for resident education and patient care: a feasibility study
.
J Grad Med Educ
.
2014
;
6
(
3
):
536
540
.
28
Schipper
S,
Ross
S.
Structured teaching and assessment: a new chart-stimulated recall worksheet for family medicine residents
.
Can Fam Physician
.
2010
;
56
(
9
):
958
959
,
e352–e354
.
29
Brisson
GE,
Neely
KJ,
Tyler
PD,
et al.
Should medical students track former patients in the electronic health record? An emerging ethical conflict
.
Acad Med
.
2015
;
90
(
8
):
1020
1024
.

Author notes

Editor's Note: The online version of this article contains the chart review reflection worksheet and the debrief session outline.

Funding: This program was supported by an educational grant from the UCSF Office of Medical Education and the UCSF Academy of Medical Educators.

Competing Interests

Conflict of interest: Dr Dhaliwal has received speaking honoraria from the Studer Group.

This study was presented as a poster at the Society of General Internal Medicine Annual Meeting, Hollywood, Florida, May 11–14, 2016.

The authors would like to thank the UCSF internal medicine residents for their participation in this study. The authors would also like to thank Drs Judith Bowen, Bridget O'Brien, and Patricia O'Sullivan for their feedback on the manuscript.

Supplementary data