There is growing awareness of a high prevalence of work-related stress and burnout in physicians,1,2  the toll it takes on physicians,3  its cost to institutions,4,5  and its consequences for patient care.6,7  Recognizing the need to make well-being a principal component of its activities, the Accreditation Council for Graduate Medical Education (ACGME) established new standards,8,9  and education and outreach activities to promote well-being in accredited programs and institutions.10,11  The multiple dimensions of the well-being standards and diverse needs of specialties make it important for programs and institutions to make reasoned choices about well-being interventions, well-being programs (sets of interventions targeted to groups such as residents or faculty), and methods for program evaluation.

To assist in these decisions, we introduce (1) a Well-Being Interventions Typology (WIT) of various approaches, and (2) a Well-Being Evaluation Schema (WES) to facilitate robust evaluation of well-being programs.

While there is no widely accepted definition of physician well-being, the construct is broader than ameliorating burnout and encompasses physical, emotional, and professional components.12,13  The ACGME well-being standards use this broader conceptualization.8,9  We aggregated components of the ACGME standards8  and published interventions to reduce burnout,14,15  enhance well-being,16  promote professional satisfaction,17  expand mentoring18  and peer support,19  and increase physician engagement20  into a single Table, the Well-being Interventions Typology (WIT). Like prior publications,14,15,20  we characterized interventions as targeting individuals (residents or faculty), work units, programs, or institutions.

Table

The Well-Being Interventions Typology (WIT)

The Well-Being Interventions Typology (WIT)
The Well-Being Interventions Typology (WIT)

To facilitate evaluation of well-being interventions, we adapted 2 existing frameworks with utility evidence—Kirkpatrick's assessment of training outcomes21  and the Best Evidence in Medical Education (BEME) scale22 —to create the Well-Being Evaluation Schema (WES) shown in the Figure. Both frameworks have been used extensively to evaluate education interventions. The current version of Kirkpatrick's model adds consideration of the relevance of interventions to the intended group, participants' engagement, required drivers to reinforce desired performance,21  leading indicators21  that offer early information about implementation success, and return on expectations to show how interventions meet key stakeholder expectations. Kaufman's expansion of Kirkpatrick's model promotes a focus on return on investment (ROI) and societal outcomes.23 

Figure

The Well-Being Evaluation Schema (WES)21,22 

Figure

The Well-Being Evaluation Schema (WES)21,22 

Close modal

To create the WES, we added 3 dimensions to the existing frameworks: (1) Level Zero (baseline or preintervention) data on distress, burnout, engagement, and other relevant metrics; (2) SMART24  goals to promote actionable and measurable interventions; and (3) an approach for evaluating non-education interventions (such as access to mental health services or reducing non-physician work in clinical settings), with Level 1 utilization data having a direct impact on outcomes of interest.

Evaluating outcomes is essential to understanding what interventions work in specific contexts, identifying best practices for adoption or adaptation, and negotiating for resources with institutional leaders. A key attribute of the WES framework is the inclusion of process and early outcome metrics as leading indicators21,25  of implementation effectiveness. Lack of participation in an intervention by targeted groups is an early leading indicator that offers critical information about attributes that may create barriers, such as the public location of a mental health clinic or the inability to attend medical appointments or scheduled wellness events due to work demands. Leading indicators can compensate for the fact that outcomes of interest often have significant time lags. In a case vignette (see the Box) and online supplemental material we highlight use of the WES to design and evaluate an institutional well-being program.

Box Using the WIT and WES to Evaluate and Improve an Institution's Well-Being Programing

Sharon and Richard, the designated institutional official (DIO) and chief well-being officer of a midsized sponsoring institution, discuss the current and planned well-being programs prior to an annual graduate medical education (GME) retreat. Areas for improvement include high burnout scores in residents and information suggesting local culture favors “toughing it out.” Richard and Sharon consulted the WIT and selected 2 new interventions intended to address the problems with local culture: enhancing resident access to medical and mental health services and having all residents develop a personal well-being plan.

Sharon suggests an existing intervention adding scribes and allowing for additional time for faculty debriefings in a family medicine continuity clinic should be expanded to 2 additional specialties to increase sample size and generalizability. She notes that studying interventions to address workflow problems and learning challenges in ambulatory settings is an opportunity for scholarship and could add to a limited body of literature.

At the GME retreat the WES is used to assess the impact of existing and planned programs. Attendees develop SMART (specific, measurable, actionable, relevant and time-bound) goalsa to track progress and projecting the return on expectationsb for the new interventions. This highlights that some existing well-being programs lack the resources needed for full implementation and sustainability. Attendees confirm use of the WES is helpful in selecting data to clarify expectations for implementation success.

Additional considerations by institutional leaders and a sample completed WES are provided as online information.

b

Kirkpatrick D, Kirkpatrick J, Kirkpatrick W. The New World Kirkpatrick Model. https://www.kirkpatrickpartners.com/Our-Philosophy/The-New-World-Kirkpatrick-Model.

Together, the WIT and the WES are expected to facilitate improved design and holistic evaluation of well-being programs. Important components are return on expectation (ROE),21  which estimates the organizational value of an intervention, and ROI,24  the rate of return (in improved well-being and associated benefits) for the funds invested in well-being programs to evaluate whether the investment is financially and conceptually sound. Analyses using ROE and ROI are important given the resources needed for well-being programming, including time and opportunity costs and prioritizing finite financial resources to well-being over other programs and activities.

We acknowledge limitations. While the overall approach is rooted in established assessment approaches and evidence-informed interventions, there is no primary validity evidence for these tools in selecting and evaluating well-being interventions. Further study is needed, and use of the WIT and WES will offer opportunities for scholarship, including identifying, describing, and disseminating information on effective sustainable interventions.

Burnout in physicians is high, and interventions are needed to promote well-being during their training and throughout their career. Given the broad implementation of well-being programs, it is critical to identify the interventions with positive returns on expectations and on investment. The WIT and the WES can assist in planning interventions with clarity around purpose and outcomes and in evaluating outcomes. This is critical for institutional decision-making and for scholarship to identify and disseminate best practices in this relatively young field.

1
Busireddy
KR,
Miller
JA,
Ellison
K,
Ren
V,
Qayyum
R,
Panda
M.
Efficacy of interventions to reduce resident physician burnout: a systematic review
.
J Grad Med Educ
.
2017
;
9
(
3
):
294
301
. doi:.
2
Dyrbye
LN,
West
CP,
Satele
D,
Boone
S,
Tan
L,
Sloan
J,
et al.
Burnout among US medical students, residents, and early career physicians relative to the general US population
.
Acad Med
.
2014
;
89
(
3
):
443
451
. doi:.
3
Shanafelt TD, Boone S, Tan L, Dyrbye L, Sotile W, Satele D
.
Burnout and satisfaction with work-life balance among US physicians relative to the general US population
.
Arch Intern Med
.
2012
;
172
(
18
):
1377
1385
. doi:.
4
Hamidi
MS,
Bohman
B,
Sandborg
C,
Smith-Coggins
R,
de Vries
P,
Albert
MS,
et al.
Estimating institutional physician turnover attributable to self-reported burnout and associated financial burden: a case study
.
BMC Health Serv Res
.
2018
;
18
(
1
):
851
. doi:.
5
Dewa
CS,
Loong
D,
Bonato
S,
Thanh
NX,
Jacobs
P.
How does burnout affect physician productivity? A systematic literature review
.
BMC Health Serv Res
.
2014
;
14
:
325
. doi:.
6
Scheepers
RA,
Boerebach
BC,
Arah
OA,
Heineman
MJ,
Lombarts
KM.
A systematic review of the impact of physicians' occupational well-being on the quality of patient care
.
Int J Behav Med
.
2015
;
22
(
6
):
683
698
. doi:.
7
Baer
TE,
Feraco
AM,
Tuysuzoglu Sagalowsky
S,
Williams
D,
Litman
HJ,
Vinci
RJ.
Pediatric resident burnout and attitudes toward patients
.
Pediatrics
.
2017
;
139
(
3
).
p
ii
: e20162163. doi:.
8
Accreditation Council for Graduate Medical Education
.
ACGME Common Program Requirements Section VI with Background and Intent
. ,
2019
.
9
Burchiel
KJ,
Zetterman
RK,
Ludmerer
KM,
Philibert
I,
Brigham
TP,
Malloy
K,
et al.
The 2017 ACGME Common Work Hour Standards: promoting physician learning and professional development in a safe, humane environment
.
J Grad Med Educ
.
2017
;
9
(
6
):
692
696
. doi:.
10
Accreditation Council for Graduate Medical Education
.
Clinical Learning Environment Review (CLER) National Report of Findings 2018, Executive Summary
. ,
2019
.
11
Bernstein
C,
Brady
D,
Brigham
T.
The culture of well-being
.
Accreditation Council for Graduate Medical Education
.
January 29, 2019
. ,
2019
.
12
Brady
KJS,
Trockel
MT,
Khan
CT,
Raj
KS,
Murphy
ML,
Bohman
B,
et al.
What do we mean by physician wellness? A systematic review of its definition and measurement
.
Acad Psychiatry
.
2018
;
42
(
1
):
94
108
. doi:.
13
Shanafelt
TD,
Bradley
KA,
Wipf
JE,
Back
AL.
Burnout and self-reported patient care in an internal medicine residency program
.
Ann Intern Med
.
2002
;
136
(
5
):
358
367
. doi:.
14
West
CP,
Dyrbye
LN,
Erwin
PJ,
Shanafelt
TD.
Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis
.
Lancet
.
2016
;
388
(
10057
):
2272
2281
. doi:.
15
Panagioti
M,
Panagopoulou
E,
Bower
P,
Lewith
G,
Kontopantelis
E,
Chew-Graham
C,
et al.
Controlled interventions to reduce burnout in physicians: a systematic review and meta-analysis
.
JAMA Intern Med
.
2017
;
177
(
2
):
195
205
. doi:.
16
Raj
KS.
Well-being in residency: a systematic review
.
J Grad Med Educ
.
2016
;
8
(
5
):
674
684
. doi:.
17
Friedberg
MW,
Chen
PG,
Van Busum
KR,
Aunon
F,
Pham
C,
Caloyeras
J,
et al.
Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy
.
Rand Health Q
.
2014
;
3
(
4
):
1
.
18
Wilson
PM,
Kemper
KJ,
Schubert
CJ,
Batra
M,
Staples
BB,
Serwint
JR,
et al.
National landscape of interventions to improve pediatric resident wellness and reduce burnout
.
Acad Pediatr
.
2017
;
17
(
8
):
801
804
. doi:.
19
Clough
BA,
March
S,
Chan
RJ,
Casey
LM,
Phillips
R,
Ireland
MJ.
Psychosocial interventions for managing occupational stress and burnout among medical doctors: a systematic review
.
Syst Rev
.
2017
;
6
(
1
):
144
. doi:.
20
Shanafelt
TD,
Noseworthy
JH.
Executive leadership and physician well-being: nine organizational strategies to promote engagement and reduce burnout
.
Mayo Clin Proc
.
2017
;
92
(
1
):
129
146
. doi:.
21
Kirkpatrick
D,
Kirkpatrick
J,
Kirkpatrick
W.
The New World Kirkpatrick Model
. ,
2019
.
22
Wolf
FM,
Shea
JA,
Albanese
MA.
Toward setting a research agenda for systematic reviews of evidence of the effects of medical education
.
Teach Learn Med
.
2001
;
13
(
1
):
54
60
. doi:.
23
Kaufman,
R,
Keller
JM.
Levels of evaluation: beyond Kirkpatrick
.
Human Res Develop Quarterly
.
1994
;
5
(
4
):
371
380
.
24
Doran
GT.
There's a S.M.A.R.T. way to write management's goals and objectives
.
Manag Rev
.
1981
;
70
(
11
):
35
36
.
25
Aubrey Daniels International
.
Developing High-Impact Leading Indicators for Safety
. ,
2019
.

Author notes

Editor's Note: The online version of this article contains information related to the design, implementation, and evaluation of a well-being program at the sample institution.

Supplementary data