Background

Nonclinical skills (eg, self-regulation, team leadership, conflict resolution) are essential for success as a chief resident (CR). The literature on programs teaching these skills reports few if any effectiveness outcomes.

Objective

We reported the outcomes of a leadership course for CRs using participants' self-reported outcomes and assessments from their program directors (PDs).

Methods

A 2-day curriculum focused on emotional intelligence competencies, including self-awareness, self-management, social awareness, and relationship management. We used a logic model to align 2017–2018 curriculum with targeted outcomes. Questionnaires before and after the course assessed short-term and intermediate outcomes for the participants and PD interviews evaluated observed changes in CRs' performance attributable to the course.

Results

A total of 74 residents participated in the course, and 65% and 59% responded to the post-course and follow-up questionnaires, respectively. Over 95% of respondents indicated developing leadership knowledge and skills and connecting with new CRs in the post-course questionnaire. During follow-up, CRs reported applying concepts learned during chief residency, using tools to address conflict, engaging in quality and patient safety projects, and continuing to interact with other participants. The relationships between reported outcomes and participants' gender/prior leadership training were not significant (P > .05), with small to medium effect sizes (0.01–0.32). All 14 PDs offered positive appraisal of the CRs, but we could not specifically attribute this growth to the course.

Conclusions

Participation in this CR leadership development course was associated with enhancement and application of leadership competencies in immediate and intermediate time frames.

What was known and gap

Being a successful chief resident (CR) requires several nonclinical skills, but the literature does not report the effectiveness of programs that teach these skills to residents.

What is new

A leadership course for CRs using participants' self-reported outcomes and assessments from their program directors.

Limitations

The study examined self-reported outcomes, and it was not possible to attribute skill growth among CRs directly to the course.

Bottom line

Participation in the course was associated with enhancement and application of leadership competencies, in immediate and intermediate time frames.

Being a chief resident is a role that requires management and leadership skills (eg, communicating, aligning groups, advocating, and implementing change). Recommended curricula of health systems science1  and of CanMEDS2  explicitly include leadership as a key competency for trainees. Yet, chief residents (CRs) frequently report low familiarity with these skills.35  There is consensus regarding a need for leadership development for physicians and that such training should begin early in training.68  For example, Blumenthal et al and others have highlighted residency as an opportunity for developing leadership3,4,911 ; they recommend nurturing non-clinical skills such as self-reflection, self-awareness, and self-regulation; team leadership; change management; negotiation; and maintaining professional networks. Because residents provide a significant portion of the care in large teaching hospitals, those who undertake leadership responsibilities can have a significant impact on patient care.

In response to this need to develop leadership skills among CRs, various programs have been established in single specialties, multiple specialties, single institutions, and national programs.1221  Still, published evidence demonstrating the effectiveness of such programs remains sparse, with studies focusing mostly on immediate program outcomes (ie, improvement in participants' knowledge, self-awareness and self-confidence, and anticipated behavior changes16,21–23) and self-reported outcomes. Several systematic reviews of physician leadership development programs have shown substantial evidence gaps, especially regarding long-term measures and high-impact outcomes.2426 

The current study was undertaken to address these gaps. Specifically, we measured immediate and intermediate outcomes associated with a longstanding leadership workshop3  offered to all CRs at the Cleveland Clinic. Outcomes included participants' self-reports as well as assessments from the CRs' program directors (PDs) regarding workshop participant performance and application of these concepts during the subsequent CR year.

Site and Setting

Cleveland Clinic is a fully integrated, multispecialty academic medical center where 985 residents and fellows trained in Cleveland in 75 Accreditation Council for Graduate Medical Education (ACGME) accredited programs and 80 additional programs in the 2017–2018 academic year. At Cleveland Clinic Florida (CCF), 100 residents and fellows trained in 11 ACGME-accredited programs.

Since 2008, a 2-day Chief Residents Leadership Workshop has been offered annually to all CRs in the initial weeks of their chief residency year.3 

Leadership Curriculum

The curriculum was based on longstanding leadership development programs at CCF (eg, Cleveland Clinic Learning Academy, Leading in Health Care, and Samson Global Leadership Academy).3,27,28  These and the current curriculum are grounded in the emotional intelligence (EI) framework of Goleman et al.29  Goals and topics addressed in the course (table 1) correspond to 4 areas of EI competencies: self-awareness, self-management, social awareness, and relationship management. The 12 course sessions were taught by CCF faculty and selected extramural invited experts.

table 1

Features of the Chief Resident Leadership Course

Features of the Chief Resident Leadership Course
Features of the Chief Resident Leadership Course

Course Participants

Current CRs starting their chief residency tenure were nominated to attend by their PDs. They represented all specialties at CCF campuses in Cleveland, Akron, and Florida, and other health centers in Northern Ohio. The course, offered in early August, was announced in various venues, including the CCF GME website,30  program leadership and council meetings, and to ACGME designated institutional officials at other hospitals in Northern Ohio who were invited to nominate CRs.

Course Evaluation

We applied a logic model31  to track and align course activities with resulting outcomes (table 2). Components of this logic model include resources devoted to the course, activities targeted to attain desired outcomes, outputs of course participation, and outcomes from CR course participation. The following 2 questions were investigated in the 2017–2018 academic year course evaluation: What are the immediate (last day of course) and intermediate outcomes (9 months after course completion) for the CRs who participate in the course? What suggestions for improvement, if any, do stakeholders have about the course?

table 2

Logic Model for Chief Resident Leadership Course Evaluation

Logic Model for Chief Resident Leadership Course Evaluation
Logic Model for Chief Resident Leadership Course Evaluation

Data Collection

Data collection for this mixed methods evaluation included questionnaires and interviews. All data collection instruments were piloted and modified based on feedback from medical students in a similar leadership course (pre- and post-course questionnaires), prior CR course attendees (CR follow-up questionnaire), and former PDs (PD interview guide). These documents are provided as online supplemental material.

We used a pre-course questionnaire to assess participants' baseline familiarity with key concepts covered in the course. To investigate short-term or immediate outcomes (table 2), we administered questionnaires with closed- and open-ended questions to participants before and immediately after the course. Intermediate outcomes were assessed using an online questionnaire administered on REDCap, a web-based platform for developing and managing surveys (Vanderbilt University, Nashville, TN). Two authors (S.M. and C.F.F.) also interviewed 14 PDs purposefully selected32  to represent core programs (anesthesiology, family medicine, general surgery, internal medicine, neurological surgery, pediatrics, obstetrics and gynecology, ophthalmology, orthopedic surgery, pathology, pediatrics, psychiatry, radiology, and urology) that had the largest number of residents and who had sent multiple CR course participants. Three of these PDs were from other hospitals in Northern Ohio. Interviews were one-on-one, semistructured, and held either in PDs' offices or by telephone. Interview duration was approximately 10 to 30 minutes. Interview questions were developed from the course content and outcomes and assessed whether PDs observed any changes in CRs' behavior, attitude, or performance attributable to the course, as well as their overall evaluation of the course (interview guide provided as online supplemental material).

This continuous quality improvement project was deemed exempt by the Cleveland Clinic Institutional Review Board.

Data Analysis

Responses to the Likert scale questions on the pre- and post-course and 9-month follow-up questionnaires were entered into SPSS 24 (IBM Corp, Armonk, NY) to compute descriptive statistics of responses and summarize participants' demographic characteristics, including gender, specialty, institutional affiliations, and prior leadership training. CR responses to open-ended questions on the abovementioned questionnaires were analyzed using Dedoose (University of California, Los Angeles, CA), a web-based software for analyzing qualitative and mixed methods research, to uncover emerging themes related to the perceived immediate and intermediate course outcomes. Fisher's exact 2-tailed tests were used to assess differences in CRs' reported immediate and intermediate outcomes across levels of gender and prior leadership training. The a priori level of statistical significance was P < .05. Cramér's V statistic served as a measure of the effect size (ES). This statistic has a lower limit of 0 and an upper limit of 1. We used Cohen's criteria33  to classify ES magnitude (0.10 = small, 0.30 = medium, and 0.50 = large). Data from the PD interviews were transcribed verbatim and coded using Dedoose. Two authors (S.M. and C.F.F.) performed a qualitative content analysis of PD interview transcripts independently and then discussed to reach consensus.34 

Seventy-four new CRs (64% [47 of 74] male) representing 28 specialties and 5 health systems in Ohio and Florida attended the course. Participants included 64 CRs from Cleveland Clinic (Ohio and Florida) and 10 CRs from other medical centers across Ohio.

Achievement of Immediate Outcomes

Pre-Course Questionnaire:

All participants were queried regarding their perceived familiarity with different leadership competencies, including EI, cultural competence, organizational change, conflict resolution, teamwork, and team building. Most indicated levels of familiarity that were below moderate, which corresponded to a mean score of 4 points on a 5-point scale.

In open-ended responses, participants reported a number of motivations for taking this course. Among the top reasons were to (1) develop or improve leadership skills, (2) better serve as CRs, and (3) comply with the recommendation of their program leadership. Some CRs indicated being motivated by their future leadership career aspirations, not having received leadership training before, and personal interest in course content. Participants indicated various reasons for seeking future leadership opportunities in health care organizations, ranging from a desire to pursue careers in academic medicine, improve and innovate health care, or assume specific leadership roles, such as department chairs, residency PDs, and chief of staff.

Post-Course Questionnaire:

Forty-seven of 74 CRs (57% [27 of 47] male) responded to the post-course evaluation (64% response rate), most of whom (57%, 27 of 47) had not had prior leadership training. Over 95% (45 of 47) of respondents indicated they developed leadership knowledge and skills represented by the first 10 curricular goals, related learning objectives of the course, and connected with CR peers they did not know before (table 3). All 47 respondents recommended the course to future CRs. Participants' responses to open-ended questions supported their quantitative ratings (table 4). For example, respondents reported increased awareness of the characteristics of effective teams, change management strategies, and conflict styles and approaches to conflict resolution. No significant relationships were found between reported immediate outcomes and participants' gender or prior leadership training (P > .05). Effect sizes ranged from 0.01 to 0.32, suggesting weak to moderate associations between immediate outcomes and CRs' gender and prior leadership training.

table 3

Immediate Outcomes Reported by Chief Residents in Post-Course Questionnaire

Immediate Outcomes Reported by Chief Residents in Post-Course Questionnaire
Immediate Outcomes Reported by Chief Residents in Post-Course Questionnaire
table 4

Immediate Outcomes Reported by Chief Residents (CRs) in Open-Ended Responses

Immediate Outcomes Reported by Chief Residents (CRs) in Open-Ended Responses
Immediate Outcomes Reported by Chief Residents (CRs) in Open-Ended Responses

Achievement of Intermediate Outcomes

Follow-Up Questionnaire:

Forty-four of 74 CRs (61% [27 of 44] male) responded to the 9-month follow-up questionnaire (59% response rate). Twenty-three of 44 respondents (52%) reported prior leadership training; 4 of these CRs were from institutions other than CCF. Ninety-five percent (42 of 44) of the respondents indicated they had applied concepts learned in the course during their chief residency, and 82% (36 of 44) used tools learned in the course to address conflicts in their programs (table 5). The most widely applied concepts included EI, conflict resolution, and communication, followed by leadership, teamwork, and team building. Forty-one percent (18 of 44) engaged in various quality and patient safety and other improvement projects (table 5). Fifty-two percent (23 of 44) reported continued interactions with other participants through various media following the course (eg, discussions by e-mail, connecting at local conferences, and collaborating on didactics and joint patient care). Those who did not report staying in contact with other CRs cited lack of time, opportunity, or distance. CRs' gender and prior leadership training were not significantly associated with the above intermediate outcomes (P > .05) and the magnitude of associations were small (ES ≤ 0.20).

table 5

Intermediate Outcomes Reported by Chief Residents in Follow-Up Questionnaire (9 Months After Course)

Intermediate Outcomes Reported by Chief Residents in Follow-Up Questionnaire (9 Months After Course)
Intermediate Outcomes Reported by Chief Residents in Follow-Up Questionnaire (9 Months After Course)

Program Director Interviews:

All 14 PDs offered positive appraisal of their CRs who had attended the course and expressed intent to continue nominating CRs for future offerings. Reasons offered for continued nominations of CRs ranged from exposing their chiefs to foundational leadership knowledge and skills to enabling and promoting a community of leaders in order to support one another and share experiences. While the PDs reported growth in the CRs over the course of their chief residency, attribution of this growth to the course was not possible, partly because some PDs were insufficiently familiar with the course curriculum. Others suggested that it would have been helpful if they had been alerted beforehand to assess CR performance in specific domains addressed by the course. Several PDs had discussed the impact of the course with their CRs before the study interviews.

Follow-up feedback from the PDs and CRs was favorable. Additional topics suggested by PDs who knew the course curriculum well included the importance of role modeling to juniors, giving feedback, leveraging one's own strengths, and practice management. Suggestions from CRs centered mainly on promoting more interactivity and including experiential learning activities like case-based scenarios, small group sessions, specific scenarios to help apply concepts taught in the workshop, and discussion of concrete examples whenever possible to illustrate a topic. Some suggested more facilitation of participant introductions and assigning seating to encourage interactions among newly acquainted attendees. To refresh the curriculum throughout the year, some PDs and CRs suggested periodic check-ins with the participants or webinars.

Course Expenses and Feasibility:

Course expenses mainly reflected the cost of travel and lodging for those CRs attending from long distances away, estimated at approximately $1,200 per person (supported by a grant), and an honorarium for one outside faculty member. Costs related to catering for 2 days were modest (∼$30 per attendee). In a full salary model like the Cleveland Clinic, costs related to course director (J.K.S. and C.F.F.) contribution of 2 full days and CRs' time were covered. Administration of the course was supported by a program manager for whom course organization was estimated to require 30 hours (∼0.02 full-time equivalent).

In this mixed methods study investigating the outcomes of a leadership course for CRs from different specialties and health centers across Northern Ohio and Florida, baseline familiarity with leadership competencies was generally low and the self-reported impact of the course on participants' knowledge of leadership competencies was high. During 9-month follow-up interviews, PDs reported growth in their CRs. In line with feedback from the study participants, additional course time will be reserved in the future for CRs to discuss application of course content to their work and solve case-based scenarios in small groups. Course goals and objectives will be shared with PDs to help with assessment of CRs' performance and selection of CRs for the course.

This study extends available findings regarding leadership development for CRs in several ways. In their meta-analysis of leadership training in graduate medical education (GME), Sadowski et al35  pointed out that “leadership curricula are heterogeneous and limited in effectiveness.” Furthermore, most available studies of leadership training programs in GME have assessed only short-term effects, most frequently in post-curriculum or pre-post surveys, and rarely included longitudinal follow-up with participants or measured higher-level outcomes at Kirkpatrick's level 3 (transfer of learning to workplace attributed to educational program) or level 4 (organizational/performance changes directly attributed to educational program).36  In this study, CRs provided specific examples (9 months post-course completion) where they applied leadership tools to alleviate interpersonal conflicts encountered during chief residency (level 3) and implemented quality improvement projects based on course concepts (level 4). Thus, by obtaining both immediate and intermediate term outcomes, the current study addresses a gap in the available literature regarding physician leadership development.2426  Although attribution to the course was largely uncertain, the current study distinctively addressed impacts of the course on participants' leadership behaviors by surveying their PDs, and is unique in including CRs from different specialties and health systems in a single course.37 

Several limitations of the study warrant comment. Bias toward course impact was introduced because CRs are almost uniformly selected for their leadership potential.38,39  Such selection bias and the lack of a control group precludes attributing outcomes to course participation. As with most other studies, this study included self-reported outcomes,40  thereby falling short of more rigorous measures of leadership competency acquisition that might be provided by serial multi-rater feedback on EI or leadership competence. Finally, while CRs from multiple institutions participated, small numbers preclude assessing whether the impact of the course is generalizable beyond the majority of participants from the sponsoring institution.

Future studies of CR leadership development should continue to focus on the concrete, intermediate to long-term impact of course participation. Additional evaluations of the impact of such leadership programs (eg, by addressing questions like: Were participating CRs' programs improved? Was learning by trainees in these programs demonstrably enhanced? Did quality and patient safety improve in the CRs' programs?) will enhance current understanding of the value of such activities and provide opportunities for future research.

This leadership development course for CRs was associated with self-reported enhancement of leadership competencies, both in immediate and intermediate time frames. It was acceptable to the participants and the PDs who recommended them.

1
Skochelak
S,
Hawkins
RE,
Lawson
LE,
Starr
SR,
Borkan
JM,
Gonzalo
JD,
eds
.
Health Systems Science. 1st ed
.
Philadelphia, PA
:
Elsevier;
2017
.
2
Royal College of Physicians and Surgeons of Canada
.
CanMEDS: Better standards, better physicians, better care
. ,
2020
.
3
Farver
C,
Smalling
S,
Stoller
JK.
Developing leadership competencies among medical trainees: five-year experience at the Cleveland Clinic with a chief residents training course
.
Australas Psychiatry
.
2016
;
24
(
5
):
499
505
. doi:.
4
Lee
TH.
Turning doctors into leaders
.
Harv Bus Rev
.
2010
;
88
(
4
):
50
58
. .
5
Morrissey
J.
The Importance of Physician Leadership
.
Trusteemagcom
. ,
2020
.
6
Ackerly
D,
Sangvai
D,
Udayakumar
K,
Shah
BR,
Kalman
NS,
Cho
AH,
et al.
Training the next generation of physician–executives: an innovative residency pathway in management and leadership
.
Acad Med
.
2011
;
86
(
5
):
575
579
. doi:.
7
Collins-Nakai
R.
Leadership in medicine
.
Mcgill J Med
.
2006
;
9
(
1
):
68
73
.
8
Stoller
JK.
Developing physician-leaders: a call to action
.
J Gen Intern Med
.
2009
;
24
(
7
):
876
878
. doi:.
9
Gawande A. Health care needs a new kind of hero
.
Interview by Gardiner Morse
.
Harv Bus Rev
.
2010
;
88
(
4
):
60
61
.
10
Blumenthal
DM,
Bernard
K,
Bohnen
J,
Bohmer
R.
Addressing the leadership gap in medicine: residents' need for systematic leadership development training
.
Acad Med
.
2012
;
87
(
4
):
513
522
. doi:.
11
Fooks
P,
Snudden
C.
Leadership and management: a neglected area of medical education
.
Med Teach
.
2016
;
38
(
10
):
1072
1073
. doi:. 2016.1219024.
12
University of South Florida Health
.
USF Health Honors & Awards
.
New workshop builds leadership skills in medical residents
. ,
2020
.
13
Leadership Skills Training Program for Chief Residents
. ,
2020
.
14
Alliance for Academic Internal Medicine
.
APDIM Chief Residents
. ,
2020
.
15
American Academy of Family Physicians
.
Chief Resident Leadership Development Program
. ,
2020
.
16
Albert Einstein College of Medicine
.
Department of Psychiatry and Behavioral Sciences
.
47th Annual Chief Residents Leadership Conference
. ,
2020
.
17
Rutgers Robert Wood Johnson Medical School
.
Department of Family Medicine and Community Health
. ,
2020
.
18
Colorado Commission on Family Medicine
.
Chief Resident Leadership Training
. ,
2020
.
19
Pettit
J,
Wilson
M.
Leadership certificate program for chief residents
.
Med Educ Develop
.
2014
;
4(1).
doi:.
20
Doughty
R,
Williams
P,
Brigham
T,
Seashore
C.
Experiential leadership training for pediatric chief residents: impact on individuals and organizations
.
J Grad Med Educ
.
2010
;
2
(
2
):
300
305
. doi:.
21
Gisondi
M,
Bavishi
A,
Burns
J,
Adler
M,
Wayne
D,
Goldstein
J.
Use of a chief resident retreat to develop key leadership skills
.
Med Sci Educ
.
2017
;
27
(
2
):
173
176
. doi:.
22
Pettit
J.
Co-leadership among chief residents: exploration of experiences across specialties
.
J Grad Med Educ
.
2015
;
7
(
2
):
203
207
. doi:.
23
Luciano
G,
Blanchard
R,
Hinchey
K.
Building chief residents' leadership skills
.
Med Educ
.
2013
;
47
(
5
):
524
. doi:.
24
Straus
S,
Soobiah
C,
Levinson
W.
The impact of leadership training programs on physicians in academic medical centers: a systematic review
.
Acad Med
.
2013
;
88
(
5
):
710
723
. doi:.
25
Frich
JC,
Brewster
AL,
Cherlin
EJ,
Bradley
EH.
Leadership development programs for physicians: a systematic review
.
J Gen Intern Med
.
2014
;
30
(
5
):
656
674
. doi:.
26
Stoller
JK.
Developing physician leaders: does it work?
BMJ Leader
.
In press.
27
Christensen
T,
Stoller
JK.
Physician leadership development at Cleveland Clinic: a brief review
.
Australas Psychiatry
.
2016
;
24
(
3
):
235
239
. doi:.
28
Mustafa
S,
Farver
C,
Bierer
SB,
Stoller
JK.
Impact of a leadership development program for healthcare executives: the Cleveland Clinic experience
.
J Health Admin Educ
.
2019
;
36
(
1
):
77
92
.
29
Goleman
D,
Boyatzis
R,
McKee
A.
Primal Leadership: Unleashing the Power of Emotional Intelligence
.
Boston, MA
:
Harvard Business School Press;
2013
.
30
Greenwald
L,
Stoller
JK.
Communicating with graduate medical trainees: the Cleveland Clinic experience
.
Perspect Med Educ
.
2013
;
2
(
3
):
142
155
. doi:.
31
Armstrong
E,
Barsion
S.
Using an outcomes-logic-model approach to evaluate a faculty development program for medical educators
.
Acad Med
.
2006
;
81
(
5
):
483
488
. doi:.
32
Patton
M.
Qualitative Research & Evaluation Methods
.
Thousand Oaks, CA
:
Sage Publications Limited;
2002
.
33
Cohen
J.
Chi-square tests for goodness of fit and contingency tables
.
In
:
Statistical Power Analysis for the Behavioral Sciences
.
Orlando, FL
:
Academic Press Inc;
1977
:
215
271
.
34
Hsieh
H,
Shannon
S.
Three approaches to qualitative content analysis
.
Qual Health Res
.
2005
;
15
(
9
):
1277
1288
. doi:.
35
Sadowski
B,
Cantrell
S,
Barelski
A,
O'Malley
P,
Hartzell
J.
Leadership training in graduate medical education: a systematic review
.
J Grad Med Educ
.
2018
;
10
(
2
):
134
148
. doi:.
36
Hammick
M,
Dornan
T,
Steinert
Y.
Conducting a best evidence systematic review. Part 1: from idea to data coding. BEME Guide No. 13
.
Med Teach
.
2010
;
32
(
1
):
3
15
. doi:.
37
Ackerly
D,
Sangvai
D,
Udayakumar
K,
Shah
BR,
Kalman
NS,
Cho
AH,
et al.
Training the next generation of physician–executives: an innovative residency pathway in management and leadership
.
Acad Med
.
2011
;
86
(
5
):
575
579
. doi:.
38
Collier
D,
Mahoney
J.
Insights and pitfalls: selection bias in qualitative research
.
World Politics
.
1996
;
49
(
01
):
56
91
. doi:.
39
Heckman
J.
Sample selection bias as a specification error
.
Econometrica
.
1979
;
47
(
1
):
153
. doi:.
40
Gonyea
RM.
Self-reported data in institutional research: review and recommendations
.
New Direct Inst Res
.
2005
;
127
:
73
89
. doi:.

Author notes

Editor's Note: The online version of this article contains the pre- and post-course questionnaires, the chief resident follow-up questionnaire, and the program director interview guide.

Funding: This work was funded in part by a grant from The Doctors Company Foundation.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

A subset of this study was presented as an innovation in medical education poster at the Association of American Medical Colleges Central Group on Educational Affairs (CGEA) Meeting, Rochester, MN, March 21–23, 2018.

The authors would like to thank Dr. Elias Traboulsi at the Cleveland Clinic for his insights and knowledge regarding graduate medical education and the need for special training of chief residents and The Doctors Company Foundation for their generous support.

Supplementary data