Background

Team-based learning (TBL) is an alternative to traditional lectures in graduate medical education, but evidence is scarce regarding its impact on knowledge acquisition and standardized testing performance.

Objective

We examined the association between resident performance on the Internal Medicine In-Training Examination (IM-ITE) and these 2 educational methods.

Methods

In 2013, the internal medicine residency program at Albany Medical College transitioned from a lecture-based curriculum to TBL. Residents enrolled in academic years 2011–2012 and 2012–2013 comprised the lecture cohort, and those enrolled in 2015–2016 and 2016–2017 the TBL cohort. Covariates included the type of medical school attended, gender, and United States Medical Licensing Examination Step 2 Clinical Knowledge scores. We performed univariate analysis and multivariable regression to determine the association between covariates and ITE scores.

Results

Of 120 residents, 60 were in the lecture cohort and 60 in the TBL cohort. The IM-ITE percent correct scores were higher with TBL than lecture (PGY-1 61.0% vs 55.0%, P < .001; PGY-2 69.0% vs 59.7%, P < .001; PGY-3 73.2% vs 61.7%, P < .001). In a multivariable regression analysis of 3 PGYs combined, the transition from lecture to TBL resulted in an increase in IM-ITE Z-score of 0.415 (P < .001), equivalent to 0.415 SD, when including the effects of all covariates.

Conclusions

Compared to a lecture-based curriculum, TBL was associated with improved resident medical knowledge acquisition as evidenced by higher IM-ITE scores.

Objectives

To determine whether the transition from a lecture-based curriculum to team-based learning (TBL) would improve resident knowledge acquisition.

Findings

In a multivariable regression analysis, a 60 resident cohort exposed to TBL achieved a significantly higher IM-ITE score compared to 60 residents exposed to a lecture curriculum.

Limitations

Single institution limits generalizability.

Bottom Line

Program directors may take interest in this emerging and objective evidence that an active learning method such as TBL can lead to more effective medical knowledge acquisition compared to traditional lectures.

Residency training requires quality educational activities to ensure residents have the requisite medical knowledge to care competently for patients and to pass their board certification examinations. Medical knowledge is traditionally taught in a series of 1-hour noontime lectures,1  though other approaches have emerged such as the academic half-day2,3  and active learning approaches.1,4  Team-based learning (TBL) is an active learning method beginning to generate traction in graduate medical education (GME).5  With TBL, learners acquire knowledge through advance reading assignments and then learn to apply this knowledge through real-world problem-solving exercises lead by a faculty facilitator. These activities commonly include an Individual Readiness Assurance Test (IRAT), Group Readiness Assurance Test (GRAT), and application exercises.6  In GME, TBL has been shown to promote learning and teamwork,7  learner engagement,8  clinical skills development,9  and learner and faculty satisfaction.10  There is limited medical literature, however, describing its effect on medical knowledge acquisition and standardized testing performance in GME.

In 2013, the internal medicine residency program at Albany Medical College transitioned from a lecture-based medical knowledge curriculum to TBL. In 2018, we described the planning, design, faculty development, lesson structure, satisfaction with, and feasibility of this curriculum.11  We hypothesized that an active learning approach using TBL would result in more effective knowledge acquisition than lectures and lead to improvements on residents' Internal Medicine In-Training Examination (IM-ITE) scores. We now report on IM-ITE performance of 2 groups of residents, one exposed to a lecture curriculum and one to a TBL curriculum.

Setting and Participants

Our program is a medium-sized, university-based internal medicine (IM) residency with 48 categorical and 25 preliminary residents. In 2013, we transitioned from a lecture-based medical knowledge curriculum to TBL. In the primary analysis, residents enrolled in academic years 2011–2012 and 2012–2013 who took the ITE comprise the lecture cohort, and those enrolled in 2015–2016 and 2016–2017 comprise the TBL cohort. Five residents in the lecture cohort and 3 in the TBL cohort were excluded as they had not taken the United States Medical Licensing Examination Step 2 Clinical Knowledge (USMLE Step 2 CK). This left 60 residents in each of the 2 cohorts. In a secondary analysis, 28 residents enrolled in 2012–2013 and 2013–2014 and exposed to both lecture and TBL in successive years were analyzed for within-resident score changes. One additional programmatic change of note involved the clinical rotation scheduling structure. Prior to 2013, residents attended afternoon ambulatory clinic twice weekly during each rotation. Beginning in 2013, we switched to a 4+1 block schedule in which 5 staggered cohorts of approximately 10 residents rotate through 4 weeks of inpatient/elective service followed by an ambulatory week with daily clinic.

Intervention

Prior to 2013, the lecture-based curriculum consisted of four 60-minute case-based lecture conferences totaling 240 minutes weekly. Cases were selected from all specialty areas to illustrate specific topics and presented to a senior faculty member for discussion. Faculty discussants, who were unfamiliar with the case to be presented, focused on diagnostic reasoning, evidence-based therapies, and additional curricular elements illustrated by the case. Presenting residents independently prepared and reviewed summaries of key teaching points of the topic of interest relating to the case. The conference learning was passive in nature with limited opportunity for interactive discussion. There were no preparatory reading assignments.

In 2013, we implemented 2 separate TBL curricula. The table of contents of the Medical Knowledge Self-Assessment Program served as the basis of both curricula. A 75-topic hospital curriculum covered 1 topic weekly in a 90-minute conference for all residents, repeating every 18 months. The ambulatory curriculum was taught during a 4-hour academic half-day of the 1-week ambulatory rotation. It was attended by 10 categorical residents weekly and offered 2 separate 120-minute TBL topic lessons. Residents had 10 ambulatory weeks yearly, 20 topics per year, and 60 over 3 years. Total TBL conference time averaged over the 5-week block schedule amounted to 138 minutes weekly. Each TBL lesson had an advance reading assignment requiring an estimated 45 minutes on average reading time. With reading factored in, approximately 200 minutes per week were devoted to TBL.

Outcomes

For all residents, we collected data on medical school attended (US allopathic or osteopathic, Caribbean, or other international), gender, USMLE Step 2 CK scores, and IM-ITE percent correct scores. Over the 6-year span of the study, the mean national ITE percent correct score increased from 63 to 66, and the mean national USMLE Step 2 CK score increased from 230 to 240.12  There were also changes in variance of scores. To make accurate comparisons, we converted both scores to Z-scores based on national means and SDs for each year. The primary study outcome variable was the difference in ITE Z-scores, reflecting knowledge acquisition under lecture and TBL. A Z-score of 0 represents the national mean and a Z-score of 1.0 represents 1 SD from the national mean. Additionally, we examined how the residents' ITE performance affected the program's overall ITE performance compared to all IM programs nationally.

Statistical Analysis

Data are presented as means and SDs for variables measured on a continuous scale with unadjusted comparisons between groups performed by Student's t test. Categorical variables are presented as counts and proportions or percentages with associations between variables assessed by chi-square analysis. Data were analyzed for 3 postgraduate years (PGYs) combined. A multiple regression analysis was conducted of ITE Z-scores as a function of independent variable of curriculum (lecture or TBL) with categorical covariables of school, gender, and PGY (1, 2, or 3) and continuous covariable of USMLE Step 2 CK Z-score. To assure statistical independence, data were analyzed for PGY subgroups (in the combined PGY analysis there may be bias as some residents would have 2 likely related ITE outcome scores and others only one). Adjusted comparisons of ITE Z-scores were by multiple regression with effects of curriculum (lecture or TBL), USMLE Step 2 CK Z-score, school, and gender. In an additional analysis, the within-resident changes in ITE Z-scores for residents exposed to both lecture and TBL (2012–2013 and 2013–2014) were assessed by a paired t test. Statistical analysis was completed using R or Minitab statistical software with significance accepted at P < .05.

The study was declared exempt by the Albany Medical College Institutional Review Board.

The primary analysis included 120 residents with 60 each in the lecture and TBL cohorts. Both cohorts had similar distributions of gender and graduation from a US medical school. The TBL cohort had a larger proportion of international and a smaller proportion of Caribbean graduates. Though exact figures are not available, attendance appeared to remain steady at about 60% of on-duty residents for both cohorts, with the exception of the 10 resident ambulatory week academic half-day TBL, which was close to 100%. At all 3 PGY levels, the unadjusted resident ITE percent correct scores were significantly higher with the TBL curriculum compared to lecture. Similarly, in a univariate analysis, the transition to TBL was associated with significantly higher ITE Z-scores at all PGY levels (Table 1). In a multivariable analysis of all 3 PGYs combined, incorporating effects of curriculum (lecture or TBL) and adjusting for gender, school, PGY, and USMLE Step 2 CK Z-score, the transition from lecture to TBL resulted in an increase in ITE Z-score of 0.415 (P < .001; Table 2). In a multivariable analysis of PGY subgroups of the lecture and TBL cohorts, PGY-1 and PGY-2 residents demonstrated statistically significant increases in ITE performance after adjustment for USMLE Step 2 CK, school, and gender. PGY-1 ITE Z-score increased by 0.393 (P = .049); PGY-2 ITE Z-score increased by 0.435 (P = .007). For PGY-3 residents the ITE Z-score increase of 0.336 was not significant (P = .06; Table 3). To further investigate the effects of the TBL transition, an additional analysis was performed of the 28 PGY-1 and PGY-2 residents exposed to lecture in 2012–2013 and then to TBL in 2013–2014 as PGY-2s and PGY-3s. Adjustments for covariates were not necessary as this paired analysis was within-resident. Among these residents, ITE Z-scores improved from -0.413 to 0.067 for a net change of +0.48 (95% CI 0.226–0.734, P = .001), further supporting the study hypothesis. Finally, in relation to all IM residency programs, these improvements resulted in our program's percentile rank for ITE performance increasing from the 15th percentile nationally in 2011–2012 to the 65th percentile in 2016–2017 (Figure).

Table 1

Resident Characteristics

Resident Characteristics
Resident Characteristics
Table 2

Three PGYs Combined: Multivariable Regression of ITE Z-Score as a Function of Covariates

Three PGYs Combined: Multivariable Regression of ITE Z-Score as a Function of Covariates
Three PGYs Combined: Multivariable Regression of ITE Z-Score as a Function of Covariates
Table 3

PGY Subgroups: Multivariable Regression of ITE Z-Score as a Function of Covariates

PGY Subgroups: Multivariable Regression of ITE Z-Score as a Function of Covariates
PGY Subgroups: Multivariable Regression of ITE Z-Score as a Function of Covariates
Figure

Program's In-Traning Examination Performance Percentile Rank Within Distribution of Mean Scores of All Internal Medicine Programs Nationally

Note: Team-based learning was introduced in 2013–2014 academic year.

Figure

Program's In-Traning Examination Performance Percentile Rank Within Distribution of Mean Scores of All Internal Medicine Programs Nationally

Note: Team-based learning was introduced in 2013–2014 academic year.

Close modal

Consistent with our hypothesis, our program's transition from a lecture-based curriculum to TBL was associated with an improvement in residents' overall IM-ITE performance even after adjustments for the increase in national mean ITE scores (Z-score) and for prior clinical standardized testing performance (USMLE Step 2 CK). To our knowledge this is the first report in the literature of such a finding. The IM-ITE has been demonstrated to correlate with rate and degree of knowledge acquisition during residency13  and predicts performance on the American Board of Internal Medicine Certification Examination (IM-CE).1416  Furthermore, higher performance on the IM-CE has been associated with better patient care outcomes.1719 

Studies have demonstrated that TBL in undergraduate medical education can lead to improvements in students' subject area standardized testing performance.2023  In contrast, the GME literature on TBL is limited, and most studies are descriptive or report on pilot applictions.5,7,10,24  Studies reporting on standardized testing learning outcomes in GME are scarce. One report describes an improvement in the clinical pathology sections of the ITE in a pathology residency, but the increase was not significant.25  Another study shows that higher attendance at 18 rheumatology TBL lessons within an IM residency curriculum was associated with significantly higher rheumatology section IM-ITE scores compared to lower attendance.4 

The reasons our TBL curriculum improved our residents' ITE performance are likely multifactorial. One may be the advance reading assignments. In a survey of IM program directors addressing the decline of programs achieving the minimum 80% IM-CE pass rate, the top reason listed was residents spending “less time independently reading.”26  Additionally, self-directed reading is associated with medical knowledge acquisition in IM training and IM-ITE performance.27  We previously demonstrated that the majority of our residents prepared for TBL by completing reading assignments ahead of time.11  Our residents hold each other accountable for their team's performance; therefore, the incentive to prepare by reading is high.

Educational method and learning theory may also help explain our results. Lecture-based teaching results in low learner engagement, with retention rates estimated as low as 5%.28  TBL is grounded in constructivist learning theory and its 4 elements: (1) it is learner centered; (2) it focuses on problem-solving; (3) it emphasizes dialogue and interaction; and (4) learner reflections help guide integration of new knowledge.29  Our TBL curriculum embodies all of these elements with its learner-centered instruction by a faculty “facilitator” guiding small group work and leading to lively discussion and immediate feedback on real-world, case-based exercises. Though our lecture-based curriculum provided cases as a starting point, the focus of both faculty and presenting residents was directed at gain of factual knowledge as opposed to problem-solving and knowledge application. Given that the ITE emphasizes synthesis and judgment rather than factual recall,13  it naturally draws on residents' critical thinking abilities. TBL, with its grounding in constructivist learning theory, has been demonstrated to improve critical thinking.29  These higher-level analytical skills which our residents gain from TBL are likely beneficial to their ITE performance.

Our study is limited by its implementation at a single residency program and retrospective and observational design. It is also possible that the transition to the 4+1 block schedule may have afforded residents more reading time. And though we controlled for some variables associated with ITE performance, we did not collect data for other variables linked to ITE performance such as age, precise resident conference attendance, self-directed resident use of other electronic educational resources, and USMLE Step 1 scores.27 

As active learning and TBL make further gains in GME, future studies should examine the generalizability of transitioning to a TBL curriculum, its impact on other standardized testing such as board certification examinations, and ultimately on objective measures of the quality of actual patient care.

Implementation of a comprehensive medical knowledge curriculum in IM using TBL is feasible, and compared to a lecture-based curriculum, TBL is associated with improved resident medical knowledge acquisition as evidenced by higher IM-ITE scores.

The authors would like to thank the residents and faculty at Albany Medical College for their participation and enthusiastic embrace of the TBL curriculum, and Maximilian Schynoll for his invaluable technical support in the manuscript preparation.

1. 
Sawatsky
AP,
Zickmund
SL,
Berlacher
K,
Lesky
D,
Granieri
R.
Understanding the challenges to facilitating active learning in the resident conferences: a qualitative study of internal medicine faculty and resident perspectives
.
Med Educ Online
.
2015
;
20
:
27289
.
2. 
Batalden
MK,
Warm
EJ,
Logio
LS.
Beyond a curricular design of convenience: replacing the noon conference with an academic half day in three internal medicine residency programs
.
Acad Med
.
2013
;
88
(5)
:
644
651
.
3. 
Ha
D,
Faulx
M,
Isada
C,
et al
Transitioning from a noon conference to an academic half-day curriculum model: effect on medical knowledge acquisition and learning satisfaction
.
J Grad Med Educ
.
2014
;
6
(1)
:
93
99
.
4. 
Torralba
KD,
Doo
L.
Active learning strategies to improve progression from knowledge to action
.
Rheum Dis Clin North Am
.
2020
;
46
(1)
:
1
19
.
5. 
Poeppelman
RS,
Liebert
CA,
Vegas
DB,
Germann
CA,
Volerman
A.
A narrative review and novel framework for application of team-based learning in graduate medical education
.
J Grad Med Educ
.
2016
;
8
(4)
:
510
517
.
6. 
Parmelee
D,
Michaelsen
LK,
Cook
S,
Hudes
PD.
Team-based learning: a practical guide: AMEE guide no. 65
.
Med Teach
.
2012
;
34
(5)
:
e275
e287
.
7. 
Brandler
TC,
Laser
J,
Williamson
AK,
Louie
J,
Esposito
MJ.
Team-based learning in a pathology residency training program
.
Am J Clin Pathol
.
2014
;
142
(1)
:
23
28
.
8. 
McMullen
I,
Cartledge
J,
Levine
R,
Iversen
A.
Team-based learning for psychiatry residents: a mixed methods study
.
BMC Med Educ
.
2013
;
13
:
124
.
9. 
Wamsley
MA,
Julian
KA,
O'Sullivan
P,
et al
Team-based learning exercise efficiently teaches brief intervention skills to medicine residents
.
Subst Abuse
.
2013
;
34
(4)
:
344
349
.
10. 
Balwan
S,
Fornari
A,
DiMarzio
P,
et al
Use of team-based learning pedagogy for internal medicine ambulatory resident teaching
.
J Grad Med Educ
.
2015
;
7
(4)
:
643
648
.
11. 
Schynoll
G,
Irish
E,
Wayne
J,
Smith
R.
Feasibility of a comprehensive medical knowledge curriculum in internal medicine using team-based learning
.
J Grad Med Educ
.
2018
;
10
(1)
:
78
83
.
12. 
National Board of Medical Examiners.
USMLE Score Interpretation Guides
.
2021
.
13. 
Garibaldi
RA,
Subhiyah
R,
Moore
ME,
Waxman
H.
The In-Training Examination in internal medicine: an analysis of resident performance over time
.
Ann Intern Med
.
2002
;
137
(6)
:
505
510
.
14. 
Babbott
SF,
Beasley
BW,
Hinchey
KT,
Blotzer
JW,
Holmboe
ES.
The predictive validity of the internal medicine in-training examination
.
Am J Med
.
2007
;
120
(8)
:
735
740
.
15. 
Rayamajhi
S,
Dhakal
P,
Wang
L,
et al
Do USMLE steps, and ITE score predict the American Board of Internal Medicine certifying exam results?
BMC Med Educ
.
2020
;
20
(1)
:
79
.
16. 
McDonald
FS,
Jurich
D,
Duhigg
LM,
et al
Correlations between the USMLE step examinations, American College of Physicians in-training examination, and ABIM internal medicine certification examination
.
Acad Med
.
2020
;
95
(9)
:
1388
1395
.
17. 
Turchin
A,
Shubina
M,
Chodos
AH,
et al
Effect of board certification on antihypertensive treatment intensification in patients with diabetes mellitus
.
Circulation
.
2008
;
117
(5)
:
623
628
.
18. 
Papadakis
MA,
Arnold
GK,
Blank
LL,
et al
Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards
.
Ann Intern Med
.
2008
;
148
(11)
:
869
876
.
19. 
Norcini
JJ,
Lipner
RS,
Kimball
HR.
Certifying examination performance and patient outcomes following acute myocardial infarction
.
Med Educ
.
2002
;
36
(9)
:
853
859
.
20. 
Levine
RE,
O'Boyle
M,
Haidet
P,
et al
Transforming a clinical clerkship with team learning
.
Teach Learn Med
.
2004
;
16
(3)
:
270
275
.
21. 
Warrier
KS,
Schiller
JH,
Frei
NR,
Haftel
HM,
Christner
JG.
Long-term gain after team-based learning experience in a pediatric clerkship
.
Teach Learn Med
.
2013
;
25
(4)
:
300
305
.
22. 
Saudek
K,
Treat
R.
Team-based learning on a third-year pediatric clerkship improves NBME subject exam blood disorder scores
.
Med Educ Online
.
2015
;
20
:
29021
.
23. 
Krase
K,
Pfeifer
E,
Swan
K.
Team-based learning sessions compared with traditional lecture in the obstetrics and gynecology clerkship
.
Obstet Gynecol
.
2018
;
132
(suppl 1)
:
14
18
.
24. 
McMullen
I,
Cartledge
J,
Finch
E,
Levine
R,
Iversen
A.
How we implemented team-based learning for postgraduate doctors
.
Med Teach
.
2014
;
36
(3)
:
191
195
.
25. 
Ranheim
EA.
A novel program for clinical pathology training for residents emphasizing high-impact and attending-level learning opportunities
.
Hum Pathol
.
2014
;
45
(2)
:
206
212
.
26. 
Willett
LL,
Halvorsen
AJ,
Adams
M,
et al
Factors associated with declining residency program pass rates on the ABIM Certification Examination
.
Am J Med
.
2016
;
129
(7)
:
759
765
.
27. 
McDonald
FS,
Zeger
SL,
Kolars
JC.
Factors associated with medical knowledge acquisition during internal medicine residency
.
J Gen Intern Med
.
2007
;
22
(7)
:
962
968
.
28. 
Cooper
AZ,
Richards
JB.
Lectures for adult learners: breaking old habits in graduate medical education
.
Am J Med
.
2017
;
130
(3)
:
376
381
.
29. 
Hrynchak
P,
Batty
H.
The educational theory basis of team-based learning
.
Med Teach
.
2012
;
34
(10)
:
796
801
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.