The study reported by Dolan et al1  in this issue of the Journal of Graduate Medical Education epitomizes the type of research that is sorely needed in medical education. The authors conducted a randomized controlled trial in which they assigned internal medicine residents at a large academic practice to receive either the standard curriculum in fracture prevention, or a standard curriculum plus repeated, spaced practice on the material over a 3- to 6-month period. They found that the intervention with repeated practice produced better retention of knowledge 10 months later, and improved the quality of clinical care (bone density screening rates, appropriate use of bisphosphonates, but not FRAX score reporting) provided by the residents during the study. This study is innovative and important in that it assesses the long-term consequences of an educational intervention, and measures its impact on both knowledge and clinical practice.

In education at all levels and in all fields, we tend to focus on short-term outcomes. That is, learning is generally assessed during, or immediately after, the time it occurs (eg, questions posed to students during a lecture, quizzes at the end of class, or final examination after a course). Short-term outcomes are important for many reasons, such as formative and summative assessment. However, when we focus exclusively on short-term outcomes, we make an assumption that is often false: we assume that short-term performance is a good predictor of performance over longer periods of time. Unfortunately, mastery demonstrated during or immediately after learning can be easily lost in the following weeks and months without continued practice.2,3  For example, numerous studies have found that a substantial portion of the basic knowledge and skills acquired in medical school is forgotten by the time individuals enter graduate training, let alone practice.47  Similar patterns occur in graduate medical education, with residents forgetting knowledge and skills that are fundamental to their training.812 

One conclusion that can be drawn from studies that assess long-term outcomes is that we can be shortsighted in how we approach educational interventions. We often devote a substantial amount of effort to facilitating initial learning, but comparatively little effort to maintaining it. For example, a recent study13  assessed the short-term and long-term consequences of redesigning a lecture-based, preclinical pediatrics course to incorporate team-based learning, a pedagogical approach that promotes active learning. Through comparing a group that received the lecture-based version of the course to a group that received the team-based learning version, the study tracked student knowledge of core concepts from the course over time. When knowledge was measured after the end of course, the students who had taken the team-based learning version performed substantially better than students in the lecture-based version. However, when students were given a follow-up knowledge assessment prior to their clerkship, the learning gains in the team-based learning group had disappeared, and the 2 groups performed at the same level. Such findings demonstrate how devoting substantial effort to facilitating initial learning can yield benefits, but those benefits can be easily lost without efforts to maintain knowledge and skills afterward.

Given that learning gains can be easily lost in the absence of continued practice, how can we help medical students, residents, and other health professionals maintain the knowledge and skills that they acquire during training? The intervention implemented by Dolan and colleagues1  provides a possible template. The intervention incorporates several mechanisms known to promote long-term retention and deeper understanding: retrieval practice, feedback, and spaced repetitions.14,15  Retrieval practice refers to the act of retrieving information from memory (eg, solving a practice problem or answering a test question), which is a potent learning event.16,17  Providing feedback increases the benefits of retrieval practice by correcting errors,18  maintaining correct responses,19  and increasing understanding.20  When such practice is spaced or distributed over time, it is more effective than massed practice (eg, cramming).21,22  Numerous studies have shown the efficacy of interventions that incorporate these 3 mechanisms on the retention and transfer of knowledge in undergraduate, graduate, and continuing medical education.2326  To our knowledge, however, the research reported by Dolan and colleagues1  is the first that links such an intervention to improved quality of clinical care.

Of course, it is challenging to assess long-term outcomes and determine the amount of continued practice needed to maintain knowledge and skills that are not regularly used in clinical practice. However, here too Dolan and colleagues1  present a potential solution: the use of technology. Advances in technology are providing educators with powerful new tools that are rapidly changing how people learn both inside and outside the classroom. At a minimum, technology can make providing continued practice more flexible, efficient, inexpensive, and scalable through automation. Yet, the real promise lies in adaptive technology that provides a personalized experience for each learner because it has the potential to exponentially increase the effectiveness and impact of educational interventions.27  Technology is also transforming the way in which health records are kept, providing new opportunities to link learning in the classroom to practice in clinical settings. The ability to assess the impact of educational interventions on clinical care is critical to improving medical education.

In conclusion, we think that the innovative approach taken by Dolan and colleagues1  represents the future of medical education. The training required to become a health professional is substantial. In order to ensure that the learning that occurs during each phase of training is retained over time, more effort must be devoted to maintaining the knowledge and skills acquired. We must also assess long-term outcomes, including the quality of clinical care, so that we can accurately judge the effectiveness of educational interventions. With medical interventions, health professionals routinely follow up with their patients: a surgery is never assumed to be a success immediately after the operation, and a drug is never presumed to be effective based solely on the initial administration. The same approach should be applied to interventions in medical education.

1
Dolan
BM
,
Yialamas
MA
,
McMahon
GT.
A randomized educational intervention trial to determine the effect of online education on the quality of resident-delivered care
.
J Grad Med Educ
.
2015
;
7
(
3
):
376
381
.
2
Bjork
RA.
Memory and metamemory considerations in the training of human beings
.
In
:
Metcalfe
J
,
Shimamura
A
,
eds
.
Metacognition: Knowing About Knowing
.
Cambridge, MA
:
MIT Press;
1994
:
185
205
.
3
Bjork
RA.
Institutional impediments to effective training
.
In
:
Druckman
D
,
Bjork
RA
,
eds
.
Learning, Remembering, Believing: Enhancing Human Performance
.
Washington, DC
:
National Academy Press;
1994
:
295
306
.
4
Custers
EJ
,
ten Cate
OT.
Very long-term retention of basic science knowledge in doctors after graduation
.
Med Educ
.
2011
;
45
(
4
):
422
430
.
5
Custers
EJ.
Long-term retention of basic science knowledge: a review study
.
Adv Health Sci Educ Theory Pract
.
2010
;
15
(
1
):
109
128
.
6
Ling
Y
,
Swanson
DB
,
Holtzman
K
,
Bucak
SD.
Retention of basic science information by senior medical students
.
Acad Med
.
2008
;
83
(
suppl 10
):
82
85
.
7
Sullivan
PB
,
Gregg
N
,
Adams
E
,
Rodgers
C
,
Hull
J.
How much of the paediatric core curriculum do medical students remember?
Adv Health Sci Educ Theory Pract
.
2013
;
18
(
3
):
365
373
.
8
Picciano
A
,
Winter
R
,
Ballan
D
,
Bimberg
B
,
Jacks
M
,
Laing
E.
Resident acquisition of knowledge during a noontime conference series
.
Fam Med
.
2003
;
35
(
6
):
418
422
.
9
Winter
RO
,
Picciano
A
,
Bimberg
B
,
Chae
M
,
Chae
S
,
Jacks
M
,
et al
.
Resident knowledge acquisition during a block conference series
.
Fam Med
.
2007
;
39
(
7
):
498
503
.
10
Fitzgerald
JD
,
Wenger
NS.
Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores?
Acad Med
.
2003
;
78
(
1
):
84
89
.
11
Cacamese
SM
,
Eubank
KJ
,
Hebert
RS
,
Wright
SM.
Conference attendance and performance on the in-training examination in internal medicine
.
Med Teach
.
2004
;
26
(
7
):
640
644
.
12
Pollack
R
,
Baker
RJ.
The acquisition of factual knowledge and the role of the didactic conference in a surgical residency program
.
Am Surg
.
1988
;
54
(
9
):
531
534
.
13
Emke
AR
,
Butler
AC
,
Larsen
DP.
Effects of team-based learning on short-term and long-term retention of factual knowledge
.
Med Teach
.
2015
Apr
21
:
1
6
.
Epub ahead of print
.
14
Larsen
DP
,
Butler
AC.
Test-enhanced learning
.
In
:
Walsh
K
,
ed
.
Oxford Textbook of Medical Education
.
Oxford, UK
:
Oxford University Press;
2013
:
443
452
.
15
Larsen
DP
,
Butler
AC
,
Roediger
HL
3rd.
Test-enhanced learning in medical education
.
Med Educ
.
2008
;
42
(
10
):
959
966
.
16
Butler
AC.
Repeated testing produces superior transfer of learning relative to repeated studying
.
J Exp Psychol Learn Mem Cogn
.
2010
;
36
(
5
):
1118
1133
.
17
Roediger
HL
3rd,
Butler
AC.
The critical role of retrieval practice in long-term retention
.
Trends Cogn Sci
.
2011
;
15
(
1
):
20
27
.
18
Butler
AC
,
Roediger
HL
3rd.
Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing
.
Memory Cogn
.
2008
;
36
(
3
):
604
616
.
19
Butler
AC
,
Karpicke
JD
,
Roediger
HL
3rd.
Correcting a meta-cognitive error: feedback enhances retention of low confidence correct responses
.
J Exp Psychol Learn Memory Cogn
.
2008
;
34
(
4
):
918
928
.
20
Butler
AC
,
Godbole
N
,
Marsh
EJ.
Explanation feedback is better than correct answer feedback for promoting transfer of learning
.
J Educ Psychol
.
2013
;
105
(
2
):
290
298
.
21
Cepeda
NJ
,
Pashler
H
,
Vul
E
,
Wixted
JT
,
Rohrer
D.
Distributed practice in verbal recall tasks: a review and quantitative synthesis
.
Psychol Bull
.
2006
;
132
(
3
):
354
380
.
22
Dempster
FN.
Spacing effects and their implications for theory and practice
.
Educ Psychol Rev
.
1989
;
1
(
4
):
309
330
.
23
Larsen
DP
,
Butler
AC
,
Roediger
HL
3rd.
Repeated testing improves long-term retention relative to repeated study: a randomized controlled trial
.
Med Educ
.
2009
;
43
(
12
):
1174
1181
.
24
Larsen
DP
,
Butler
AC
,
Lawson
AL
,
Roediger
HL
3rd.
The importance of seeing the patient: test-enhanced learning with standardized patients and written tests improves clinical application of knowledge
.
Adv Health Sci Educ Theory Pract
.
2013
;
18
(
3
):
409
425
.
25
Larsen
DP
,
Butler
AC
,
Roediger
HL
3rd.
Comparative effects of test-enhanced learning and self-explanation on long-term retention
.
Med Educ
.
2013
;
47
(
7
):
674
682
.
26
Larsen
DP
,
Butler
AC
,
Aung
WY
,
Corboy
JR
,
Friedman
DI
,
Sperling
MR.
The effects of test-enhanced learning on long-term retention in AAN annual meeting courses
.
Neurology
.
2015
;
84
(
7
):
748
754
.
27
Butler
AC
,
Marsh
EJ
,
Slavinsky
JP
,
Baraniuk
RG.
Integrating cognitive science and technology improves learning in a STEM classroom
.
Educ Psychol Rev
.
2014
;
26
(
2
):
331
340
.