Geriatrics Curricula for Internal and Family Medicine Residents: Assessing Study Quality and Learning Outcomes

The number of geriatrics fellowship–trained physicians (geriatricians), who provide appropriate care for older adults, is not expected to meet the needs of a growing aging population.^1,2  As a result, many older adults will rely on generalist physicians for their care.^1,2  Education in geriatrics for internal medicine (IM) and family medicine (FM) residents is an important step to equip future generalists and specialists to care for older adults. Equally important, geriatrics training is needed for all specialty and subspecialty residencies.^3–5 

Geriatrics education is a common topic in IM and FM literature.^6–14  Knowledge, skills, and attitudes have historically been included in IM and FM geriatrics competencies.^7,8  Recently, 26 minimum geriatrics competencies (MGCs) for IM and FM residents were developed.¹⁵  Despite a plethora of recommendations, various educational curricula available on PubMed, and an extensive collection of geriatrics teaching materials from online medical education data sets, such as POGOe¹⁶  and MedEdPORTAL,¹⁷  prior reviews of geriatrics curricula^11,18  did not focus specifically on IM and FM, nor did they examine quality, so an evidence gap remains. We performed a systematic review to answer the following overarching question: What is the most effective method of teaching geriatrics to IM and FM residents? Accordingly, we conducted a review of published geriatrics curricula for IM and FM residents that asked the following questions: (1) What kinds of learning outcomes were measured? (2) How were learning outcomes measured? and (3) What was the quality of the studies?

We followed the Best Evidence Medical Education (BEME) guidelines¹⁹  (with modifications) to conduct a systematic review of quantitative studies of geriatrics curricula for IM and FM residents. The modifications consisted of the omission of coding items 1, 7, 9, 10, 11, and 13; the omission of the formal review protocols from BEME guidelines; and the addition of the Medical Education Research Study Quality Instrument (MERSQI), an instrument with validity evidence to assess study quality.^20–22 

Curriculum development for medical education consists of 6 steps: problem identification and general needs assessment, targeted needs assessment, goals and objectives, educational strategies, implementation, and evaluation and feeedback.²³  US accreditation bodies for the different levels of medical education require written curricula with fully developed educational objectives, educational methods, and evaluations.²³  To meet these requirements, we defined that information about the curriculum needed to include 3 steps: (1) learning objectives or geriatrics competencies; (2) teaching methods; and (3) evaluation of learning outcomes. The methods of evaluation (learning outcomes measurement) had to be accomplished by conducting a comparative study, which required either (1) a randomized controlled trial of the intervention versus traditional teaching (control) in 2 or more study groups; (2) a nonrandomized trial of the educational intervention versus traditional teaching (control) in 2 or more study groups; or (3) pretests and posttests or surveys before and after the intervention in a single group or 2 or more study groups. Additional inclusion criteria were that the curriculum had to (1) include geriatrics content; (2) have been designed for IM or FM residents, either alone or in combination with other disciplines; and (3) have been published in an English-language publication between 2003 and 2015.

We excluded curricula that were designed mainly for undergraduate medical education, fellows, or continuing medical education, or were designed for mixed learners (medical students, residents, or fellows); curricula primarily intended for palliative care education; or study design that was noncomparative, such as observational studies, focus groups, case series, review articles, and systematic reviews.

The second author (M.D.) conducted a pilot study that covered geriatrics and palliative care curricula obtained via PubMed only. The study selection process is shown in the figure. For potentially eligible abstracts, articles were retrieved for full review. Final decisions on which articles should be included were made by the first author (data sources and search strategies provided as online supplemental material).

The data collection sheet included the article's first author's name, journal and year of publication, study participants, study setting, rotation type, resident type, sample size, funding, learning objectives (defined as specific and measurable objectives or geriatrics competencies), teaching methods, study design and evaluation methods (learning outcomes measurements), learning outcomes, significance of learning outcomes, effect size, quality of MERSQI study items,^20–22  and strength of findings based on BEME.¹⁹  For each study, the first author recorded (1) learning outcomes to levels 1 to 4 of Kirkpatrick criteria¹⁹; (2) learning objectives or competencies to the 26 MGCs and 7 domains developed by a group of national experts¹⁵ ; and (3) an effect size using Cohen's d (the difference of means between 2 groups or pretest and posttest in a single group divided by the standard deviation of the control group or pretest).²⁴  Effect sizes of 0.2, 0.5, 0.8, and 1.3 of Cohen's d were defined as small, medium, large, and very large, respectively.²⁴ 

The MERSQI, developed by Cook et al²⁰  and Reed et al,^21,22  is used to grade the quality of medical education studies. The instrument consists of 6 domains (study design, sampling, type of data, validity of evaluation instrument, data analysis, and highest outcome). Total MERSQI scores range from 5 (lowest quality) to 18 (highest quality). Each domain is assigned a maximum score of 3. A higher score indicates higher quality of the individual or all domains combined.^20–22 

BEME methods were used to grade the strength of the findings, ranging from 1 (weakest) to 5 (strongest).¹⁹  Grade 1 is defined as “no clear conclusions can be drawn and not significant”; Grade 2 as “results are ambiguous, but there appears to be a trend”; Grade 3 as “conclusions can probably be reached based on the results”; Grade 4 as “results are clear and very likely to be true”; and Grade 5 as “results are unequivocal.”¹⁹ 

Due to the heterogeneity of studies, we decided that it would be inappropriate to combine study results in a meta-analysis, so we instead performed a descriptive analysis. We used chi-square to compare categorical variables. Due to the small number of studies, we used nonparametric tests to compare numerical variables, with P < .05 considered to be statistically significant. Data entry and analysis were performed using SPSS version 22 (IBM Corp, Armonk, NY).

The study selection process is shown in the figure. Fourteen studies met the inclusion criteria.^25–38 

The 14 studies are summarized in table 1. Implementation, rotation sites, participants (IM or FM residents or other specialty trainees), duration, and sample size of each geriatrics curriculum varied. Seven curricula were implemented as a required geriatrics rotation.^{25,27–30,32,33}  Rotation sites were inpatient,^{25,29,32,33,35,36}  long-term care,^26,31  home visit,²⁹  outpatient clinic,^27,28,30,38  or some combination.²⁹  Three studies reported that curricula were embedded in IM rotations.^28,36,37  The duration of the geriatrics experience varied from 4 weeks³³  to 2 years.³⁰  All but 1 rotation (FM)²⁶  and 2 mixed (IM, IM-pediatrics, FM)^29,38  were designed for IM residents. Sample size varied from 28 to 180 residents,^32,37  and on average was 75. Seven studies were funded.^{25,27,28,30,32,33,38} 

As shown in table 2, 7 of the 14 studies stated learning objectives explicitly,^28,32–37  and 7 stated them implicitly.^{25–27,29–31,38}  Teaching methods varied and included lecture or case-based lecture^{27,29,31–34,36} ; didactics^25,28,29 ; web-based³⁰  or electronic medical record modules²⁶ ; cues to action³⁸ ; decision support^27,37  or simulation combined with video and case discssion³⁸ ; grand rounds²⁹ ; interdisciplinary team meetings or rounding^29,32 ; geriatrics patient rounds^25,29,32 ; nursing home rounds³¹ ; morning case report^28,29 ; independent study projects²⁹ ; formative feedback^35,38 ; and direct observation.²⁵  Learning outcomes were matched to Kirkpatrick levels 1 to 4.¹⁹ 

As shown in table 2, no study covered all 26 MGCs and 7 competency domains. Of the 14 studies, 3 covered only 1 of the 26 MGCs,^28,32,34  and 6 covered only 1 of the 7 competency domains.^{25,28,32,34,36,38}  One study covered 14 of the 26 MGCs and 6 of the 7 competency domains.³¹  Median coverage of MGCs from 14 selected geriatrics curricula was 4.

Learning outcomes are summarized in table 3. All but 1 study³¹  reported at least 1 statistically significant change in learning outcomes after the educational intervention. Of the 14 studies, 2 reported changes in educational experience (Kirkpatrick Level 1).^37,39  One of 2 studies showed statistically significant changes in educational experience, which were reflected in higher posttest scores.³⁶  Eleven studies reported changes in attitudes (Kirkpatrick Level 2a).^{25–29,31,33,34,36–38}  Among these, 7 studies showed statistically significant changes in learners' attitudes before and after the described curricula.^{25,28,31,34,36–38}  Thirteen studies reported changes in knowledge and/or skills (Kirkpatrick Level 2b).^{25–33,35–38}  Of these, 10 reported statistically significant changes in knowledge and/or skills.^{25,29–33,35–38}  Five studies reported behavioral changes in learners (Kirkpatrick Level 3),^25–28,30  and 3 of these changes were statistically significant.^25,26,30  No study reported changes in professional practice or benefits to patients (Kirkpatrick levels 4a and 4b).

We were unable to calculate effect size of the educational intervention in most studies, mainly because no standard deviations were reported. Effect size is shown in table 3. In addition to effect size, sizeable changes in medical educational interventions for learning outcomes, based on the authors' comments, are shown in table 3.

Methods of analyzing learning outcomes are summarized in table 3. Designs for assessing learning outcomes also varied. All studies^25–38  used a pretest and posttest, with 29% of posttests conducted at the end of the rotation. One study used a randomized controlled trial design,³⁰  and another compared 2 groups without randomization.²⁵  The remaining studies used preintervention and postintervention in a single group.^{26–29,31–38}  Other assessment methods included focus groups,²⁶  objective structured clinical examinations,³³  encounter checklists,³⁰  chart reviews or audits,^25,28,30  order entry tracking,³⁰  direct observation,^26,28  surveys,²⁷  standardized patient ratings,³⁸  management logs,²⁷  and a participatory process of monitoring reports.²⁸ 

Time to assess learning outcomes is shown in table 3 to demonstrate when the effect of medical intervention occurred. Response rate is also shown in table 3.

Total MERSQI scores and the 6 domain scores are summarized in the online supplemental material. Mean total MERSQI score for the 14 studies was 10.5 and is shown in table 2, indicating that the studies' overall methodological quality was moderate. All studies were conducted in a single institution. In 8 studies,^{26,27,30,32–36}  rates of follow-up with participants were either less than 50% or not reported. Validity evidence for evaluation instruments was generally poor in all studies. However, data analysis was found to be appropriate for study design and type of data in all 14 studies. The average strength of findings from the 14 studies based on BEME was 3. Eleven studies were graded as 3,^{25,26,29,31–38}  1 was graded as 1, 1 as 2, and 1 as 4.^27,28,30 

This review of IM and FM quantitative studies of geriatrics medicine curricula found that only 1 curriculum focused on FM. Learning outcomes were at low to moderate Kirkpatrick levels, yet the effects of educational intervention on attitude and knowledge were sizeable. Time of assessment via pretests and posttests (immediately or in a delayed fashion) and evidence of validity of pretests and posttests were not reported in the majority of studies. Overall study quality, as measured by MERSQI, was moderate, which was comparable to other studies of medical education research. No single study was more promising than others or stood out. The majority of educational interventions appeared feasible and could be applied or modified to apply to other institutions.

We found that learning outcomes in published geriatrics curricula met only low to moderate Kirkpatrick levels.¹⁹  However, geriatrics curricula for IM and FM residents had somewhat higher levels of Kirkpatrick criteria than other educational studies in medicine, dentistry, nursing, and pharmacy education, which have been limited to levels 1 to 2b of Kirkpatrick criteria.^39,40  Instead of pretest and posttest assessments of attitudes and knowledge, future geriatrics curriculum studies should consider assessing resident behaviors with standardized patients or patients, as well as conducting assessments more distant from the curriculum intervention. While patient outcomes represent the highest assessment level,^41–43  this is rarely feasible, due to the complexity of factors affecting patient outcomes and the large numbers of patients required. Unlike the more generous 1990s and early 2000s, little to no funding is available for this type of study. To increase the numbers and diversity of residents and to facilitate comparisons of different educational interventions, IM and FM educators should consider joining regional or national groups to study geriatrics medicine curricula.

All 14 studies except 1 reported statistically significant changes in at least 1 learning outcome.²⁷  Statistically significant changes in learning outcomes are not enough.²⁴  The effect size of a medical intervention is more important than statistical significance.²⁴  Unfortunately, we were able to calculate the effect size of only a few learning outcomes from several studies. The effect size from these learning outcomes was large. Nevertheless, changes in attitude and knowledge were sizeable. In other systematic reviews, the effect size or sizeable changes were not reported.^39,40  Given the small sample sizes in many of the studies in this review, these findings will need to be confirmed on a larger scale in future studies.

Learning outcomes were mainly measured by simple pretests and posttests, similar to other educational studies.^39,40  Traditionally, several methods have been used to evaluate learning outcomes.²³  Single groups with pretests and posttests are easy to implement, but because positive outcomes could be the result of natural maturation or factors unrelated to the curriculum, this method has the potential to be biased.²³  At the other end of the spectrum, controlled trials with or without randomization are complicated to implement, but are also much less likely to be biased.²³  However, nonrandomized designs are common in educational research, and are not considered to be inferior to randomized controlled trials in other areas.⁴⁴  Inherent challenges are apparent in the design and conduct of randomized controlled trial studies in medical education, such as complexity, resource-intensive evaluation,^23,44  and potential difficulty in assessing changes in professional practice and patient benefits. Also, randomization does not control for other sources of errors in education research, such as differences in implementation across settings.⁴⁴  Dividing trainees into 2 groups (ie, medical education intervention and control groups within the current structure of residency training) could pose ethical and practical challenges, and it is difficult to conduct randomized studies with residents who typically move with each rotation.⁴⁴ 

The quality of the 14 selected studies was moderate based on the MERSQI.^20–22  The MERSQI has been applied to educational research in surgery, nursing, IM residency training, complementary medicine education, Internet-based instruction, physical examination education, cultural competency, and other fields,⁴⁵  but not to geriatrics education. Its use for systematic review of geriatrics curricula, therefore, is new. Medians for total MERSQI scores and the 6 domain scores for geriatrics curricula were comparable to published medical educational studies.⁴⁵ 

The strength of the 14 studies' findings was moderate (mean grade on the BEME scale was 3, on a scale of 1 to 5).¹⁹  The BEME scale has only infrequently been reported in the literature. One systematic review of 104 articles on the effectiveness of case-based learning reported that 22 articles had grade 1; 40 had grade 2; 22 had grade 3; 13 had grade 4; and no article had grade 5.⁴⁰  The mean grade on the BEME scale was 2.2,⁴⁰  which was lower than that of the studies in this review. BEME methods¹⁹  have not been used in geriatrics education, and are highly recommended.

In summary, this is the first systematic review to address the aforementioned 3 questions. In addition, we are the first to use MERSQI^20–22  to assess curriculum quality and BEME methods¹⁹  to grade the strength of findings or report effect sizes in geriatrics curricula.

Our review has several limitations. First, our inclusion criteria eliminated a large number of published IM and FM geriatrics curricula, and resulted in a small number of published articles that met our criteria. Our intent was to highlight the need to measure learning outcomes in geriatrics curricula. In an era of outcome-based medical education,^41,43  evaluation of learning outcomes should be included in geriatrics curricula, and perhaps in all curricula. Second, only the first author was involved in the entirety of the project, which means that some aspects were addressed by only 1 individual. Finally, we did not use all available information resources, and we may have missed eligible studies in Scopus, CINAHL, MEDLINE, or Google Scholar.⁴⁶ 

We recommend that future geriatrics curriculum studies consider assessing resident behaviors with standardized patients or patients, as well as conducting assessments more distant from the curriculum intervention. IM and FM educators should also consider expanding beyond their institution, with regional or national groups, to study geriatrics medicine curricula.

Between 2003 and 2015, few published geriatrics medicine curricula for FM and IM residents included assessment of learning outcomes. Overall, study quality based on MERSQI^20–22  and BEME criteria¹⁹  was moderate, with low to moderate Kirkpatrick levels assessed. Studies often focused on only a few geriatrics competencies. Effects of educational intervention on attitude and knowledge were sizeable. The majority of geriatrics curricula reviewed can be applied or adapted to other institutions.