A Decade of Teaching and Learning in Internal Medicine Ambulatory Education: A Scoping Review Open Access

In 2009, the Review Committee for Internal Medicine of the Accreditation Council for Graduate Medical Education (ACGME) revised the program requirements for internal medicine (IM) residency programs in an attempt to improve ambulatory education. These requirements included: (1) developing ambulatory training models that minimize inpatient-outpatient conflicts; (2) ensuring the completion of at least 130 half-day sessions over a 30-month period; (3) evaluating residents on individual practice-based measures; (4) improving coordination of care and clinic access for patients; and (5) providing supervision that includes longitudinal mentoring.¹ 

Concerns about the future primary care workforce and a recognition that outpatient training in many programs was flawed triggered the program requirement changes.^2,3  Although there are many contributing factors to the projected shortage, one source may be that many residents felt more equipped to practice inpatient medicine than outpatient medicine upon graduation from residency.⁴  This mismatch may push residents into subspecialty training or inpatient-focused careers.

Professional societies, including the Society of General Internal Medicine (SGIM)⁵  and the American College of Physicians (ACP),⁶  have echoed the need for ambulatory training reforms. However, concerns about the readiness of programs to implement these changes have been raised.^7,8  Given the calls for innovation and improvement and the potential concerns about implementation, it is essential that we reflect on what progress has been made and what barriers remain. As a community of educators, it is also crucial to assess how successful we have been in collaborating across institutions and in disseminating successful interventions. Our aim was to conduct a scoping review to explore innovations in IM ambulatory education since the revision of the 2009 ACGME Program Requirements in Internal Medicine.

Scoping reviews are used to “examine the extent, range, and nature of research activity; … to summarize and disseminate research findings; and to identify research gaps in the existing literature.”^9,10  Given our aim to explore a broad topic, a scoping review was best suited to provide a sense of the range of educational research activity. As we wanted to capture work published contemporaneously to the 2009 revision, we reviewed studies published from 2008 onward. Therefore, our eligibility criteria were studies published between 2008 and 2017 in the setting of IM residency training programs in the United States and pertaining to ambulatory care training. We adhered to the methodologic framework laid out by Arksey and O'Malley.¹⁰ 

A search strategy utilizing both index terms and keywords was designed by a medical librarian (R.P.) with expertise in search strategy development. A search query was designed to yield all results containing terms related to IM, residency, and ambulatory care. A second search query was designed to yield all results containing terms related to IM, graduate medical education, ambulatory care, and teaching/training/curriculum. PubMed (National Center for Biotechnology Information, Bethesda, MD), Education Resources Information Center (ERIC, US Department of Education, Washington, DC), and Scopus (Elsevier, Atlanta, GA) databases were searched. Results for both searches were combined, and duplicate results were removed, with the final searches run in May 2017. MedEdPortal (Association of American Medical Colleges, Washington, DC) was searched manually by the lead author (A.C.) using a variety of keyword combinations.

Based on a pilot review of the literature, articles were assigned the following themes: clinic redesign, curriculum development, evaluating resident practice/performance, teaching methods, program evaluation, and faculty development. Titles and abstracts were reviewed by one author (A.C.) to exclude irrelevant results and assign results to the appropriate themes. After initial screening, all articles in a theme were read by a research team member who completed full-text review to confirm relevance, perform data extraction, and complete a quality assessment. While some studies could potentially have been included under multiple themes, for this review, all studies were assigned to a single theme. In the case of a disagreement, the entire research group would discuss the assignment and come to a consensus. Seventeen articles were transitioned from one theme to another after team member discussions.

Quality assessment was conducted using the Medical Education Research Study Quality Instrument (MERSQI) for all studies when complete data were available.¹¹  To ensure consistency, 5 articles were successively reviewed and scored by all authors. Authors then discussed any inconsistencies in ratings and finalized characterizations of the tool. At the conclusion of the pilot process, each article received a score by all reviewers within 1 point. Using the scores of the final 3 articles from the pilot process, the calculated Krippendorff alpha was 0.884, indicating high interrater reliability.¹² 

Literature searches yielded 967 potentially relevant studies; 182 (19%) underwent full data extraction and are included in this scoping review (figure 1). The most frequent reasons for exclusion included interventions that were noneducational (n = 175, 22%), international (n = 151, 19%), non-IM (n = 97, 12%), nonresidency (n = 62, 8%), and inpatient focused (n = 58, 7%), among others. Studies were grouped into 1 of 6 themes (table). Only 1 study was identified as exclusively a faculty development program. Data on the characteristics of each of the 182 studies are provided as online supplemental material.

The average MERSQI score of identified studies was 9.47, and 86% (157 of 182) were single-institution studies. While the number of studies published each year increased slightly toward the end of the period examined, that increase was seen mostly in the clinic redesign and curriculum development themes (figure 2). There was no clear upward trend in the quality of studies as assessed by the MERSQI score (figure 3).

Thirty-four articles (19%) and 24 MedEdPortal submissions (13%) whose primary focus was curricular innovation were identified. Of those 58 articles and submissions, the most common topics addressed were geriatrics and palliative care (n = 10, 17%), chronic pain and substance abuse (n = 8, 14%), quality improvement (n = 6, 10%), and transitions of care (n = 6, 10%), among others.

A number of studies evaluated the effectiveness of geriatrics, palliative care, and advance care planning curricula. Two studies showed improvements in knowledge and self-assessed skills after didactic content, and they both combined traditional didactics with online simulation or role playing.^13,14  Multiple studies explored the benefits of mandated rotations in concert with didactic content,^15–17  and all found high learner satisfaction and improved knowledge on posttesting, but no changes in attitudes or behaviors.

Numerous curricular interventions focusing on chronic pain and substance abuse were published in the last several years. Low levels of trainee comfort with substance abuse management were seen across needs assessments.¹⁸  Most successful interventions combined didactic exposure with either simulation or clinic-based quality improvement interventions. One intervention explored resident training in the screening, brief intervention, and referral to treatment (SBIRT) approach at 4 different sites combined with clinic-wide SBIRT implementation, finding improved identification and documentation.¹⁹  Another study explored the impact of 10 hours of substance abuse instruction combined with preceptor SBIRT training, showing improved knowledge and skills with 6-month retention.²⁰ 

O'Sullivan and colleagues²¹  examined the impact of resident education on smoking cessation and rotation through a dedicated smoking cessation clinic, demonstrating improvements in identifying active smokers, an increase in counseling and prescription of therapies, and improved resident confidence. Two studies looked at the impact of introductory “boot camps” for new interns, with a multiday intervention that included didactics and clinic orientation/tours showing improvements in knowledge and clinical confidence.^22,23 

The relatively limited number of published curricular interventions may be due to the large number of residency programs using extramural curricula (developed at outside institutions for broader use)²⁴ ; however, few groups published examinations of the implementation or impact of these modules in their individual residency programs. Additionally, very few curricula were published in MedEdPortal, limiting the ability to share curricular resources. Upon review of MERSQI subscores, validity evidence for studies was infrequently cited. Most studies used pretest and posttest methods, with no studies demonstrating patient-level outcomes and only 8% (n = 15) showing changes in behavior; knowledge-based outcomes were most frequent, often with demonstration of only short-term retention.

Forty-nine articles (27%) whose primary focus was on developing, implementing, and/or evaluating efforts to redesign ambulatory clinic practice were identified. Of those 49 articles, the most common topics addressed were improving quality of care (n = 14, 29%); X + Y scheduling (models that alternate blocks of inpatient rotations with dedicated ambulatory blocks; n = 12 [24%]); end-of-year handoffs (n = 5, 10%); and transitions of care from inpatient to outpatient (n = 6, 12%), among others.

The highest-rated articles on improving quality of care through clinic redesign described interventions such as medication reconciliation training,²⁵  depression screening protocols,²⁶  creation of an interdisciplinary chronic illness management practice,²⁷  and development of an electronic health record (EHR) decision tool to improve geriatric screening.²⁸  Many of these studies were able to show changes in behavior using EHR data.

Many programs have moved to X + Y scheduling with the aim of improving resident-patient continuity and minimizing inpatient-outpatient conflicts. Three articles focused on continuity with the introduction of an X + Y schedule with mixed results.^29–31  Several studies focused on resident satisfaction with an overall trend toward improvement in reducing inpatient-outpatient conflict^30,32–35  and resident satisfaction with ambulatory education,^30,33,36  but no studies found an increase in the desire to pursue primary care careers.^29,32,36  Only 2 studies evaluated improvement in quality of care in X + Y schedules with mixed results.^31,37  Shalaby and colleagues³⁸  provided a practical review of crafting X + Y schedules. No included studies compared different X + Y permutations, making it unclear which option would be optimal. Further investigation is needed regarding optimal X + Y structures and their long-term effects on trainees and patients.

A survey³⁹  found that only 34% of programs had a year-end handoff process to transition patients from outgoing postgraduate year 3 (PGY-3) residents to incoming PGY-1 residents. Of the 6 articles focusing on ambulatory handoffs, only 3 implemented standardized documentation processes,^40–42  and no changes in patient-level outcomes were noted. Doctoroff et al⁴³  found that residents' patients have less-robust outpatient follow-up after hospitalization as compared with patients with faculty primary care physicians (PCPs). Two of the 49 studies described hospital follow-up visit programs in residency clinics.^44,45  Two other studies reported on resident-driven quality improvement projects that addressed communication during the inpatient-outpatient transition.^46,47  None of these studies showed significant reductions in 30-day readmission rates; however, improvements in resident skills, knowledge, and confidence were noted, and the interventions were well-received.

Although opiate use for chronic pain and substance abuse have become major issues in the ambulatory setting, our search found only 3 of the 49 articles (6%) that described how residency clinics are addressing this issue with a clinic redesign-based approach. Holt and colleagues⁴⁸  described an addiction clinic embedded in the resident practice, staffed by residents, a chief resident, a clinical psychologist, and board-certified addiction medicine specialists, which was associated with high patient and trainee satisfaction.

Forty-one articles (23%) were identified whose primary focus was evaluating resident practice and performance. The most common topics addressed were use of entrustable professional activities (EPAs) and the assessment of transitions of care (both n = 7, 17% each). A wide variety of tools were used in studies to evaluate outcomes. For example, Kessler and colleagues⁴⁹  used self-developed questionnaires to assess medical knowledge, Lundberg⁵⁰  used a previously validated tool to assess communication skills in PGY-2s, and Stark et al⁵¹  demonstrated the feasibility of evaluating professionalism using a 360° National Board of Medical Examiners tool. However, many studies identified in this theme were for single institutions and used institution-specific instruments.

Multiple studies evaluated the use of EPAs, such as hospital transitions of care, demonstrating the potential for the development and incorporation of EPAs to assess specific competencies.^52,53  Meade and colleagues⁵⁴  described a practical method for identifying curricular milestones that can be used to assess whether a resident can be entrusted with increased responsibility, demonstrating across multiple programs the potential application of milestones. A perspective⁵⁵  went further and suggested modifying the primary care exception rule to mandate the evaluation of residents' competence using the existing milestones framework. Legault and colleagues⁵⁶  noted the high frequency of inadequate or inaccurate documentation in resident discharge summaries. They suggested potential solutions, such as clinic and curricular innovations, and even an EPA.⁵² 

Multiple studies identified disparities in the care of patients with resident PCPs (versus faculty PCPs), in areas such as diabetes care, cancer screening, discharge follow-up, and readmission rates.^43,57–62  Given those disparities, further research is needed in assessing resident competence via EPAs and in determining what factors predict worse outcomes in patients seen by residents to identify areas for intervention and find the balance between autonomy and supervision.

Nearly all studies were for single institutions and relied on resident self-assessment or evaluation of EHR documentation, with few studies using direct clinical observation. However, 3 of the 41 studies (7%) describe the evaluation of resident practice through direct observation, demonstrating logistical feasibility and trainee acceptance.^50,63,64  Enhancing opportunities for direct observation may enable better assessment of competency and improved supervision.

Nineteen articles (10%) focused on evaluating and comparing teaching methods used in ambulatory education. The most common teaching modalities evaluated included web-based (n = 10, 53%) and clinical practice as education (n = 4, 21%).

Multiple studies evaluated the use of web-based instruction as an adjunct to other teaching methods or as a stand-alone teaching method, showing successful improvements in knowledge-based outcomes. Sullivan and colleagues⁶⁵  completed a randomized educational trial that compared access to interactive web-based training on opiate prescribing for chronic pain with access to practice guidelines among residents in 5 different residency programs. They found that the web training group had greater improvements in knowledge and self-rated competence.⁶⁵  Cook et al⁶⁶  developed case-based web modules and, using non–case-based modules as a control, completed a randomized crossover trial in 2 academic residency programs, demonstrating that while learners preferred case-based teaching methods, they did not result in superior knowledge retention.

Relatively few studies directly explored changes in clinical practice as a teaching method. Petersen and colleagues⁶⁷  described a trial that randomized faculty-trainee dyads to either conference room presentations or examination room presentations after the trainee completed patient assessments, finding increased time spent with patients and greater patient, trainee, and faculty satisfaction in the examination room presentation model.

A wide variety of teaching methods were used in studies included in this review, although relatively few directly compared outcomes from different teaching methods. The ease of applying web-based training makes it a potentially appealing method. However, little research has been done looking at the benefits of combining clinical practice (and practice transformation) with online training modalities, which may enable higher-level educational outcomes. The articles in the teaching method theme, on average, achieved the highest MERSQI scores, with especially high marks on study design (many studies compared 2 separate groups or even randomized participants). Given the lack of evidence on optimal instructional methods and the methodologic flexibility enabled by a lack of direct impact on patients, further intrainstitutional and interinstitutional, nonrandomized and randomized, controlled trials should be pursued to further clarify optimal didactic methods for postgraduate trainees.

Fourteen articles (8%) related primarily to comprehensive program evaluation. The most common topics addressed related to understanding the current state of national ambulatory care education (n = 4, 29%), evaluating clinic structure and the effect on resident experience (n = 4, 29%), and evaluating of ambulatory education (n = 3, 21%), among others. These studies tended to be national surveys with good response rates; however, most relied on program self-assessment at the trainee satisfaction level.

A few studies used surveys to understand the current state of ambulatory care education or how new regulations affected ambulatory care education.^7,68,69  The majority of residency clinics are hospital based and provide care to patients with lower socioeconomic status. A majority had firm systems with teams caring for patients, and clinic schedules were often secondary to inpatient scheduling needs. The overall experience in clinic was highly variable and perceived levels of trainee stress were high.⁷  Trainee stress was due to competing demands in continuity clinic and poor knowledge about how to manage ambulatory issues. Multiple perspective articles highlighted the need to focus on residents' schedules, emphasizing the importance of patient continuity, teams, and mentors.^70–72  The greatest challenges were in implementing the ACGME requirements, recruiting core faculty, and performing competency-based assessment.^68,69 

Kisiel and colleagues⁷³  observed residents' perspectives on effective outpatient education using focus groups, finding the teacher-learner relationship to be of paramount importance. Creating a second continuity clinic experience at a free clinic site, a private practice clinic site, and a patient-centered medical home clinic were all associated with improved resident satisfaction.^74,75  However, one study that evaluated components of the continuity clinic experience found that while most residents were satisfied with individual components of their ambulatory experience, no single feature was predictive of a career choice in primary care.⁷⁶ 

Larger-scale program evaluation studies noted several similarities to single institution studies previously discussed, with high levels of trainee dissatisfaction and significant clinical and curricular deficiencies. Structural changes, such as X + Y models and team-based care and interventions, to increase the educational quality of the clinic and didactic experience have the potential to address many of these concerns.

Internal medicine residency programs in the United States have sought to improve ambulatory training since the 2009 revision of the ACGME Program Requirements in Internal Medicine. This scoping review identified 182 studies since 2008 relating to ambulatory education; however, the quality and generalizability of the interventions were heterogeneous. Several topics were heavily represented across multiple themes, with interventions relating to quality improvement, transitions of care, and substance abuse being particularly common. Interventions combining curricular changes and clinic restructuring showed the most promise for reaching higher-level outcomes. Given the critical role of clinic faculty, we were surprised to find so few descriptions or evaluations of faculty development programs. Systematic faculty development is likely happening at programs around the United States, but identifying best practices remains a challenge.

Since the 2009 revision for IM residency programs, progress toward meeting several of these requirements has been made. The X + Y block schedules have shown the capacity to minimize inpatient-outpatient conflicts, ensure completion of the requisite number of half-day clinic sessions, reduce trainee stress, and increase satisfaction. Programs have been evaluating residents on practice-based measures and publishing findings, some of which have spurred clinical process transformations. However, while studies evaluating resident practice noted outcome-level disparities for patients, nearly all clinic redesign studies showed only changes in trainee satisfaction, attitudes, and behaviors. Further investigation is needed to find ways to leverage team-based care to ensure care coordination and clinic access for residents' patients. Finally, while EPAs and other competency-based assessments have the potential to determine the level of supervision required for each resident, there was little mention of the longitudinal mentoring component required by the ACGME.

Despite the calls for innovation, the pace of publications has appeared to remain relatively static. The average MERSQI score of studies in this review was 9.47, with no clear trend toward improved quality in the last decade. While it is challenging to compare MERSQI scores among studies (as the optimal methodology depends on the question being asked), a meta-analysis⁷⁷  of studies using MERSQI scores across multiple educational arenas found a median score of 11.3. On examination of MERSQI subscores for articles in this review, studies had markedly lower scores in the study design, data type, and validity domains. Of the 182 studies evaluated as part of this scoping review, only 25 (14%) involved more than 1 institution, limiting generalizability. Most studies relied on self-reported data from clinics and trainees. Very few studies included in this review used standardized tools for evaluation or discussed the validity evidence for their chosen tools, making cross-study comparisons of different interventions challenging. Many studies assessed outcomes at the satisfaction and knowledge/attitudes/skills level (often only immediately postintervention). Few evaluated behavioral and process changes and even fewer assessed patient-level outcomes. Moving beyond self-reported behaviors and EHR-level data into directly observed behaviors and patient-level outcomes will improve learning, teaching, and the quality of scholarship.

However, these calls for higher-level methodology are not new and are not easy to answer. Studies involving multiple institutions, randomization, long-term follow-up, and patient-level outcomes are difficult to perform and require significant investment of time and money. Randomized, controlled trials examining the impact of different work hour regulations (the Flexibility in Duty Hour Requirement for Surgical Trainees [FIRST]⁷⁸  and Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education [iCOMPARE]⁷⁹  trials) were made possible because work hours were seen as a patient safety issue. Given the evidence showing disparities in care for patients in residency clinics compared with patients in faculty practices (with discrepancies at the patient satisfaction, patient access and engagement, and health outcome levels), it is time we made a similar patient safety and equity argument for research funding and regulatory exceptions to evaluate innovative new educational and clinical models in the ambulatory setting. This advocacy can and should take place within relevant professional societies (Alliance for Academic Internal Medicine [AAIM], Society of General Internal Medicine [SGIM], American College of Physicians [ACP], etc), accreditors (ACGME), and governmental agencies (Agency for Healthcare Research and Quality).

This scoping review has multiple limitations. Because it is not a systematic review, it does not reflect the total sum of all work published since 2008 relating to ambulatory IM training. Additionally, we had limited access to unpublished interventions and data. We relied on independent MERSQI scoring, potentially introducing bias into our quality assessment. Finally, we limited our search to studies from US-based IM programs to help improve the ability to compare and contrast interventions, but that does not mean that we cannot learn from our non-IM and international colleagues.

While this review focused on the ambulatory training environment in IM residency programs, we believe the lessons learned apply to ambulatory care in other fields, especially pediatrics, family medicine, and IM subspecialties. We likely share many of the same challenges, and cross-discipline discussion, innovation, and advocacy would be beneficial to all.

Since the 2009 revision of the ACGME Program Requirements in Internal Medicine, some innovative clinical and educational programs have been developed. We highlighted many of the successful published interventions in the intervening years, with clinic redesign efforts and curricular innovations the most frequently explored. There are significant opportunities for collaboration among ambulatory care training programs to explore a wider application of innovations, improve evaluative methods, and move toward the creation of ambulatory education best practices.