https://orcid.org/0000-0001-6149-5842
Shortcomings in health care outcomes have prompted calls for continued reform in medical education.1,2 The American Medical Association (AMA) Accelerating Change in Medical Education initiative has supported innovators in graduate medical education (GME) through the Reimagining Residency (RR) initiative.3,4 The 5-year, $20 million RR initiative funded 11 projects after receiving 252 letters of intent. The main focus areas of the RR projects have been competency-based medical education, transitions, the learning environment, health systems science, and workforce (Figure).
Projects Sponsored by the American Medical Association Reimagining Residency Program
Projects Sponsored by the American Medical Association Reimagining Residency Program
The RR initiative is the first effort of this scale in GME, so understanding the experience of these projects is an important opportunity to learn what hinders and supports innovations in GME. To map the strategic landscape, we used the lens of a SWOT (strengths, weaknesses, opportunities, threats) analysis to process insights from the grant teams’ experience. The SWOT framework examines both internal factors that influence an organization (strengths and weaknesses) and external environmental factors (opportunities and threats),5 seeking to identify future strategic directions.
The authors analyzed the 11 teams’ initial proposals and annual progress reports for 4 years (2020 to 2023), representing 820 pages of narrative content. To narrow the analysis, we focused on items related to the following topics: project rationale and anticipated challenges, achievements, changes to goals, experienced challenges, and reflections on what the project team had learned. Content was manually extracted, representing over 100 pages of text and 62 475 words (80 002 tokens). SWOT creation proceeded in 3 steps: (1) summarization of themes across projects; (2) categorization of themes into SWOT domains; and (3) extracting relevant quotes for each theme to count projects by theme. An iterative process was used that involved extensive human expert review and discussion of the output from large language models (LLMs),6,7 generative artificial intelligence (AI) models trained on vast amounts of text data to generate human-like language responses. Further methodologic details, including LLM prompts, can be found in the online supplementary data. All authors had familiarity with the content because the RR teams met at least twice yearly in person and because principal investigators discussed projects together in monthly virtual meetings. The authors include project leaders (A.F.W, K.T., B.T.G., A.L.D, J.P.T.C.) who had written many of the reports and the AMA Vice President for GME Innovations (J.S.A.) who read all of the reports in their entirety.
The Table shows the summaries that were edited by all of the authors and identifies how prevalent each issue was within the project reports. In order to verify that the projects accounted for in a particular theme truly discussed the theme referenced in the project report, the authors manually reviewed text flagged by the LLM as the signal for counting that project under a particular theme. In a few cases, the quote was determined insufficient or inappropriately attributed. Through serial discussions among the author team, we revised the content and created a final SWOT analysis that first considers the external influences (opportunities and threats) and then turns to the internal characteristics (strengths and weaknesses), an approach recommended to identify strategic directions.8 Based on this analysis, the authors propose strategic considerations for future efforts to innovate in GME.
SWOT on Innovation Projects in GME
Opportunities
Regarding opportunities to take in relationship to the changing health care landscape, the analysis found 3 significant themes: (1) evolutions in health care delivery; (2) social momentum for change; and (3) advanced technologies. Ambulatory practice, telemedicine, and team-based care all present important learning needs to address that may not be in current residency curricula dominated by inpatient experiences. Interprofessional teams are important opportunities to improve patient care and training.
Social issues increase calls for innovation to address social determinants of health, improve workforce diversity, and support trainee well-being. With heightened awareness of these issues since the start of the COVID-19 pandemic, hospital systems, and regulatory bodies may be more open to change.
Innovative technologies like telemedicine, point-of-care ultrasound, and real-time location systems have enabled new ways of collecting data. These data sources, as well as the electronic health records, can fill gaps in assessment tools and create processes that truly support competency-based medical education.9
Threats
Pressures from the surrounding context threaten innovation in GME: RR projects faced cultural inertia, risk-averse health care systems, and conflicting priorities. Without any standardization across programs and institutions in GME, especially in information technology, innovations successful in one setting are not easily adapted to other settings. The corporate structures in health care increasingly prioritize productivity and procedural volume to support the financial bottom line. Government and private equity funders bring additional challenges for educators to navigate, and private companies working in education have their own financial interests. Residents are increasingly joining unions to protect their own interests. Making a business case is essential for continued use and evolution of successful innovations.
Strengths
In the RR initiative, the enthusiasm from project champions and institutional leaders demonstrated the potential for transformative change. The sizable 5-year grants provided support to overcome many hurdles and significant volunteerism supported the efforts. The GME community is primed and motivated to change. Collaborations between institutions and regulatory bodies in the grants reinforced the shared commitment to improving education and care for patients. Creativity, flexibility, and persistence allowed projects to adapt to different settings and evolve in the face of the COVID-19 pandemic. Local, focused faculty development was essential to support dissemination of innovation.
Weaknesses
Challenges within projects come from complex management needs as well as political, logistical, and bureaucratic hurdles. Institutional financial constraints and turnover of key personnel within grant teams impacted continuity and progress. Managing logistics and creating valuable products is challenging without imposing additional burdens on already overburdened health care workers. Disjointed systems exist at different stages of training and function in silos with different priorities, curricula, and assessment systems. Programs and institutions have different levels of readiness and engagement. The rising workload compression in residency leads many residents to feel that they have no time to invest in innovation projects, an essential element to success. With GME already operating on thin margins with overextended trainees and leaders, there is little wiggle room within the system to respond to stress.
Discussion
This SWOT analysis of GME innovations—seen through the lens of the 11 RR projects—highlights strategic considerations essential for future successful innovations in GME. The RR projects spanned many institutions and different mission areas, so these insights around the SWOT for GME innovation have the potential to be broadly useful to educators. These projects demonstrate the appetite for innovation and potential to improve GME, the opportunities to use evolving technology for educational assessment and training, and the magnitude of the barriers efforts might encounter.
Methodologically, we used generative AI to augment human experts’ thematic analysis of project reports; this facilitated rapid identification of themes across heterogenous projects and may be a useful approach for others. AI has limitations, such as “hallucination,” the confabulation of seemingly realistic false conclusions. These were screened for by authors familiar with projects and reports verifying the output. No hallucinations were observed, though some themes were inappropriately assigned. AI included themes across the input text but overemphasized findings early in the input context window, a limitation reported for long-context window analyses10 ; the authors addressed this limitation by verifying the selected quotes from the text used by the LLM to count a particular report under each theme were appropriate and editing when necessary. This allowed some revision of emphasis of themes during the editing process. Future work could employ more sophisticated methods, like retrieval-augmented generation, to ensure outputs are supported by data elements. In this case, the authors had generated the grant report data and their lens-informed interpretation of insights as in qualitative research. However, using this technique to explore unfamiliar data might require other techniques to account for bias.
Conclusions
The RR initiative highlights both the potential and challenges of fostering innovation in GME. The SWOT analysis underscores the importance of leveraging technology, social momentum, and institutional collaboration while addressing barriers such as financial constraints and cultural resistance. Using AI to support human analysis of projects facilitates the process of making connections that can speed forward progress, but these technologies are only tools to support the work. The innovations require creativity and human engagement to build flexible, sustainable models that integrate competency-based education with evolving health care demands, ensuring long-term success in transforming GME.
Just as a SWOT shows an organization its competitive advantage, the collective experience of the RR grant projects shows that momentum for change in GME can come from dissatisfaction with the status quo. This is the moment for change, and we have ample tools and technologies to meet that moment.
References
Editor’s Note
The online supplementary data contains an overview of generative AI augmentation of expert thematic analysis.
