Development, Implementation, and Evaluation of an Open Access, Level-Specific, Core Content Curriculum for Emergency Medicine Residents

Educational leaders in graduate medical education (GME) must provide robust clinical and didactic experiences to prepare residents for independent practice.¹  Regulatory bodies provide a framework for skills and medical knowledge to be acquired during residency, but leave the finer details of specific content, formalized objectives, and educational strategies to programs.^2,3  Programs traditionally tackle didactic experiences individually, requiring tremendous resources with variable content exposure and quality.^4,5 

Medical educators face a number of challenges in delivering high-quality instruction. They often have competing clinical and administrative responsibilities and may have variable resources available to develop and implement innovative educational content.^6,7  Crowdsourcing has been suggested to mitigate the burden of work required to create novel didactic experiences for medical learners,^8,9  but there is no centralized or comprehensive process for core content instruction in any specialty, only limited examples of resources at single locations or for specialized topics.^10–13  The nature of emergency medicine (EM) practice and resident clinical schedules places constraints on didactic time and the majority of programs that use a block weekly format (ie, a weekly half day reserved for didactics).^14–16  Addressing the needs of multiple learner levels within the available didactic time and structure presents an additional challenge. Despite growing support for level-specific didactics, this is far from the norm.^16–20 

Contemporary learners value a focus on teamwork, alternatives to traditional lectures, and the use of technology.^21–25  As an example, residents commonly use online educational resources, including free open access medical education (FOAM).^26–29  This is particularly true in EM, with nearly 98% of residents surveyed using some form of online resource or social media for learning for at least 1 hour per week.³⁰  Multiple studies have demonstrated clear benefit from learner-centric approaches (prioritizing the needs of the learner) such as small group discussion, peer learning, and individualized guidance.^26,31–34  Educators can enhance learning by promoting interactivity during didactics, which has been shown to improve engagement and knowledge retention.^4,25,35–40  The flipped classroom is an example of an interactive learner-centric approach particularly well-suited to the needs of contemporary resident learners. The flipped classroom model incorporates learner-directed study of core concepts in preparation for more interactive didactics, allowing a classroom focus on higher-order thinking.⁴¹  Numerous studies both within and outside of GME demonstrate the benefits of flipped classroom over traditional lectures, including increased learner satisfaction and improved knowledge acquisition.^11,41–56 

We created Foundations of Emergency Medicine (FoEM)⁵⁷  as a national, free, open access, online EM curriculum to address the needs of contemporary residents and medical educators around the country. Our objective was to build standardized, level-specific, core content courses for EM residents utilizing learner-centric educational strategies.

We designed and implemented a national curriculum for postgraduate year (PGY) 1 and PGY-2 EM residents in the United States. The design and implementation process took place from 2014 to 2018 (Table 1), and we collected outcome data in 2019. The development team (K.G.M., N.W., J.B.) employed Kern's model of curriculum development⁵⁸  as a conceptual framework for the development, implementation, and evaluation of FoEM's core content Foundations I (F1) and II (F2) courses, as summarized in the Box.

We reviewed the literature to identify key variables to consider in building educational content for EM trainees, including structural constraints on didactic time (eg, critical care rotations, routine night shift work), variability between 3-year and 4-year EM training programs, and high preference for FOAM and multimedia resources.^20,24,59,60  EM-specific best practice guidelines recommend level-specific instruction, shorter lectures that incorporate active learning, use of the flipped classroom model, clinically relevant small group discussions, and timely feedback with individualized guidance.^16,20,31,52  Other recommended techniques to improve retention and long-term recall include interleaving, guided notes, practice testing, and spaced repetition.¹⁶  Barriers to implementing best practice techniques include a high burden of work for content development and limited faculty buy-in.^24,61 

Educational theories that informed the development of FoEM courses include social constructivism, transformative learning theory, cognitive load theory, and deliberate practice.^18,62–64  A summary of FoEM goals and educational strategies is included in the box; course objectives are included in the online supplementary data. F1, an EM core content course targeted toward PGY-1 learners, is outlined in Table 2; F2, an EM critical care course targeted toward PGY-2 learners, is outlined in Table 3. Fundamental emergent and critical care topics were identified using the American Board of Emergency Medicine's Model of the Clinical Practice of EM (EM Model), which specifies core content for the specialty, and the Accreditation Council for Graduate Medical Education (ACGME) program requirements for EM.^2,59,65,66  Focused content that challenges learners based on training year and experience aligns with cognitive load theory.^18,63  We prioritized the use of learner-centric strategies recommended for the target population, including incorporation of technology, collaboration, and targeted feedback. F1 and F2 make use of a flipped classroom model, allowing directed self-learning using vetted multimodal asynchronous resources (termed “Learning Pathways”) and interactive, clinically oriented small group discussion in the classroom using Foundations cases (tables 2 and 3). Small group cases, led by faculty or senior resident instructors, allow learners to develop clinical reasoning skills and demonstrate simulated patient care (stabilization, diagnosis, management, and disposition) in the classroom setting. Norms of collaborative problem-solving, critical reflection, and peer learning using Foundations cases exemplify social constructivism and transformative learning theory.⁶²  Weekly structured simulated cases within each course support deliberate practice and include timely targeted feedback.⁶⁴  Essential learning points are summarized and shared with learners to fill knowledge gaps and allow for spaced repetition. Coordinated multiple-choice knowledge assessments paired with each unit, provided by third parties, reinforce concepts and help remediate knowledge deficits.

All content is available online. Residency programs can register with FoEM for free and have access to all available content on the FoEM website.⁵⁷  This site includes content to support learner navigation of each course and resources for site leader implementation and management, summarized in the online supplementary data.

The development of the F1 and F2 courses involved a multistep, multiyear process (Table 1) and was led by our authors with advanced training and experience in EM education. The original F1 curriculum was created and piloted at a single 4-year EM residency program beginning in 2014 and a second 3-year EM residency program in 2015. In 2016, a leadership team was recruited to expand the concept, create implementation resources (online supplementary data), develop the FoEM website⁵⁷  using Wordpress, and launch a national pilot. Implementation resources included the following recommendations: required asynchronous preparation in the flipped classroom model, incorporating 30 units of F1 into in-person didactics for PGY-1 learners, and use of faculty or senior resident instructors to facilitate small group learning. We recruited pilot sites through a national EM residency program listserv. We held orientation meetings to discuss implementation recommendations with member sites. The majority of sites followed provided guidelines; however, programs were permitted to modify implementation to address program-specific needs. In 2017, we expanded our leadership team and utilized crowdsourcing to refine the F1 curriculum and expand content for F2. Volunteer EM physicians, primarily recruited from pilot sites, developed content, which was vetted by EM education experts. We used this coproduction and peer-review approach to create 90 F1 cases, 60 F2 cases, 60 coordinated handouts of teaching points for the F1 and F2 organ system-based units, and more than 60 hours of vetted multimodal asynchronous assignments. Given the logistical limitations of our volunteer-based effort, we relied on a mix of FOAM and paid third-party asynchronous resources and coordinated assessments (tables 2 and 3). Since 2017, national use of F1 and F2 has continued to expand as a result of direct recruitment via the EM residency program listserv, word of mouth, and program exposure at professional society conferences through scholarly presentations and abstracts. We utilized an evaluative annual survey to gather user feedback from participating sites. This feedback informed iterative refinement of both courses. Total expenses have ranged from $2,000 to $5,000 annually and were supplied by individual donors, institutional donors, and grant funding. Itemized annual expenses include website development and maintenance ($700–$2,000), administrative costs ($500–$2,000) and contributor appreciation ($200–$1,700).

We administered annual online evaluative surveys to FoEM site leaders and resident learners at the end of the academic year to inform further development and ensure continuous quality improvement. We sent an electronic link to the survey by email to individual leaders and learners utilizing the Qualtrics platform (Qualtrics, Provo, UT). We report key survey results from our 2019 survey (online supplementary data) to reflect the program in its mature form. Our study team of expert EM physician educators developed the survey based on literature review and course objectives to maximize content validity. We incorporated established guidelines for survey research.^67,68  The survey consisted of multiple-choice, numerical, and free-response items. Prior to implementation, we piloted the survey with a small group of reference subjects, including faculty, fellows, and residents. We made revisions for clarity based on results of piloting. We calculated and reported descriptive statistics for items with discrete answer choices. Two analysts (H.C.W., J.J.), experienced in qualitative methods, analyzed free response data using a thematic approach. The study was determined to be exempt by the Institutional Review Board at Stanford University.

Between 2016 and 2020, registration for FoEM course use expanded from 2 sites with 36 learners to 154 sites and 4453 learners (Figure). As of December 2020, registration for the 2020–2021 academic year (AY) included 198 sites and 5453 learners, with growing use at international sites (13) and for new learner types (46 clerkship sites, 9 advanced practice provider programs).

For the 2018–2019 AY, more than half (130 of 247, 53%) of ACGME-accredited EM residency programs in the United States⁶⁹  registered for FoEM. We distributed our online evaluative survey in June 2019 to eligible site leaders and learners. We excluded sites that reported limited content use (eg, piloted only a few cases or units, newly accredited programs without trainees) or registered past the midpoint of the academic year. Ninety-eight of 102 eligible site leaders (96%) and 1618 of 2996 learners (54%) completed the survey. Survey data demonstrated that 97% of sites (95 of 102) utilized the F1 course and 76% of sites (74 of 102) utilized the F2 course. One hundred percent of site leaders were “satisfied” or “very satisfied” with FoEM content and reported average preparation time of 1.16 hours a week (SD 0.79). Seventy-three percent (72 of 98) reported use of coordinated assessments, and 61% (49 of 81) reported that FoEM helped identify learners with gaps in medical knowledge or clinical application. Ninety-three percent of learners (1499 of 1612) were “satisfied” or “very satisfied” with FoEM content. Learners also indicated adherence to asynchronous assignments (mean 1.6 hours a week, SD 0.96). Results of qualitative analysis show that leaders value FoEM's usability, large volume of material, content quality, adaptability, organization, resident-faculty interaction, and resident-as-teacher opportunities. Barriers included resident attendance, conference structure, technology limitations (eg, website download issues, access to asynchronous links), and faculty engagement. Additionally, faculty expertise in small group teaching, oral board style case presentation, and succinct review of targeted learning points was variable. Recommendations for improvement included diversifying and expanding content, frequently updating content, focusing more on evidence-based medicine, and ensuring accessibility of asynchronous resources. Learners identified similar benefits and improvements as leaders. Additionally, learners appreciated facilitation of asynchronous learning and recommended improved technology (eg, improved usability of website, updating inactive links), a refined tracking method, and expanded self-assessments.

Within 3 years (2015–2018), we developed, implemented, and refined 2 level-specific, learner-centric core content courses in EM on a national level. To our knowledge, this endeavor is the first of its kind in any medical specialty.^10–13  F1 and F2 have been widely utilized across US EM residency programs and are viewed positively by faculty leaders and resident learners.

The F1 and F2 courses fill an identified need in GME and provide an easy way for EM program leaders to adopt best practices in education.^{4,25,26,31–35,37–40,70}  The centralized, free, accessible nature of FoEM reduces the burden of work for educators and allows incorporation of level-specific core content education using proven effective learning methods such as active learning, small group collaboration, interleaving, and spaced repetition.^71,72  Further, the flipped classroom model employed by the courses allows for formal implementation of individualized interactive instruction, adhering to recommended guidelines by providing vetted asynchronous resources for independent study.^1,73  These structured Learning Pathways address concerns in the medical education community regarding the scope and quality of online resources as learners move away from use of traditional textbooks.^26–29 

The FoEM curricular model could be replicated to create free, open access, core content for new learner types and other medical specialties. Crowdsourcing has helped ensure feasibility and quality assurance. F1 and F2 content was created and vetted by faculty around the country, incorporating diverse perspectives and accounting for local practice norms. Content is also consumed by a national audience, generating robust feedback and peer review. Of note, our program relies on substantial volunteer effort.⁷⁴  Faculty participation in other settings may be variable and subject to institutional norms regarding the value of digital scholarship for promotion. Finally, maintenance of content and sustainability should be considered, as widespread adoption creates substantial administrative work and, once developed, content must readily adapt to changing needs, course feedback, emerging literature, and updated practice guidelines. FoEM content is updated annually based on user feedback, and a comprehensive content revision is performed at least every 3 years to align with emerging literature and updated practice guidelines.

Although the FoEM courses are standardized, participating programs must address their own unique needs and barriers; this may result in high variability in course implementation or adherence to recommended implementation guidelines. In particular, access to third-party asynchronous assignments and assessments (eg, textbooks, paid online learning platforms) is variable by institution. While we present FoEM as an innovation, we include supporting data based on annual stakeholder surveys. As with all surveys, ours is subject to potential selection bias and both individual and item non-response bias.⁷⁵  Although these biases may exist, given the large number of leader and learner responses, we believe that our fundamental findings are unlikely to be markedly affected. We chose to limit this innovation to a single medical specialty as a pilot. While it is possible the results may not be generalizable to other settings, we believe that there are enough similarities in the needs and delivery of all GME that, with content modification to meet specialty-specific needs, this model may help others create similar programs.

Stakeholders identified several benefits, barriers, and strategies for improvement. FoEM will continue to incorporate user feedback to optimize strengths, troubleshoot weaknesses, and curate curricula. Short-term interventions include updating all F1 and F2 cases to ensure adherence to current evidence-based guidelines and expanding options for self-study resources in our Learning Pathways. Further study is needed to understand the impact of F1 and F2 courses on objective learning outcomes, including medical knowledge and clinical care delivery. Additional topics for future investigation include learner preferences and adherence to asynchronous assignments and characteristics of use of resident instructors within the FoEM model. We are currently working to develop additional FoEM content to support other learner groups, including a Foundations III course for PGY-3 learners and targeted content for medical students, advanced practice providers, and emergency care providers in international settings. Though there are limitations to the curriculum and outcomes as described above, the FoEM curricular model appears to be beneficial, generalizable, and feasible to implement on a large scale. FoEM may be used as a prototype to expand content to new learner types and different specialties across a wide variety of practice environments.

We developed and implemented a learner-centric, level-specific, national EM residency curriculum that has been widely adopted in the United States.

The authors would like to thank the numerous faculty, residents, students, and coordinators who contributed content and expertise to the development and sustainability of Foundations of Emergency Medicine courses.