Background

The COVID-19 pandemic displaced newly matched emergency medicine “pre-interns” from in-person educational experiences at the end of medical school. This called for novel remote teaching modalities.

Objective

This study assesses effectiveness of a multisite Accreditation Council for Graduate Medical Education (ACGME) sub-competency-based curricular implementation on Slack during the first wave of the COVID-19 pandemic in the United States.

Methods

Emergency medicine residency programs were recruited via national organization listservs. Programs designated instructors to manage communications and teaching for the senior medical students who had matched to their programs (pre-interns) in spring/summer 2020. Pre- and post-surveys of trainees and instructors assessed perceived preparedness for residency, perceived effectiveness of common virtual educational modalities, and concern for the pandemic's effects on medical education utilizing a Likert scale of 1 (very unconcerned) to 5 (very concerned). Data were analyzed using descriptive statistics and the t test.

Results

Of 276 possible residency programs, 28 enrolled. Of 324 possible pre-interns, 297 (91.7%) completed pre-surveys in April/May and 249 (76.9%) completed post-surveys in June/July. The median weeks since performing a physical examination was 8 (IQR 7–12), since attending in-person didactics was 10 (IQR 8–15) and of rotation displacement was 4 (IQR 2–6). Perceived preparedness increased both overall and for 14 of 21 ACGME Milestone topics taught. Instructors reported higher mean concern (4.32, 95% CI 4.23–4.41) than pre-interns (2.88, 95% CI 2.74–3.02) regarding the pandemic's negative effects on medical education.

Conclusions

Pre-interns reported improvements in residency preparedness after participating in this ACGME sub-competency-based curriculum on Slack.

Objectives 

We aimed to assesses the effectiveness of a virtual multisite ACGME sub-competency-based curricular implementation for newly matched senior medical students transitioning from UME to GME during the first US COVID-19 surge.

Findings 

After implementation via Slack at 28 residency programs, pre-interns' self-perceived preparedness for residency increased both overall and for 14 of 21 ACGME Milestone topics taught.

Limitations 

Participation was voluntary, objective knowledge acquisition was not measured, only one specialty (emergency medicine) was represented, and the Hawthorne effect may have played a role.

Bottom Line 

Traditional methods of training for UME-to-GME transitions are vulnerable to disruption, and the social media implementation of this novel ACGME sub-competency-based curriculum demonstrates significant improvement in self-perceived preparedness for residency.

Senior medical students between the time of the Match and graduation (“pre-interns”) recognize a disconnect between pre-clinical coursework and clinical application,1  making for a daunting transition into residency. Beyond the baseline challenges of this transition, the COVID-19 pandemic in the United States displaced pre-interns from rotations in spring 2020 and forced a transition to safer virtual educational modalities.24 

Short intensive in-person boot camps have previously increased pre-intern preparedness1,58  across many specialties, including emergency medicine (EM), internal medicine, general surgery, obstetrics and gynecology, orthopedic surgery, and other specialties. The Accreditation Council for Graduate Medical Education (ACGME) includes asynchronous learning as an option for providing structured education within residency training programs.9  Up to 20% of planned didactic experiences in EM residency programs may occur asynchronously as individualized interactive instruction, which requires faculty supervision and an evaluative component.10  Several specialties have previously used asynchronous curricula to teach residents,1115  and best practices for incorporating asynchronous education into graduate medical education (GME) curricula for any specialty have been recommended.16,17  Residency programs have flexibility in determining how they offer asynchronous learning experiences. Asynchronous learning that occurs in a virtual collaborative platform in the context of residency education is novel, but is a concept supported by learning theories. Connectivism promotes collaborative modalities when additional knowledge is gained from group interactions.18,19 

Social media platforms such as Slack (Slack Technologies, San Francisco, CA) offer invitation-based private workspaces as virtual classrooms for groups to share and build knowledge. In line with Knowles' adult learning theory20  and Deci and Ryan's self-determination theory,21  such platforms facilitate asynchronous and self-directed learning, as pre-interns can complete such curricula autonomously.22  Slack was previously assessed as a virtual classroom by our Slack Intern Curriculum (SIC) Consortium23  and 2 other medical education groups.24,25  Pre-interns reported improvements in confidence, comfort, and perceived preparedness for residency (PPR) regarding clinical topics gleaned from the ACGME Emergency Medicine Milestones.26  As the pandemic waxes and wanes, and as future global, national, or regional emergencies surface, such innovative remote learning solutions will play an important role in both undergraduate medical education (UME) and GME. Furthermore, even in a non-pandemic environment, virtual learning experiences may prove a valuable transition tool.

As medical educators discovered an unprecedented need for remote teaching platforms to bridge the pandemic-widened educational gap between medical school and residency, we aimed to create and evaluate a curriculum that could be delivered by residency programs to their incoming interns before they graduate from medical school. The curriculum would be grounded in educational theory, based on the ACGME Milestones, and designed for implementation on social media. Our objectives were to understand pre-intern and instructor perceptions regarding the pandemic's impact on medical education, utilization and perceived effectiveness of various virtual educational modalities, and trends in PPR with curricular implementation.

Setting and Participants

This study was prospective and occurred virtually from March to July 2020. No research incentives were offered. In spring 2020, we used national medical organization message board posts to recruit EM residency programs that would be willing to deliver a curriculum to their pre-interns. Of 276 US EM residency programs, 28 enrolled and designated at least one instructor, the institutional title for whom varied (program director, assistant/associate program director, clerkship director, other academic faculty, or chief/senior resident). Residency programs varied in location (mainly Northeast region), primary training site setting (14 were university and 14 were community), length of training (22 were 3-year and 6 were 4-year duration), and number of pre-interns (range 6–25).27  See online supplementary data for individual program details and see Figure 1 for regional breakdown and mean pre-intern class size. Our average pre-intern to instructor ratio was 5:1. Each institution was provided complete set-up and implementation instructions (see online supplementary data). Each instructor contacted and taught only their own institution's pre-interns. We included all incoming EM pre-interns matched to a participating residency, and their participation was voluntary. See online supplementary data for the pre-intern introduction email.

Figure 1

Enrolled Emergency Medicine Residency Programs by Region

Abbreviation: Avg, average.

Note: Enrolled residency programs numbers and average incoming class size by region, utilizing the Association of American Medical Colleges regional breakdown of US emergency medicine residencies.28  Figure created with permission from MapChart.

Figure 1

Enrolled Emergency Medicine Residency Programs by Region

Abbreviation: Avg, average.

Note: Enrolled residency programs numbers and average incoming class size by region, utilizing the Association of American Medical Colleges regional breakdown of US emergency medicine residencies.28  Figure created with permission from MapChart.

Close modal

Interventions

Kern's 6 steps of curriculum development served as our framework to identify gaps, conduct a needs assessment, develop goals and objectives, formulate an educational strategy, then plan the implementation and evaluation.29  See online supplementary data for our application of Kern's framework. We assembled a team at the primary institution consisting of EM academic faculty and residents with prior years of SIC research experience and evaluated each of the 23 ACGME sub-competencies in EM26  for compatibility with virtual learning. Our team identified sub-competencies that could be taught and evaluated via problem-based learning and case discussion on Slack message boards. We deferred sub-competencies that require in-person learning, such as multitasking, professional values, and team management. We considered 8 sub-competencies amenable, and we extracted the 21 respective levels 1 and 2 Milestones for curricular construction and assessment. See online supplementary data for our selected ACGME sub-competencies and Milestone topics. We matched these 8 sub-competencies with 8 common EM clinical scenarios, covering a breadth of anatomic systems and patient presentations. We addressed each sub-competency several times in various contexts throughout the curriculum. Clinical cases consisted of introductions, visual stimuli, sequential clinical questions, templated answers, conclusions, and resources for further reading. Whenever applicable, we used clinical media including imaging results and procedure videos, embedded in the text of each message board post as images, videos, or hyperlinks such that instructors could simply copy and paste the text for the post.

Instructors had access to our curriculum and utilized it for implementation between Match Day and the first clinical day of internship in 2020. Aside from reading the SIC materials provided, no instructor training was required. Within each institution's private SIC channel on their private Slack workspace, instructors posted the case material for their pre-interns, typically over multiple days per case. Message board posts could be seen by all invited channel members, and there was no penalty for wrong answers. We advised instructors to cover all cases provided and in the recommended order, but to personalize content to their teaching methods and promote discussion. While 2 different implementation schedules were suggested to participating instructors, each residency program was free to set the implementation schedule for its own educational activities. Sessions were not coordinated to be delivered simultaneously across residency programs, and pre-interns and instructors did not have access to the Slack message boards from residency programs other than their own. See online supplementary data for curriculum sample material and message board dialogue.

Outcomes Measured

Our curriculum author team, with their prior UME, GME, and SIC research experience, developed the pre-intern and instructor surveys. We based pre-intern PPR surveys on the same sub-competencies (levels 1 and 2 Milestones) taught with our curriculum. We sought internal feedback on survey clarity and usability and incorporated this into the final surveys prior to distribution. See online supplementary data for survey copies. We provided each participant with a survey introduction disclosing the voluntary and anonymous nature and describing data collection. Informed consent was obtained with the first survey question. We collected non-identifying demographic information to allow comparison of pre- and post-surveys. Given that each pre-intern was asked to indicate their residency program institution, to protect anonymity we did not ask them to disclose the medical school attended. Pre-intern respondents rated their PPR regarding the 21 levels 1 and 2 Milestones extracted from our 8 selected sub-competencies on both pre- and post-surveys. Our research group has measured PPR previously in this manner utilizing a Likert scale (1–5).23  We additionally surveyed both pre-interns and instructors on their experiences with education and training during the pandemic, the effectiveness of commonly used virtual medical education modalities, and the effectiveness of the SIC. We elicited feedback about the SIC from both parties on post-surveys. We used a paid, secured SurveyMonkey account to construct and distribute surveys and collect data (SurveyMonkey Inc, San Mateo, CA).

Analysis of Outcomes

We determined sample size by the number of consenting pre-intern survey participants. We assigned non-identifiable respondent codes to allow for matching between pre- and post-survey responses. In addition to descriptive statistics for stand-alone questions, we employed unpaired t test analysis to compare ordinal pre- and post-survey Likert scale (1–5) response distributions for PPR. While t tests are infrequently used to assess for differences between ordinal data, given the large sample size we can assume that responses approach a normal distribution by the central limit theorem, allowing valid use of this analysis methodology. We did not give post-survey participants their pre-survey responses. We analyzed the PPR data using R version 3.6.1 for Windows (R Foundation for Statistical Computing, Vienna, Austria).

The study protocol was exempt as determined by each of the 28 participating institutional review boards, informed consent was obtained at the beginning of all surveys, and no funding was sought or obtained.

Of 324 possible pre-interns, 297 (91.7%) completed pre-surveys in April/May and 249 (76.9%) completed post-surveys in June/July. See Figure 2 for a flow chart of pre-intern participant inclusion and exclusion at every stage of enrollment. All 28 (100%) participating residency programs completed instructor surveys. On a Likert scale from 1 (poor) to 5 (excellent), instructors rated pre-intern engagement with the curriculum to be better-than-average (mean 3.51, 95% CI 3.26–3.76).

Figure 2

Pre-Intern Enrollment Flow Chart

Abbreviation: RR, response rate.

Note: Enrollment flow chart showing pre-interns at each stage of enrollment. Non-discoverable unique identifier codes were created by pre-interns on both pre- and post-surveys for pairing capabilities, but pre-intern data could still be used whether they opted to take both surveys or only one.

Figure 2

Pre-Intern Enrollment Flow Chart

Abbreviation: RR, response rate.

Note: Enrollment flow chart showing pre-interns at each stage of enrollment. Non-discoverable unique identifier codes were created by pre-interns on both pre- and post-surveys for pairing capabilities, but pre-intern data could still be used whether they opted to take both surveys or only one.

Close modal

At the time of the pre-survey, pre-interns reported the number of weeks since performing a physical examination in the emergency department (median 24, IQR 12–32) and in any clinical setting (median 8, IQR 7–12). They reported the number of weeks since their last in-person didactic (median 10, IQR 8–15). They also reported on which asynchronous learning modalities they had been using since Match Day. The most commonly reported modalities (the top 3 were online modules, live virtual lectures, and podcasts) were used in the post-survey question regarding educational effectiveness.

On post-surveys, the pre-interns provided subjective reports of the effectiveness of most frequently cited virtual educational modalities, including the SIC (see Figure 3). They also reported their total weeks of clinical rotation displacement (median 4, IQR 2–6), with the most commonly cancelled rotations being intensive care and anesthesia. Of the 249 post-survey participants, 48 (19.3%) reported graduating medical school early, but only 11 (4.4%) started residency early. Both pre-interns and instructors reported their level of concern that COVID-19 had negatively affected medical education and training. On a Likert scale of 1 (very unconcerned) to 5 (very concerned), the mean concern level was significantly lower for pre-interns (2.88, 95% CI 2.74–3.02) than for instructors (4.32, 95% CI 4.23–4.41).

Figure 3

Subjective Asynchronous Educational Intervention Effectiveness

Abbreviation: FOAMed, free open access medical education.

Note: Mean pre-intern Likert scale (1–5) report of effectiveness of various asynchronous educational modalities they reported using. Error bars represent the 95% CI. Pre-interns were instructed to select “not applicable” for any educational modality they had not used.

Figure 3

Subjective Asynchronous Educational Intervention Effectiveness

Abbreviation: FOAMed, free open access medical education.

Note: Mean pre-intern Likert scale (1–5) report of effectiveness of various asynchronous educational modalities they reported using. Error bars represent the 95% CI. Pre-interns were instructed to select “not applicable” for any educational modality they had not used.

Close modal

Table 1 depicts pre- and post-survey trends in pre-intern PPR regarding the 21 milestone topics taught. There was a significant increase in PPR both overall and regarding 14 of 21 milestone topics. Participants reported feeling more prepared in a wide variety of EM skills, including forming a differential diagnosis, forming a diagnostic plan, recognizing indications for point of care ultrasonography, and interpreting radiographic images.

Table 1

Breakdown of Estimated Costs for Construction, Implementation, and Collaboration Oversight for 8-case Slack Intern Curriculum at 28 Residency Programs

Breakdown of Estimated Costs for Construction, Implementation, and Collaboration Oversight for 8-case Slack Intern Curriculum at 28 Residency Programs
Breakdown of Estimated Costs for Construction, Implementation, and Collaboration Oversight for 8-case Slack Intern Curriculum at 28 Residency Programs

Costs associated with curricular construction, implementation, assessment, and all other materials were borne “in kind” and there was no additional financial support sought or received. See Table 2 for the estimated cost breakdown.

Table 2

Summary Data on Perceived Preparedness for Residency for Enrolled Emergency Medicine Pre-Interns Regarding Levels 1 and 2 Milestone Topics Extracted From 8 ACGME Sub-Competencies

Summary Data on Perceived Preparedness for Residency for Enrolled Emergency Medicine Pre-Interns Regarding Levels 1 and 2 Milestone Topics Extracted From 8 ACGME Sub-Competencies
Summary Data on Perceived Preparedness for Residency for Enrolled Emergency Medicine Pre-Interns Regarding Levels 1 and 2 Milestone Topics Extracted From 8 ACGME Sub-Competencies

Pre-interns reported significant disruptions to clinical educational experiences due to the pandemic but were less concerned about this than instructors. Pre-interns cited podcasts, question banks, and free open access medical education (FOAMed) as the most effective virtual learning modalities, while textbooks and journals articles were cited as least effective. After implementation of our ACGME sub-competency based SIC, pre-interns reported significant improvements in PPR both overall and regarding 14 of our 21 selected milestone topics.

Pre-interns' relatively low concern about the pandemic's negative effects on their education may be due to several factors. Prior literature has suggested there is incongruence between medical student and educator perceptions of preparedness for clinical practice,33  and this pattern has also been demonstrated at more advanced stages of medical training.34  Also, the nature of EM residency application requires frontloading coursework into the senior year of medical school, leaving most pre-interns with little remaining coursework by Match Day.35  Additionally, medical students were generally already comfortable with virtual educational modalities and were using these methods prior to the pandemic.36  The use of podcasts, question banks, and FOAMed may be generational; today's medical students are more technologically inclined and engaged with modern resources than texts and articles, which is an effect that has previously been found among EM residents.37 

Compared with prior SIC pilot data,23  significant PPR improvements were found for more milestone topics (14 of 21 in the present study versus 6 of 21 previously) and sub-competencies (5 of 8 in the present study versus 1 of 8 previously). Overall PPR improved as well—an effect previously demonstrated in Slack-based curricula.12  The modest (albeit statistically significant) mean PPR improvements were likely due to the limited time frame and inevitable difficulties creating a virtual classroom that promotes experiential learning theory. Some aspects of Kolb and Kolb's 6 characteristics of experiential learning38  were integrable (eg, collaboration involves transactions between the pre-interns and the virtual environment), but virtual classrooms themselves are not holistic models of the clinical world to which pre-interns must soon adapt. The nature of the curricular design was grounded in Deci and Ryan's self-determination theory, which required pre-interns to be intrinsically motivated to engage and to take an active role in learning.21  While they received instructor feedback, there were no extrinsic rewards. The way in which pre-interns engaged provides additional evidence of success in developing a virtual curriculum also underpinned by Knowles' adult learning theory.20  Instructors' reports of pre-intern engagement suggested pre-intern readiness to learn and perceptions that the learning was applicable. The SIC also offered a unique opportunity for shyer pre-interns to open up in a virtual setting. Similar to our findings, other pandemic-time virtual classroom social interactions have previously been described as different than in-person venues.4  Some of our instructors found success scheduling real-time sessions in which all pre-interns could engage in discussion on their Slack workspace—a concept which has been previously recommended in the literature.39  Pre-interns oftentimes contributed their own anecdotes or resources to contribute to group knowledge.

While many milestone topics showed significant increases in PPR after the curriculum, one limitation is the lack of objective data on knowledge or skill acquisitions to fully support Kirkpatrick level 2b.40  Only 28 (10.1%) of 276 possible EM residency programs enrolled, which we speculate was due to other commitments during the active pandemic surge at the time of recruitment, and possibly also to email fatigue from national EM organization message board posts. We also were limited by lack of control data and likely the Hawthorne effect because subjects were aware they were being studied. It would have been useful to compare SIC-participating PPR trends with a national EM pre-intern trend during the unprecedented pandemic surge because non-participants may also have experienced PPR improvement. Pre-intern participation attrition was another issue, as family commitments and moving were cited as common barriers toward the end of implementation. Unpaired t tests were thus implemented due to sample size loss between pre- and post-survey response groups, but given the large initial sample size, we felt this approach was valid despite the groups lacking true independence. Notably, the skewed geographic distribution of participating programs prevents generalizability to all EM programs at this time. Additionally, although we believe such a curriculum is generalizable to other specialties and time frames in GME, there is currently no hard evidence to support this speculation.

Future investigations will focus on evaluating curricular outcomes beyond PPR by evaluating pre-interns with methods of clinical knowledge assessment that are well-established and show peak levels of validity evidence. Incorporating pre-interns' preferred modalities for virtual education and curricular reconstruction using the updated ACGME Emergency Medicine Milestones (effective date July 1, 2021),26  is needed. Using various other specialties' milestones41,42  for generalizability across specialties and using higher-level milestones for longitudinal GME training via individualized interactive instruction are also necessary. Finally, as enrollment of participating residency programs grows, we will use the instructor surveys to glean more high-quality data on factors leading to their residency programs' enrollment, as well as instructor perceptions of their pre-interns' PPR trends.

Despite the limitations to our study, we believe that residency program leadership and educational faculty can use our findings to guide implementation of virtual UME-to-GME transition programs with the intent of increasing pre-intern PPR.

After participation in our virtual ACGME sub-competency based SIC, pre-interns showed significant improvement in PPR both overall and regarding 14 of 21 milestone topics addressed. Pre-interns reported displacement from significant in-person educational experiences due to the pandemic but were less concerned about the pandemic's effect on their education than instructors. Among the most common pandemic-time virtual interventions, pre-interns cited online and interactive options as more effective than traditional offline and printed options.

The authors would like to acknowledge the ultrasound faculty at St. John's Riverside Hospital, Drs Angela Cirilli, Ellen Kurkowski, and Connie Yu, for providing ultrasound media for the curriculum. They also wish to acknowledge all collaborators in the Slack Intern Curriculum Consortium for their virtual teaching efforts. The full list of members of the Slack Intern Curriculum Consortium is included in supplemental data.

1. 
Ataya
R,
Dasgupta
R,
Blanda
R,
Moftakhar
Y,
Hughes
P,
Ahmed
R.
Emergency medicine residency boot camp curriculum: a pilot study
.
West J Emerg Med
.
2015
;
16
(
2
):
356
361
.
2. 
Dedeilia
A,
Sotiropoulos
M,
Hanrahan
J,
Janga
D,
Dedeilias
P,
Sideris
M.
Medical and surgical education challenges and innovations in the COVID-19 era: a systematic review
.
In Vivo
.
2020
;
34
(
suppl 3
):
1603
1611
.
3. 
Almarzooq
Z,
Lopes
M,
Kochar
A.
Virtual learning during the COVID-19 pandemic
.
J Am Coll Cardiol
.
2020
;
75
(
20
):
2635
2638
.
4. 
Singhi
E,
Dupuis
M,
Ross
J,
Rieber
A,
Bhadkamkar
N.
Medical hematology/oncology fellows' perceptions of online medical education during the COVID-19 pandemic
.
J Cancer Educ
.
2020
;
35
(
5
):
1034
1040
.
5. 
Blackmore
C,
Austin
J,
Lopushinsky
S,
Donnon
T.
Effects of postgraduate medical education “boot camps” on clinical skills, knowledge, and confidence: a meta-analysis
.
J Grad Med Educ
.
2014
;
6
(
4
):
643
652
.
6. 
Laack
T,
Newman
J,
Goyal
D,
Torsher
L. A
1-week simulated internship course helps prepare medical students for transition to residency
.
Simul Healthc
.
2010
;
5
(
3
):
127
132
.
7. 
Lamba
S,
Wilson
B,
Natal
B,
Nagurka
R,
Anana
M,
Sule
H.
A suggested emergency medicine boot camp curriculum for medical students based on the mapping of Core Entrustable Professional Activities to Emergency Medicine Level 1 Milestones
.
Adv Med Educ Pract
.
2016
;
7
:
115
124
.
8. 
Okusanya
O,
Kornfield
Z,
Reinke
C,
et al
The effect and durability of a pregraduation boot camp on the confidence of senior medical student entering surgical residencies
.
J Surg Educ
.
2012
;
69
(
4
):
536
543
.
9. 
Accreditation Council for Graduate Medical Education.
ACGME Common Program Requirements (Residency)
.
2021
.
10. 
Estes
M,
Gopal
P,
Siegelman
JN,
Bailitz
J,
Gottlieb
M.
Individualized interactive instruction: a guide to best practices from the council of emergency medicine residency directors
.
West J Emerg Med
.
2019
;
20
(
2
):
363
368
.
11. 
Chang
T,
Pham
P,
Sobolewski
B,
et al
Pediatric Emergency medicine asynchronous e-learning: a multicenter randomized controlled solomon four-group study
.
Acad Emerg Med
.
2014
;
21
(
8
):
912
919
.
12. 
Pourmand
A.
Impact of asynchronous training on radiology learning curve among emergency medicine residents and clerkship students
.
Perm J
.
2018
;
22
:
17
055
.
13. 
Smart
J,
Segura Olson
A,
Muck
A.
Asynchronous curriculum “socially synchronized”: learning via competition
.
West J Emerg Med
.
2018
;
20
(
1
):
6
8
.
14. 
Bhashyam
A,
Dyer
G.
“Virtual” boot camp: orthopaedic intern education in the time of COVID-19 and beyond
.
J Am Acad Orthop Surg
.
2020
;
28
(
17
):
e735
e743
.
15. 
Monday
L,
Gaynier
A,
Berschback
M,
et al
Outcomes of an online virtual boot camp to prepare fourth-year medical students for a successful transition to internship
.
Cureus
.
2020
;
12
(
6
):
e8558
.
16. 
Rivera
R,
Smart
J,
Sakaria
S,
et al
Planning engaging, remote, synchronous didactics in the COVID-19 pandemic era
.
JMIR Med Educ
.
2021
;
7
(
2
):
e25213
.
17. 
He
S,
Lai
D,
Mott
S,
et al
Remote e-work and distance learning for academic medicine: best practices and opportunities for the future
.
J Grad Med Educ
.
2020
;
12
(
3
):
256
263
.
18. 
Flynn
L,
Jalali
A,
Moreau
K.
Learning theory and its application to the use of social media in medical education
.
Postgrad Med J
.
2015
;
91
(
1080
):
556
560
.
19. 
Sandars
J,
Patel
R,
Goh
P,
Kokatailo
P,
Lafferty
N.
The importance of educational theories for facilitating learning when using technology in medical education
.
Med Teach
.
2015
;
37
(
11
):
1039
1042
.
20. 
Knowles
M.
Self-directed learning: a guide for learners and teachers
.
Organ Stud
.
1977
;
2
(
2
):
256
257
.
21. 
Deci
E,
Ryan
R.
Intrinsic Motivation and Self-Determination in Human Behavior
.
New York, NY
:
Springer;
1985
.
22. 
Zackoff
M,
Real
F,
Abramson
E,
Li
S,
Klein
M,
Gusic
M.
Enhancing educational scholarship through conceptual frameworks: a challenge and roadmap for medical educators
.
Acad Pediatr
.
2019
;
19
(
2
):
135
141
.
23. 
McLean
M,
Huls
T,
Cotarelo
A,
et al
Social media bridges the training gap between match day and internship with ACGME Milestone-based clinical case curriculum
.
AEM Educ Train
.
2020
;
52):e10503.
24. 
Hill
J,
LaFollette
R,
Grosso
R,
Axelson
D,
Hart
K,
McDonough
E.
Using Slack to facilitate virtual small groups for individualized interactive instruction
.
AEM Educ Train
.
2018
;
3
(
1
):
92
95
.
25. 
Messman
A,
Malik
A,
Ehrman
R.
An asynchronous curriculum for teaching practical interpretation skills of clinical images to residents in emergency medicine
.
J Emerg Med
.
2020
;
58
(
2
):
299
304
.
26. 
Accreditation Council for Graduate Medical Education.
Emergency Medicine Milestones
.
2021
.
27. 
FREIDA.
https://freida.ama-assn.org/. Accessed October 5,
2021
.
28. 
Association of American Medical Colleges.
Organization of Student Representatives: 2016 Handbook
.
2021
.
29. 
Kern
D,
Thomas
P,
Hughes
M.
Curriculum Development for Medical Education. 2nd ed
.
Baltimore, MD
:
Johns Hopkins University Press;
2009
.
30. 
Martin
K.
Medscape Residents Salary & Debt Report 2020
.
2021
.
31. 
Stowell
JR,
Vohra
TT,
Luber
SD.
Emergency medicine resident clinical hours: a national survey
.
J Emerg Med
.
2015
;
48
(
4
):
474
80.e804
.
32. 
Katz
B.
2019–2020 Emergency Physician Compensation Report
.
ACEP Now
.
2021
.
33. 
Kellett
J,
Papageorgiou
A,
Cavenagh
P,
Salter
C,
Miles
S,
Leinster
SJ.
The preparedness of newly qualified doctors—views of foundation doctors and supervisors
.
Med Teach
.
2015
;
37
(
10
):
949
954
.
34. 
Davis
DA,
Mazmanian
PE,
Fordis
M,
Van Harrison
R,
Thorpe
KE,
Perrier
L.
Accuracy of physician self-assessment compared with observed measures of competence: a systematic review
.
JAMA
.
2006
;
296
(
9
):
1094
1102
.
35. 
Jarou
Z,
Hillman
E,
Kellogg
A,
Lutfy-Clayton
L,
Pelletier-Bui
A,
Shandro
J,
eds.
EMRA and CORD Student Advising Guide
.
Irving, Texas: Emergency Medicine Residents' Association; 2019.
2021
.
36. 
Wynter
L,
Burgess
A,
Kalman
E,
Heron
J,
Bleasel
J.
Medical students: what educational resources are they using?
BMC Med Educ
.
2019
;
19
(
1
):
36
.
37. 
Mallin
M,
Schlein
S,
Doctor
S,
Stroud
S,
Dawson
M,
Fix
M.
A survey of the current utilization of asynchronous education among emergency medicine residents in the United States
.
Acad Med
.
2014
;
89
(
4
):
598
601
.
38. 
Kolb
A,
Kolb
D.
Learning styles and learning spaces: enhancing experiential learning in higher education
.
Acad Manag Learn Educ
.
2005
;
4
(
2
):
193
212
.
39. 
Liang
Z,
Ooi
S,
Wang
W.
Pandemics and their impact on medical training: lessons from Singapore
.
Acad Med
.
2020
;
95
(
9
):
1359
1361
.
40. 
Yardley
S,
Dornan
T.
Kirkpatrick's levels and education “evidence”
.
Med Educ
.
2012
;
46
(
1
):
97
106
.
41. 
Accreditation Council for Graduate Medical Education.
Milestones by Specialty
.
2021
.
42. 
Doolittle
B,
Talwalkar
J.
Observations: Milestones for combined training programs—innovation or burden?
J Grad Med Educ
.
2014
;
6
(
2
):
386
387
.

Author notes

Editor's Note: The online version of this article contains resources used in the study.

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: Jennifer Beck-Esmay, MD, is a paid contributor for EM:RAP. EM:RAP was referenced in our curriculum as one of many recommended educational resources for emergency medicine trainees. However, these recommendations were created by the curriculum authors (Drs. Cotarelo, Huls, Kulkarni, McLean, and Park) prior to Dr. Beck-Esmay enrolling in the research project.

This work was previously presented at the Society for Academic Emergency Medicine Annual Meeting, May 11–13, 2021; Council of Residency Directors in Emergency Medicine Academic Assembly, April 11–15, 2021; and New York American College of Emergency Physicians Scientific Assembly, July 6–8, 2021, Bolton Landing, NY.

Slack is a trademark of Slack Technologies Inc. The Slack Intern Curriculum research project is an independent research project and is not affiliated, sponsored, authorized, or otherwise associated with or by Slack Technologies Inc.

Supplementary data