Implementation of a Web-Based Tool With Text Message Prompts to Improve End-of-Shift Assessments for Emergency Medicine Residents

Assessment and feedback are critical components of the education and training of medical trainees.¹  The Accreditation Council for Graduate Medical Education (ACGME) requires that residency programs provide trainees with assessment of clinical performance,²  but does not specify how such assessments and feedback should be conducted or recorded.

One potentially useful form of trainee assessments and feedback occurs at the end of each clinical day or shift. These end-of-shift assessments (ESA) overcome limitations of less frequent end-of-rotation assessments and allow for contemporaneous feedback to trainees.^3–5  However, ESA add daily burden to clinicians to complete and document these assessments, limiting systematic submissions. ESA, when traditionally documented using paper forms, also adds significant administrator burden to collect and collate submissions.

Electronic platforms can assist ESA by prompting clinicians to perform ESA and automating the collection of qualitative and quantitative feedback on trainee performance. Electronic ESA platforms have been used for surgical residents^1,4  and medical students.^1,5  There is some evidence that electronic ESA programs improve completion rates above paper forms.^6–8  Still, evidence of uptake and utility of computerized ESA programs remains scant. In this study, we examine the implementation of a web-based tool with text message prompts to assist mobile ESA (mESA) in an emergency medicine (EM) residency program.

We implemented the system in January 2018 in a 3-year EM residency program where trainees rotate through 4 emergency departments (EDs). Each site is an academic, tertiary care hospital, with a combined annual volume of approximately 200 000 ED patient visits.

The mESA has several core components (box 1, figure 1). For faculty, at the scheduled shift start time, the program sends a text message indicating which trainee is to be assessed and assigns a randomly generated content area. Content areas are based on ACGME EM milestones⁹  and include communication, medical decision-making, efficiency, professionalism, and focused history and physical examinations (see table in online supplemental material). The intent is to “prime” the faculty member to provide feedback and allow them to prospectively collect experiential data during the shift to inform their assessment.

One hour prior to shift end, the mESA program sends a text hyperlink to a secure mESA website where the faculty member completes a 3-question assessment form (screenshots provided as online supplemental material). If feedback is not submitted by 8am the day following the prompt, a reminder text is sent. Each month, faculty received the text: “In [month] you completed [#] out of [#] assessments for a [%] completion rate.”

For trainees, at shift start, the mESA program texts them the name of the assigned faculty member and content area. One hour prior to shift end, the program texts: “Did [faculty] provide you with any in-person feedback on [X] performance today? Please reply Y or N.”

Prior to mESA implementation, all 4 sites used paper ESA cards. After internally testing the mESA program, participants were able to opt out at any time. Following a stepwise process, site 1 ran for 3 months, then sites 2 and 3 were added in month 4, and site 4 in month 5.

We evaluate our mESA using the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework.^10,11  Data sources included: a database of incoming and outgoing text messages and web-based feedback (Microsoft Access), a web-based database of survey results (REDCap), and descriptive residency leadership feedback collected via email.

Reach was assessed by examining the representativeness of faculty opting into the program and measuring the number of residents participating. Effectiveness was assessed through surveys from faculty, trainees, and residency leadership and through ESA reports from trainees. Adoption was assessed by examining the percentage of ESA prompts where faculty submitted online trainee feedback. We examined the relationship between program burden (ie, number of prompts) and online submission rates by faculty using correlation coefficients. Implementation quality was assessed by examining technological errors and issues. Maintenance was assessed by examining trends in ESA submission rates over time.

Approximately 2 months after mESA program initiation at site 1, we invited the EM faculty at this site to participate in a 9-question (1 open- and 8 closed-ended) online survey (provided as online supplemental material). The survey, developed by the authors (clinical educators) without further testing, was designed to gauge faculty members' experience with providing residents with end-of-shift feedback before and after the implementation of the web-based tool. We also evaluated faculty comfort and satisfaction levels with the new program and solicited faculty feedback and suggestions for improvement. The faculty survey was only sent to site 1 because this was the first of our 4 sites to come online. Survey feedback from this site helped inform our rollout process at the other sites.

Similarly, approximately 2 months after mESA program initiation, we invited EM residents to participate in a 10-question (1 open- and 9 closed-ended) online survey (faculty and resident surveys provided as online supplemental material). A single email invitation was sent to complete the survey. The resident survey was designed to assess resident perceptions of how often they received performance feedback from faculty before and after implementation of the program. We also assessed satisfaction with the new system and solicited suggestions for improvement. After approximately 1 year, residency leaders were surveyed over email to provide descriptive feedback on mESA.

The institutional review board at the University of Pittsburgh approved the survey study. The remainder of the data was collected as quality improvement.

We targeted 100 eligible faculty to participate in the mESA. Four opted out (96% reach). Feedback from faculty who opted out suggested they preferred to provide feedback through other methods (n = 2) or did not own a smartphone (n = 2). Of 80 eligible trainees, 79 were enrolled in the mESA program over the 22 months of implementation, with only one opting out (99% reach).

Eighteen faculty from site 1 (85% of eligible faculty at this site) completed the survey (10 faculty with up to 10 years of practice, 8 with >10 years). Faculty endorsed the following reasons for not providing ESA feedback when using paper cards (prior to mESA): 14 forget, 14 felt they were too busy, 13 felt the residents were too busy, and 4 felt discomfort with providing feedback. Thirteen of 18 (72%) agreed that receiving a prompt at the beginning of a shift made it easier for them to collect relevant data to inform feedback; 17 of 18 (94%) agreed that the reminder at the end of the shift was useful to provide in-person feedback, and 16 of 18 (89%) would recommend mESA to other training programs. Other feedback included concern about the narrow scope of content areas (n = 2) and feeling burdened by the number of prompts (n = 3).

The survey was sent to 48 trainees (total number in our program at a given time), and 24 (50%) responded (year 1: n = 12; year 2: n = 6; year 3: n = 6). Sixteen of 24 (67%) trainees reported that receiving a prompt at the beginning of a shift specifying the content area for assessment made it easier to focus on it during the shift, and 14 (58%) felt the end of shift prompt was helpful to remind them to approach the attending for feedback. Nineteen of 24 (79%) would recommend the mESA program to other residencies. Other feedback included supportive statements about improvements relative to paper cards (n = 3) and concern that content areas may be too specific and place barriers to feedback in other areas (n = 2). Text message feedback from trainees confirmed that in-person ESA feedback was provided 90% of the time.

After approximately 1 year, 7 residency leaders (4 program directors and 3 core education faculty) were surveyed via email to provide feedback on mESA. This was solely intended to collect descriptive feedback from core leadership. They universally agreed that the mESA provided added value to the quality of assessment, primarily through improved face-to-face feedback at shift end, improved feedback at the completion of a rotation, and summary assessments for the clinical competency committee (box 2).

Between January 3, 2018 and November 4, 2019, mESA prompts were sent on 7792 unique shifts. Total number of prompts received by faculty varied from 2 to 401 (mean = 81 prompts per faculty member). Faculty submitted online ESA feedback of trainees on 3526 unique shifts (overall ESA submission rate 45%). Forty percent of ESA were prompted between 7am and 7pm, 34% were prompted between 7pm and 12am and 27% were prompted between 12am and 7am. When examining the mESA completion rates by time of day, 54% were missing when prompted between 7am and 7pm, 50% when prompted between 7pm and 12am, and 56% when prompted between 12am and 7am (chi-square test; P < .0001).

Most mESAs were prompted at site 1 (70%), with 14% and 12% at sites 2 and 3 respectively, and 5% at site 4, due to the higher number of resident shifts at site 1 during the study period and because that site was the first to utilize the system. We invested approximately $20,000 to build, implement, and maintain the system. We did not identify any technological errors, such as erroneous, missing, or delayed messages. We did not identify any delays in texts being received. One technological issue that came up during implementation was how to handle situations where a faculty was working with 2 trainees on the same shift for the same amount of time. We initially solved this by randomly choosing only one of the trainees. A later version of the program now identifies faculty-trainee dyads by specific shift name. A second mitigated issue involved asking enrollees to alert us if their phone number changed and creating an alert system if text messages are not delivered. Shift schedule changes were handled by building a program interface for the administrator to manually document such alterations.

Figure 2 depicts completion rates, which stabilized around month 6 of 22. There was no evidence that the number of total prompts received by a faculty member was associated with increased missing web-based ESA submissions (correlation coefficients non-significant; figure 3).

This is the first known study to lend support to use of an electronic ESA tool in an EM residency. Utilizing the RE-AIM framework, we found evidence of reach, effectiveness, implementation quality, and maintenance for our intervention, but suboptimal adoption.

Although our study focused on EM, findings are relevant to most specialties utilizing shift work. Assessment tools are most useful when they are acceptable to trainees and faculty, reliable, valid, cost-effective, and can impact future practice.¹²  For training leadership, ESA is a practical and effective means to help trainees improve¹³  and correlates with clinical competency committee's decisions about trainees' milestone proficiency.¹⁴  For trainees, it strengthens the apprenticeship model, wherein trainees work closely with more senior physicians.¹⁵  While residents model their trainers, their personal growth is greatly augmented by directed feedback on clinical performance. Such feedback can be best incorporated if it is specific, timely, and actionable. For administration, mESA reduces burdens by centralizing collection of feedback.¹⁶ 

Our finding of good mESA reach may be due to a combination of perceived need for improving the process of ESA, the willingness to try technological solutions, and the process of opt-out (as opposed to opt-in) enrollment. In support of effectiveness, we found that the majority of faculty and trainees sampled approved of the mESA tool. Feedback from residency leadership largely echoed what was provided by site faculty and residents and added that the database of mESA submissions provides value by making milestone and end-of-rotation summary assessments more data-driven. The mESA tool was stable and did not crash or lose data during implementation, providing evidence of program quality.

The 45% of web-based submissions by faculty was lower than other studies of paper ESAs,¹⁷  but higher than the historic submission rates at our sites (estimated to be around 30%). We designed the mESA program with a focus on descriptive (not quantitative) feedback, which has a higher burden to complete for faculty, but potentially results in more useful information for training leadership. Additionally, residents reported high rates of real-time feedback, exceeding the completion rate of web-based assessments. This demonstrates that the mESA prompted real-time discussions about performance for formative assessment during the pilot. We included monthly feedback to faculty on the percentage of ESA submissions completed in hopes of influencing future submission rates. Lower rates of participation overnight may have been due to fatigue at the end of those shifts. It may be reasonable to utilize alternative methods of data collection based on time of day such that overnight shift workers are prompted after sleeping to complete mESAs.

A limitation of the study is a lack of control or comparison group. Secondly, we were unable to recruit individuals who never participated in mESA to participate in a descriptive interview, limiting our understanding of the barriers influencing initial engagement and participation. Furthermore, as the surveys did not have prior evidence of validity, respondents may have interpreted questions differently than intended.

Future mESA design considerations should include departmental incentives tied to completion or reducing the frequency of mESA. We noted clusters of low responders among the faculty and found that certain faculty members are simply not comfortable with smartphone technology.

Although it is likely inevitable that certain faculty members will have poor participation regardless of the tool used, further research is needed to determine the reasons for the low rates. Because our assessment system is web-based, future iterations of the program could include sending prompts via email or other means to faculty without smartphones. Design improvements could include limiting prompts to shift-start for trainees and shift-end for faculty. Finally, future studies examining the relationship between ESA feedback and other measures of trainee performance would be beneficial.

We found that a mobile smartphone ESA program reached its intended users, was found to be effective by stakeholders, and was adopted at moderate and stable rates over time.