Can Incoming Interns Be Entrusted to Recognize Medical Emergencies? Implementation of a Vignette-Based Cross-Cover Assessment

Interns must be able to accurately recognize urgent clinical situations and know when to notify a senior resident for assistance.^1,2  These skills are especially important during times of cross-cover, defined as caring for hospitalized patients when the primary team is absent.³  Cross-coverage responsibilities have increased in the past decade, in part due to the Accreditation Council for Graduate Medical Education (ACGME) work hour restrictions and a resultant increase in night float shifts.^4,5  Cross-cover is now pervasive, with one study estimating that a patient is cared for by a cross-covering physician for 61% of their hospital stay.⁶ 

During cross-cover, the intern often has reduced supervisory oversight and relies on information handed off from the primary team. To our knowledge there are no reliable assessment methods of cross-cover skills. This is a critical gap because failure of an intern to recognize a scenario that requires urgent evaluation risks patient safety. Further, there is an incomplete understanding of what factors influence cross-cover skills, rendering it difficult to develop preventative or remediation strategies.

We designed a vignette-based study to assess if graduating medical students appropriately determined clinical urgency of cross-cover scenarios. Our secondary aim was to determine if medical knowledge or confidence was associated with this ability.

Participants included 60 senior medical students enrolled in an internal medicine residency preparation course in the spring of 2020, just prior to graduation. Students had applied to an internal medicine categorical or preliminary year and had completed core clinical clerkships as well as 15 to 16 months of post-clerkship rotations, including 2 subinternships and 3 to 4 clinical electives. Experienced physicians consisting of 12 internal medicine senior medical residents and 16 hospitalist faculty were also invited to take the assessment. Experienced physicians were selected given high volume of cross-cover experience. Participation was consented.

Four clinical vignettes were presented as a handoff in the I-PASS format.⁷  Two vignettes were modified based on prior assessments created by the authors,^8,9  while the other 2 were created for this study. Each vignette was accompanied by 5 to 6 cross-cover scenarios representing a distinct clinical question (online supplementary data).

To demonstrate content validity and response process validity,¹⁰  experienced physicians were asked whether the content of the sign-out was realistic and contained sufficient information on a 5-point Likert scale (1=strongly disagree, 2=disagree, 3=neither disagree or agree, 4=agree, 5=strongly agree, N/A=I do not know). They were also asked about sign-out length (1=far too short, 2=too short, 3=similar in length, 4=too long, 5=far too long, N/A=I don't know).

For each scenario, respondents were asked about the action they would take, including whether they would evaluate the patient at the bedside. Students were asked whether they would notify their supervising resident (1=yes, within minutes; 2=yes, within hours; 3=no), and their level of agreement with the statement that they felt comfortable managing the scenario independently (1=strongly disagree, 2=disagree, 3=agree, 4=strongly agree). All respondents were asked to rate the urgency of each scenario (1=low, 2=moderate, 3=high), each carefully defined (Figure). Scenarios that ≥75% of experienced physicians agreed were of high clinical urgency were considered “emergent.” At the end of each scenario, respondents took a medical knowledge quiz created by the authors that was related to vignette content and presented in multiple-choice format, totaling 62 possible answer choices.

Scenarios that experts agreed (>75% consensus) were low risk were excluded from the analysis. All other scenarios were included. Retrospectively, student performance was placed in 1 of 3 groups based on ability to stratify clinical urgency: Group 1 included the quartile of students who underestimated urgency, group 2 included students in the second and third quartiles and who most accurately assessed urgency, and group 3 included the quartile who overestimated urgency. We compared differences in group means between the 3 urgency estimator categories for medical knowledge, action, and confidence using analysis of variance and post-hoc Tukey Honestly Significant Difference test.

This study was determined exempt by The University of Michigan Hospital's Institutional Review Board.

Fifty-eight students (96.7%) and 22 experienced physicians (78.6%) participated. Experienced physicians included 11 faculty members, 8 senior residents, and 3 chief residents. Experienced physicians agreed that content of the sign-out was realistic (4.18±0.80) and that its length was realistic (2.87±0.52). Three experienced physicians stated “N/A” regarding content and sign-out length. There was more variation in response regarding whether sufficient information was provided (3.4±1.1).

Seventeen scenarios were included in analysis with an average urgency rating of 2.39±0.15 from experienced physicians. Students in the lowest quartile rated these scenarios as lower urgency and also were less likely to notify their supervising resident in a timely manner and less likely to report evaluating a patient at bedside compared to those who overestimated urgency (Table).

There was no difference in medical knowledge score or comfort level managing the scenario independently among the 3 groups (Table). Experienced physicians performed better on the medical knowledge assessment compared to medical students (91.9±3.1% vs 82.3±5.3%, P<.001). Interestingly, the performance of experienced physicians was consistently high (ie, low within-group variation), whereas the student group was statistically different, providing further evidence of content validity.

In this study we show that senior medical students preparing to transition to residency vary in their abilities to risk stratify potentially urgent cross-cover scenarios. Students that underestimate urgency report they would be less likely to notify their supervising resident and evaluate a patient at bedside, which has implications for patient safety. Interestingly, a student's ability to recognize urgent clinical scenarios appears to be independent of medical knowledge.

The ability to accurately estimate urgency helps inform how residents should “triage” pages that they receive in the hospital—matching both timeliness of care and resources available.¹¹  Underestimating urgency can lead to a lack of appropriate timely diagnosis and treatment of potentially life-threatening conditions, especially in times of limited supervisory oversight such as cross-cover care. Overestimating urgency may lead to an increase in resources utilized, overuse of diagnostic testing, and perhaps unnecessary treatment.

It was surprising that quiz scores were not associated with ability to determine urgency, given that the questions were related to the vignette content. Our findings raise questions about the extensive use of medical knowledge tests to define competency prior to graduation and imply that more targeted assessments may be necessary prior to entrusting interns with cross-cover responsibilities. There is currently a national call for meaningful assessment data at the transition to residency¹² ; such data could provide important baseline information for the learner and allow for testing efficacy of interventions.

Limitations include that this was a single center study with a small number of participants, which limits generalizability. Additionally, the computerized examination does not represent the clinical setting where a provider may call the nurse for more information, look up information online, or ask a colleague for assistance. We did not ask students about prior cross-cover experience, which could have affected performance.

Next steps include broadening to more participants. Additionally, it would be interesting to compare performance to other available assessments including board examinations, clinical grades, or residency milestone data.

In this single center study, we demonstrate variability in graduating medical students' ability to estimate clinical urgency of cross-cover pages and appropriately notify supervisory residents.