Abstract
Effective communication during patient care transitions is essential for high-quality patient care.
The purpose of this study was (1) to objectively assess patient handoff skills of internal medicine residents, and (2) to evaluate correlations between clinical experience and patient handoff skill self-assessment with directly observed skill.
We studied simulated patient handoffs in postgraduate year (PGY)-1 and PGY-2 residents between July 2011 and September 2011, using a standardized scenario in an observed structured handoff exam (OSHE). Our design was a posttest-only, with nonequivalent groups. Assessment used a previously published checklist for evaluating handoff skills. Residents were asked about clinical experience with patient handoffs and about their self-confidence in performing a patient handoff independently. We evaluated between-group differences on OSHE checklist performance, patient handoff experience, and self-confidence and used multiple regression analyses to assess the association between performance, experience, and confidence.
Forty-seven PGY-1 residents and 38 PGY-2 residents completed the study. Interrater reliability was substantial (intraclass correlation = 0.68). There was no significant difference in OSHE performance by PGY-1 residents (mean = 79%, SD = 4.6) and PGY-2 residents (mean = 82%; SD = 7.6; P = .07). The PGY-2 residents were significantly more experienced (P < .001) and confident (P < .001) than PGY-1 residents were, yet clinical experience and self-confidence did not significantly predict OSHE performance.
Clinical experience and self-assessment do not predict skills in simulated patient handoffs, and residents with substantial clinical experience still benefit from further skills development.
Editor's Note: The online version of this article contains an appendix of key concepts in transitions of care and inpatient handoffs.
Patient handoff skills are critical to patient safety in the inpatient setting.
A study of the relationship between self-assessments and direct observation of internal medicine residents' handoff skills found second-year residents more confident than interns. The added year of training did not produce improvements in objectively assessed handoff skills.
Single-site study, small sample, assessment of handoff skills confined to 1 case, self-assessment instrument with limited validity.
Clinical experience and self-assessment do not predict handoff skills, and even residents with substantial clinical experience may benefit from further formal skills development.
Introduction
Effective communication during patient care transitions is essential for high-quality patient care.1,2 Inpatient handoffs are frequent events, and poor communication is known to be a major source of patient risk.3 The Accreditation Council for Graduate Medical Education (ACGME) requires that residency programs monitor the quality of patient handoff skills among trainees.4 Several recent articles have proposed conceptual frameworks for teaching and assessing these skills.5–9 They call for structured training and the use of rigorous assessment measures to reliably assess patient handoff skills.
Earlier research about patient handoffs primarily concerned system factors and process standardization. Examples include templates10 and mnemonics.11 Research on patient handoff skill acquisition and retention remains sparse. Traditionally, residents learned to perform patient handoffs using the apprenticeship “see one, do one, teach one” approach, with this model assuming that experienced trainees have greater skills. However, research shows that experience is not a proxy for skill in general health care delivery,12 and gaps exist even in basic skills, such as maintenance of aseptic technique.13,14 The belief that additional clinical experience during residency improves handoff skills remains untested.
Little is known about patient handoff skill assessment.5–7 A systematic review identified 18 studies about resident patient handoff skills, yet none included direct observation of patient handoffs or a reliable communication skills assessment.15 Although there has been an increase in the development and use of more objective assessment methods,16–19 a recent systematic review of 10 educational interventions in patient handoffs found 9 primarily used participant self-report to assess the quality of the intervention and its effect on handoff skills.20 Self-reporting is an acceptable outcome measure used in many high-quality studies.21 However, reliance on self-confidence to assess clinical skills is not optimal because that approach is inaccurate.22,23
Reliable assessment of each trainee's ability to perform patient handoff is a critical part of a residency assessment system, and our study had 2 aims: (1) to objectively assess patient handoff skills by internal medicine residents, and (2) to evaluate correlations between clinical experience and self-assessment of patient handoff skill with directly observed skill.
Methods
Study Design and Setting
We studied patient handoff skills among internal medicine residents at Northwestern University, an urban university-affiliated program from July 2011 through September 2011. The study used a posttest-only design with nonequivalent groups. Residents rotate at 2 sites where inpatient clinical experiences on general medicine wards and subspecialty units involve twice-daily patient handoffs from day to night teams (7:00 am and 7:00 pm). Patient handoffs are performed and received by postgraduate year (PGY)-1 residents and include in-person, verbal communication and review of a printed sign-out form generated from the electronic medical record. There is no formal policy on supervision of daily patient care handoffs by more senior trainees.
The Northwestern University Institutional Review Board approved the study, and participants provided informed consent before skills assessment.
Participants
All 47 categorical and preliminary PGY-1 and 38 categorical PGY-2 internal medicine residents were eligible to participate in July 2011. All residents completed a formal, standardized handoff curriculum during their first month of PGY-1 training. The PGY-1 residents received a 2-hour curriculum in July 2011, which included didactic content, readings, and web-based case studies. Topics included inpatient, discharge, and verbal and written patient handoffs. The PGY-2 residents received a 4-hour curriculum using the same topics and materials in July 2010.
Procedure
The PGY-1 and PGY-2 residents were asked to perform a simulated patient handoff using a previously published, standardized scenario in an observed structured handoff exam (OSHE).23 The OSHE included a previously written admission history and physical with an assessment and plan about a patient admitted to a general medicine floor with a diagnosis of community-acquired pneumonia. Residents reviewed the written history and physical individually and reviewed a written clinical update and video describing a change in the patient's condition. Each resident was given a written sign-out form prepopulated with demographic information, allergies, code status, and medications. Four sections of the sign-out form were left blank: (1) a brief clinical history, (2) the hospital course, (3) the active problem list, and (4) a “To Do” list. After reviewing the documents and video, each resident was given 15 minutes to complete the 4 blank sections of the written sign-out form. Once the written form was complete, residents performed a verbal patient handoff to 1 of 2 faculty receivers with expertise in patient handoff skills (A.D., M.K.), capped at 5 minutes, and faculty who received the handoff provided 10 minutes of individual feedback. All handoffs used the same scenario, and were audio-recorded.
Outcomes and Measurement
Skills were assessed by the primary faculty raters using the Handoff Clinical Evaluation Examination (CEX), a published checklist developed in conjunction with the OSHE.24 The Handoff CEX rates handoff quality using 7 items, including domains of organization/efficiency, communication skill, content, clinical judgment, humanistic qualities/professionalism, setting, and overall sign-out competence. Items were scored from 1 (unsatisfactory) to 9 (superior) and were averaged for a total score. Validity evidence for using the assessment tool include its content, prior response by resident physicians, and use in a variety of health care settings where patient handoffs occur.5,7,10,24–26 Performance was calculated as the percentage of the total score out of the total possible score. A 40% random sample of audio recordings was rescored by a second rater using the same checklist to assess interrater reliability. This rater was blind to the residents' name, PGY status, and first score.
Demographic information (age, sex, medical school attended, United States Medical Licensing Examination [USMLE] steps 1 and 2 scores) was collected. Residents were asked about actual clinical experience with patient handoffs (number performed in actual clinical care), and their self-confidence in performing a patient handoff independently on a scale of 0 (not confident) to 100 (extremely confident).
Primary outcome measures were checklist scores on the Handoff CEX by PGY-1 and PGY-2 residents. Secondary outcome measures were associations of patient handoff OSHE performance and resident clinical experience and self-confidence.
Data Analysis
Demographic differences between PGY-1 and PGY-2 residents were assessed using Pearson's χ2 test and 2-tailed, independent-sample t tests. Between-group differences on OSHE checklist performance, patient handoff experience, and self-confidence were evaluated using 2-tailed, independent-sample Mann-Whitney U tests. Interrater reliability was assessed using the intraclass correlation coefficient across all checklist items. Multiple regression analysis was used to assess the association between OSHE checklist performance, self-confidence, and patient handoff experience. All analyses were performed using IBM SPSS Statistics software, version 20.0 (SPSS Inc, Chicago, IL).
Results
All 47 PGY-1 residents (100%) and 38 PGY-2 residents (100%) consented to participate and completed the entire protocol. There was no significant difference in age, sex, or USMLE step 1 or 2 scores between groups.
Interrater reliability was substantial across the 7-item Handoff CEX checklist (intraclass correlation = 0.68). figure 1 displays PGY-1 and PGY-2 resident performance on the OSHE. There was no significant difference between overall OSHE performance of PGY-1 residents (mean = 79%, SD = 4.6) and PGY-2 residents (mean = 82%; SD = 7.6; P = .07). When assessed at the individual level, only 1 of 7 checklist items (14%)—organization—differed significantly, favoring PGY-2 residents (P = .002).
figures 2 and 3 show residents' clinical experience (mean number of handoffs performed) and self-confidence to perform patient handoffs by postgraduate year. The PGY-2 residents were significantly more experienced (P < .001) and confident (P < .001) than were PGY-1 residents.
When controlling for age, sex, PGY status, and USMLE step 1 and 2 scores, patient handoff clinical experience did not significantly predict OSHE performance. Self-confidence related to handoffs also did not predict performance when controlling for the same variables.
Discussion
We developed a multimodal curriculum in patient handoffs, including didactic content, self-study, and online modules. We studied the reliability of data obtained from an assessment of residents' patient handoff skills using the OSHE and Handoff CEX. We found the Handoff CEX easy to use and to incorporate into our residency evaluation system. All residents were able to complete the assessment in the allotted time. Required faculty time was reasonable because it took approximately 15 minutes to administer the Handoff CEX and to provide individualized feedback. Data obtained from the Handoff CEX also showed substantial reliability between examiners. This finding is important because obtaining a reliable assessment of patient handoff skills in a simulated setting is a first step toward ensuring trainee competence in actual clinical care.
Surprisingly, PGY-1 and PGY-2 residents performed at a similar level on an objective communication skills assessment. These findings challenge traditional training models that assume additional clinical experience improves patient handoff skills. The PGY-2 residents in this study had much more clinical experience than the PGY-1 residents did, yet they performed similarly on the handoff OSHE. An earlier observational study of critical care physicians also showed poor handoff communication skills in experienced clinicians.27 These results are concerning if senior residents are expected to supervise actual patient handoffs performed by PGY-1 residents as suggested by other investigators.5
Our findings also suggest that PGY-1 and PGY-2 residents were not able to accurately self-assess their handoff skills, with no correlation between objectively measured handoff skill and residents' self-confidence. One prior observational study of pediatrics residents showed self-assessed effectiveness of a handoff was higher than were ratings from the handoff receiver.28 Our study advances what is known about patient handoff skills by showing the limitations of self-assessment of professional competence as compared with an independent, objective evaluation.12,13,22,23,29,30
Effective communication skills are a critical component of high-quality patient care, and our findings raise several questions. What are the best methods to teach and assess patient handoff skills? What is the minimum acceptable skill level for residents at various levels of training? As shown in this study and others, reliance on clinical experience alone is insufficient to achieve competence.31 Use of the mastery model is 1 way to boost clinical skills to high achievement levels. In mastery learning, training time varies, whereas education outcomes are uniform.32 Mastery learning programs feature deliberate practice,33 predetermined minimum passing standards, and periodic objective assessment. This format allows residency programs to accurately document skill achievement and retention. Downstream improvement in the quality of patient care has also been demonstrated.34,35 All residents at our institution now complete a mastery learning program in patient handoff skills. Residents receive handoff education and must meet a minimum passing standard set by an expert panel on the simulated assessment before performing handoffs in actual clinical care. Additional instruction and feedback are available as needed for remediation.
Our study has several limitations, including being conducted at a single, university-based program. Only 1 case was assessed during the OSHE, and it is not known how participants would perform in handoffs of multiple patients as is common in real clinical settings. The PGY-1 residents completed a 2-hour web-based curriculum on transitions of care before assessment. The PGY-2 residents completed a similar but more extensive 4-hour curriculum 1 year earlier, and the effect of PGY-1 residents' more recent completion of a curriculum on OSHE performance is unknown. Self-confidence was assessed using a survey instrument with limited validity evidence, and lack of pretest Handoff CEX scores does not allow a full evaluation of the effect of the intervention. Finally, we did not link patient handoff skills displayed in a standardized setting to performance in actual clinical care or patient outcomes.
Conclusion
In conclusion, use of the OSHE and Handoff CEX is feasible and produces reliable data. Clinical experience and self-confidence do not predict performance of simulated patient handoffs. Future studies are needed to link improved skills in simulated patient handoffs to better patient outcomes.
References
Author notes
All authors are at Feinberg School of Medicine, Northwestern University. Aashish Didwania, MD, is Assistant Professor in Medicine and Internal Medicine Residency Director; Michael Kriss, MD, is Chief Medical Resident in Medicine; Elaine R. Cohen, MEd, is Research Assistant in Medicine; William C. McGaghie, PhD, is the Jacob R. Suker, MD, Professor of Medical Education in the Center for Education in Medicine; and Diane B.Wayne, MD, is the Dr John S. Appleman Professor and Vice Chair of Education in Medicine.
Funding: Northwestern Memorial Hospital and Excellence in Academic Medicine Act by the Illinois Department of Healthcare and Family Services supported this research. This research was also supported in part by the Jacob R. Suker, MD, professorship in medical education and by grant UL 1 RR025741 from the National Center for Research Resources, National Institutes of Health (NIH). The NIH had no role in the preparation, review, or approval of the manuscript.
The authors wish to thank Douglas E. Vaughan, MD, for his support and encouragement of this work and the Northwestern Memorial Hospital internal medicine residents for their dedication to patient care and education.