ABSTRACT
Residencies have incorporated high-value care (HVC) training to contain health care expenditures. Assessment methods of HVC curricula are limited.
In our clinical skills laboratory, we evaluated the effectiveness of HVC curricula using standardized patients (SPs) to determine if there is a correlation with performance in counseling, history and physical, HVC knowledge, and demographics.
Through ambulatory cases, SPs evaluated postgraduate year 2 (PGY-2) residents using checklists to determine if they obtained the chief complaint, medical and social history, focused physical examination, and conveyed information regarding patient management. Investigators scored knowledge-based questions on the need for imaging in low back pain, annual stress testing in coronary artery disease, and chest x-ray for gastroesophageal reflux disease. Univariate analysis was used to calculate percentage distribution of residents' ordering of inappropriate tests.
All 56 PGY-2 residents participated in the study and completed at least 2 of 3 HVC cases. Analysis showed that 48% (27 of 56) ordered at least 1 inappropriate test. Residents who ordered unnecessary testing had similar performance in history and physical as well as knowledge of HVC. Inappropriate ordering was significantly associated with poorer performance in counseling (mean percentage counseling score of 68% versus 56% for those who ordered inappropriately, P < .001) and communication skills (mean percentage communication score of 74% versus 71% for those who ordered inappropriately, P < .003). There were no patterns for ordering by demographics.
Our evaluation of residents during SP encounters found a correlation between the use of inappropriate testing and lower counseling and communication skills.
Residency programs seek to teach high-value care (HVC), yet assessment of these HVC curricula is limited.
A study evaluated delivery of HVC using standardized patients, and assessed the relationship between HVC knowledge, demographics, and communication and counseling skills.
Single site, single specialty study reduces generalizability.
There is a correlation between the use of inappropriate testing and lower counseling and communication skills.
Introduction
Health care expenditures in the United States are estimated to account for 18% of the gross domestic product and are growing faster than the economy.1 Several investigators concluded that of the $750 billion health care dollars wasted per year, $210 billion comes from unnecessary testing.2
To teach residents to be stewards of limited health care resources, many residency programs have incorporated high-value care (HVC) into their curriculum. Internal medicine program directors surveyed over 3 years reported high interest in HVC and improvement in trainees avoiding unnecessary testing.3 Three themes in teaching HVC have emerged to date: knowledge transmission, reflective practice, and a supportive environment.4 The continued development of meaningful assessment tools may further help the goal of translating knowledge into practice.5
In an effort to measure delivery of HVC through the use of inappropriate testing, we evaluated simulated standardized patient (SP) encounters in our clinical skills laboratory and examined if there was a correlation between resident performances in regards to counseling, history taking and physical examination, HVC knowledge, and demographics.
Methods
The University of Connecticut Internal Medicine Residency developed a curriculum that included didactics based on the American College of Physicians' HVC series. As part of this curriculum, interns participate in three 2-hour didactic sessions. The HVC concepts are also reinforced twice annually during resident-led HVC case presentations covering a variety of topics.
To assess the impact of our HVC curriculum, the 2015 postgraduate year 2 (PGY-2) ambulatory case series utilized SPs in the clinical skills lab, which was modified to include items related to the practice of HVC. Specifically, low back pain (LBP), coronary artery disease (CAD), and gastroesophageal reflux disease (GERD) cases were used to assess residents' ability to counsel patients on and deliver HVC.
The PGY-2 residents were provided written information to frame their visit prior to the encounter. They had 20 minutes with the SP. The SPs were instructed to ask the residents, if not already mentioned, about the need for imaging in nonspecific LBP, annual stress testing in stable CAD, and a chest x-ray for a patient with GERD. Immediately following, residents were given 5 minutes to complete a postencounter note consisting of knowledge-based questions related to the HVC relevant to the case. Concurrently, SPs completed checklists of predetermined elements that a resident should have gathered from, or communicated to, the SP. Program faculty observed the encounters via video to complete a milestone-based assessment. Clinical skills faculty also observed to ensure SP consistency. There was 1 SP for each clinical scenario.
The history and physical checklists ranged from 15 to 25 items that assessed residents' skill in collecting information regarding the chief complaint as well as their medical and social histories. Counseling checklists ranged from 7 to 12 items and assessed whether the resident conveyed information regarding management. The HVC items were included on these checklists. Checklists were scored as yes (correct) or no (incorrect). The summary score for each case was a percentage of correct answers. Since ordering information was the primary outcome, we excluded the ordering item from the counseling score during our analysis.
Communication skills were measured during each encounter using the Master Interview Rating Scale, a 5-point Likert scale using weighted descriptors of scores 1, 3, and 5.6
Two physician investigators (J.D.B. and J.C.), following predetermined criteria regarding the need for testing, independently and blindly scored the postencounter note questions as correct or incorrect. When discrepancies between the 2 raters occurred, final decisions were made by senior physician investigators.
We gathered information for each resident, including age, sex, medical school type, age at and years since graduation, and PGY-1 and PGY-2 in-training examination scores.
The University of Connecticut Institutional Review Board deemed this study exempt.
Our outcome measure was the ordering of inappropriate tests in an ambulatory case, designed to evaluate the utilization of HVC principles, as the scenarios did not warrant testing. During each encounter, SPs evaluated residents' performance in history and physical, counseling as well as communication. This information was used to determine if a correlation existed with the ordering of inappropriate tests. Univariate analysis presented percentage distribution of residents' ordering, including imaging for LBP, stress testing in CAD, and chest x-ray in a patient with GERD. To compare group differences of ordering status (inappropriate versus appropriate), chi-square testing was used for categorical explanatory variables, such as sex and medical school type. Student's t test was used for the continuous variables, such as age, years since graduation, PGY-1 or PGY-2 in-training examination scores, and all performance variables. Significance levels were determined and reported as P < .05 for both tests.
Results
All 56 PGY-2 residents participated, completing at least 2 of the 3 cases. Table 1 compares characteristics of residents who ordered to those who did not. No significant patterns for ordering were found across demographic variables. Forty-eight percent (27 of 56) ordered at least 1 inappropriate test; specific cases had variable frequencies of ordering (figure).
Residents who ordered had similar performance in history and physical as well as knowledge regarding the appropriate use of tests. Inappropriate ordering was associated with poorer performance in counseling (a mean percentage counseling score of 68% versus 56% for those who ordered inappropriately, P < .001) and communication skills (a mean percentage communication score of 74% versus 71% for those who ordered inappropriately, P < .003; Table 2).
Discussion
Our study found an association between lower performance in counseling and communication skills with the practice of HVC without a correlation in knowledge. It is encouraging that more than half of the residents refrained from ordering unnecessary tests. Despite knowing that testing was unnecessary, residents with worse performance in the areas of counseling and communication may succumb more easily to pressure from SPs to order additional tests, limiting their ability to practice HVC. Our study suggests that residents require not only the knowledge of HVC principles, but also the skills to engage patients in shared decision-making (SDM).
An estimated 85% of LBP cases never have a definitive diagnosis. This uncertainty is difficult for both patients and learners.7 One explanation why providers order imaging despite knowledge of the guidelines is the belief that patients consider this good care.8 With increasing emphasis on SDM in graduate medical education, it can be expected that patients' beliefs may influence providers' ordering patterns.9 Engaging in SDM requires that providers communicate effectively and have the knowledge to guide the conversation.10 It is also important for physicians to recognize that their own values and perceptions may affect how they involve a patient in SDM.11 Preference misdiagnoses by physicians that misalign with patient preferences can lead to harm and increased cost for testing.12 The ability to understand patients' values and beliefs and guide the SDM process improves with experience.13
Training through patient encounters in clinical skills and role modeling during bedside rounds are potential ways to develop these skills. As noted in the meta-analysis by Stammen et al,4 reflection of ordering practices with specific feedback, as well as creating a supportive environment, was found to be an important aspect of translating knowledge into practice. Ongoing monitoring of our residents with real-time performance feedback from faculty acculturated and trained in HVC and SDM principles may be beneficial.4
This study has several limitations. Our findings of lower performance in counseling and communication associated with inappropriate testing was a posteriori hypothesis that emphasizes the exploratory nature of the study. Although a correlation was found, this study does not prove causation. Future interventional studies should explore if improving counseling or communication leads to better delivery of HVC. There may be additional barriers to applying HVC to clinical practice that were not identified. The checklists used were not evaluated for validity evidence. There is also the potential that counseling and communication skills are not independent of each other, which cannot be determined with this study. Our study is focused on PGY-2 residents from a single institution. A larger sample could aid in a more confident interpretation of the contribution of fixed characteristics. Finally, we lack understanding of how our faculty practices HVC and SDM and how this shapes our residents' clinical practice style.
To advance our residents' practice of HVC, we must now focus on how to improve their communication and counseling skills with an emphasis on SDM. This could be accomplished through the use of SP encounters in clinical skills with direct observation, real-time feedback, and analysis of their performance.
Conclusion
When assessing residents during simulated SP encounters, our study found no significant difference in low-value test ordering based on resident demographics, HVC knowledge, or performance in history and physical. However, there was a correlation between counseling and communication skills, and the utilization of inappropriate testing.
References
Author notes
Editor's Note: The online version of this article contains the master interview rating scale and 3 checklists for the high-value care cough, heart disease, and low back pain.
Funding: The authors report no external funding source for this study.
Competing Interests
Conflict of interest: The authors declare they have no competing interests.
This study was presented as a poster at the ACGME Annual Educational Conference, National Harbor, Maryland, February 25–28, 2016, and the Alliance for Academic Internal Medicine Skills Development Conference, National Harbor, Maryland, October 20–22, 2016.
The authors would like to thank Carol A. Pfeiffer, PhD, for her guidance in the development of this study utilizing the clinical skills laboratory.