ABSTRACT
Subspecialty consultation in inpatient medicine is increasing, and enhancing performance of consultation services may have a broad-reaching impact. Multisource feedback is an important tool in assessing competence and improving performance. A mechanism for primary team resident feedback on performance of consult services has not been described.
We developed and evaluated an instrument designed to assess internal medicine (IM) subspecialty inpatient consult service performance. We hypothesized that the instrument would be feasible to administer and provide important information to fellowship directors.
The instrument was administered in 2015 and 2016 at a single academic center. All IM residents were invited to evaluate 10 IM subspecialty consult services on 4 items and an overall satisfaction rating. The instrument allowed for free-text feedback to fellows. Program directors completed another survey assessing the impact of the consult service evaluation.
A total of 113 residents responded (47 in 2015 and 66 in 2016, for a combined response rate of 35%). Each of the 4 items measured (communication, professionalism, teaching, and pushback) correlated significantly with the overall satisfaction rating in univariate and multivariate analyses. There were no differences in ratings across postgraduate year or year of administration. There was considerable variation in ratings among the services evaluated. The 7 program directors who provided feedback found the survey useful and made programmatic changes following evaluation implementation.
A primary team resident evaluation of inpatient medicine subspecialty consult services is feasible, provides valuable information, and is associated with changes in consult service structure and curricula.
Introduction
The role of subspecialty consultation in inpatient medicine is increasing.1,2 Within academic medical centers, consult interactions frequently take place between residents and subspecialty fellows. Fellows may have a significant positive impact on patient care and the trainees they interact with, including affecting resident education and career choice.3–5 Therefore, enhancing the performance of subspecialty consultation services can have a broad-reaching impact.
Multisource feedback is a critical element of assessing competence and improving performance.6,7 While residents may provide feedback to fellows when they rotate on subspecialty services, to our knowledge, a mechanism for feedback from primary team residents on consult service performance has not been described. Obtaining such feedback faces logistical challenges due to the number of evaluations that would require completion and difficulty in systematically capturing data from resident-fellow dyads. Here, we describe the development and implementation of an instrument designed to assess consult services as a whole, with an option for individual fellow feedback. We hypothesized that the instrument would be feasible to administer and provide important information to fellowship program directors (PDs).
Methods
Setting and Participants
The evaluation was developed and administered annually to all residents in the Massachusetts General Hospital Department of Medicine beginning in April 2015. Internal medicine subspecialty PDs were invited to complete a survey examining the impact of the instrument on their fellowship programs in February 2017.
Instrument Development
To develop this instrument, the authors first reviewed the literature on multisource feedback, which highlighted 5 domains that were assessed across previously utilized instruments for physician performance: professionalism, clinical competence, communication, management skills, and interpersonal relationships.7 Studies examining resident-fellow interactions also highlighted pushback (reluctance to perform consultations) and teaching of residents as important elements of interactions.4,8,9
Six medicine subspecialty fellowship directors participated in instrument development via e-mail and/or a face-to-face meeting. Communication, professionalism, teaching, and pushback domains were considered most important. An overall satisfaction rating and consultation frequency were also assessed. Overall satisfaction has been postulated to be an important measure of consult service effectiveness.10 The instrument evaluated 10 medicine subspecialty consult services and contained 60 questions rated on a 5-point scale, with optional free text for the evaluation of individual fellows (provided as online supplemental material). The survey asked residents to consider only their interactions with consult services on inpatient primary teams. The instrument was piloted in May 2014 and revised before administration in 2015 and 2016 via a web-based survey tool (Qualtrics LLC, Provo, UT).
Result Reporting
Fellowship directors received annual deidentified results for all consult services with only their own service identified. In February 2017, this group was invited to complete an anonymous survey assessing the impact of the evaluation (provided as online supplemental material). The survey was developed by 1 author (E.M.M.) and revised based on cognitive interviewing performed with faculty not participating in the survey.
The study was approved by the Partners Institutional Review Board.
Statistics
Pearson correlation coefficients (r) were used to assess the relationship between each domain and the overall satisfaction score. Multivariable linear regression models with generalized estimating equations were used to examine the effects of consult service, postgraduate year (PGY), frequency of consultation, and year of survey administration, using SAS version 9.4 (SAS Institute Inc, Cary, NC).
Results
A total of 113 residents responded (47 of 162 in 2015, 29% response rate; 66 of 161 in 2016, 41% response rate). Respondents included 45 PGY-1, 35 PGY-2, 32 PGY-3 or PGY-4, and 1 unknown. Each domain showed moderate to strong correlation with the overall satisfaction rating (all correlation coefficients significantly different from zero with P < .0001; table 1). Free-text responses evaluating specific fellows increased from 45 in 2015 (median = 4 per service; range, 2–8) to 126 in 2016 (median = 12.5 per service; range, 5–20).
Each of the 10 services was rated differently by residents across all domains (all with P < .001; table 2). In multivariable analyses, only the specific service predicted domain and overall satisfaction scores. Higher frequency of consultation correlated with more positive overall satisfaction but had no effect on other domains. Responder PGY or year of survey administration did not affect ratings.
Seven of 10 fellowship directors completed the survey assessing consult evaluation impact. All found the evaluation useful, and all made changes to their programs following its administration. Five enhanced the fellowship curriculum to address resident-fellow interactions and fellow teaching skills, 1 redesigned the consult triage process, and 1 added a physician extender to facilitate team communication. All used the evaluation results in fellow feedback.
Discussion
An annual web-based resident assessment of inpatient consults performed by 10 internal medicine subspecialties was completed by less than half of residents but was associated with several fellowship program changes and was highly acceptable to fellowship directors. Each of the 4 subsections of the assessment tool (communication, professionalism, teaching, and pushback) correlated with the overall satisfaction score.
Our results support the hypothesis that feedback on consult service performance from primary medical teams is feasible to obtain and valued by fellowship PDs. Such feedback may differ significantly from the feedback residents may provide when they rotate on consult services where resident-fellow incentives and schedules are aligned, facilitating positive interactions.4 Feasibility of administration was made possible by evaluating consult services as a whole and using a limited number of questions to increase the response rate. Despite challenges in surveying residents,11 the response rate increased by 40%, and the number of specific fellow comments nearly tripled from 2015 to 2016. Fellowship directors utilized the survey results in fellow feedback and made changes in curricula and consult service structure, suggesting that the instrument can facilitate change at the fellowship program level. One potential mechanism is that enabling fellowship directors to see results of other fellowship programs within the institution facilitated the sharing of best practices.
Year to year, fellow variation and frequency of consultation did not have a major impact on resident perception of consult services. One explanation is that subspecialty faculty, emphasis on teaching within the division, consult service structure (fellow workload, communication mechanisms, rounding structure), or reputation affect resident perception.4 Our findings suggest that efforts to improve consult performance should address these factors in addition to fellow-specific skills.
Our study has several limitations. We could not account for the possibility that experiences on consult service rotations affected resident responses. However, responses did not differ by PGY, and this effect may be limited. Our responses may have been limited by recall bias and recent experiences. The acceptability and feasibility of this approach may be different in other programs. Finally, our study did not assess whether use of the assessment instrument produced actual improvements in fellow consult performance.
Next research steps could include comparisons of fellow consult performance over time in programs using this instrument compared with other feedback mechanisms. In addition, future research could explore whether this instrument could be utilized as an outcome measure in studies examining other interventions aimed at improving consult service performance.
Conclusion
Our web-based instrument (designed to assess internal medicine subspecialty consult service performance) was feasible to implement in a large internal medicine residency program. Fellowship directors reported using the tool to provide feedback to fellows and make changes to fellowship curricula.
References
Author notes
Funding: This work was supported by a grant that Dr Miloslavsky received from the Massachusetts General Hospital Center for Educational Innovation and Scholarship.
Competing Interests
Conflict of interest: The authors declare they have no competing interests.
Preliminary findings of this work were presented at the American College of Rheumatology Annual Meeting, Washington, DC, November 11–16, 2016.
The authors would like to thank Dr Beverly Biller, Professor of Medicine, Harvard Medical School, Department of Medicine, Massachusetts General Hospital, for her thoughtful contributions to the design and implementation of the consult service evaluation.
Editor's Note: The online version of this article contains the consult service evaluation and the program director survey.