Residents are expected to assume the role of teacher and educate their patients, junior colleagues, and medical students. This expectation has been highlighted in the Accreditation Council for Graduate Medical Education milestones. Institutions have responded by focusing on the evaluation of residents' communication and teaching skills, and by implementing residents as teachers programs. While the American Board of Internal Medicine recommends a patient satisfaction questionnaire to assess residents' patient education skills, this method has little validity evidence and does not provide contemporaneous feedback. Observational assessment tools to measure patients' perspectives of the education provided by residents are lacking. Our objective was to develop observational assessment tools that could be used to evaluate patient education provided by internal medicine residents. We examined evidence for the validity and reliability of these tools.

For the patient assessment tool, we adapted 15 questions from the Agency for Healthcare Research and Quality Consumer Assessment of Healthcare Providers and Systems (CAHPS) clinician and group surveys that assess patients' experiences with their providers. We selected items from the health literacy and communication domains that addressed the ability to educate patients. The CAHPS surveys are psychometrically tested, are used to provide benchmark data on patient experiences, and ask whether the provider used explanations and instructions that were understandable and whether he or she listened, showed interest, addressed barriers to proposed plans, as well as answered patients' questions.

We then developed a faculty observational assessment tool using a set of prompts that mirrored the selected CAHPS patient assessment tool items. The development of the faculty assessment tool was guided by Messick's validity framework. Behavioral anchors were used for each prompt to improve the reliability of assessments. The observational tool also allowed for free-text comments and feedback on areas of deficiencies. Beta-testing and revision of the instrument occurred over a 3-month period after review by experts in medical education, statisticians, and clinic preceptors. Core educational faculty underwent 1 hour of rater training using the assessment tool. Faculty interrater reliability was calculated for the finalized tool. There was substantial overall agreement between faculty members using the observational assessment tool (Fleiss κ = 0.61). The tool is provided as online supplemental material.

At the end of academic year 2014, faculty at 2 institutions assessed the performance of residents educating patients in the inpatient ward or clinic setting. During the same encounter, patients rated the residents' patient education skills using the adapted CAHPS questionnaire. The faculty and patient assessments were analyzed for correlation between responses.

In the pilot, 7 faculty members evaluated 18 of 44 eligible residents (41%). A total of 10 of 18 observations (56%) occurred on the inpatient ward and 8 (44%) in the clinic. The majority who were evaluated were postgraduate year (PGY) 1s (56%, 10 of 18), and 28% (5 of 18) and 17% (3 of 18) were PGY-2s and PGY-3s, respectively. Residents and faculty reported that the time to complete the observation and provide feedback was similar to completing a mini-clinical evaluation exercise. Rating distributions by faculty and patients spanned the entire scale. To evaluate the predictive validity of the faculty observational assessment tool, the Pearson correlation of responses with the patient survey was calculated. There was a strong positive association between faculty and patient ratings of residents. The correlation between faculty's and patients' observational assessments was 0.59 (P = .026).

This pilot project provides preliminary evidence of the validity of tools to assess residents' skills in delivering patient education. The value of the instrument is that it can be used to tailor feedback to residents. A unique feature of the tool is the correlation between patients' perceptions and faculty's assessments of residents' patient education skills. Future research will extend these observations to include more residents across multiple sites over a longer period of time.