Successful Implementation of a Direct Observation Program in an Ambulatory Block Rotation

Graduate medical education (GME) has undergone a dramatic shift in recent years, as accreditation bodies and educators have led the charge to transform physician training by focusing on educational outcomes rather than processes as the optimal means of ensuring physician competence. This approach, termed competency-based education and training,¹  has been reinforced by the Accreditation Council for Graduate Medical Education (ACGME) and its new accreditation system.² 

Direct observation of trainees is a cornerstone of competency-based education,³  and a prerequisite for resident assessment.⁴  Certain core competencies, such as interpersonal and communication skills and patient care, cannot be adequately assessed without direct observation. Direct observation has been shown to increase feedback frequency,⁵  identify clinical deficiencies otherwise unrecognized,^6,7  increase learners' confidence,⁸  and improve learners' communication skills.⁹  For these reasons, systematic direct observation of trainees' clinical skills is required by the Liaison Committee on Medical Education and the ACGME.^10,11 

Despite the central role and clear benefits of direct observation, its implementation in GME has proved challenging.^12–16  Studies indicate that little time is spent conducting direct observation during an average medical workday,¹²  and a substantial percentage of residents state that they have never been directly observed doing certain important clinical tasks.^14,15  Barriers to frequent use include limited faculty time, faculty attention required toward clinical tasks, trainee reluctance to be observed, and concerns about the impact on the patient-resident relationship.^12,13 

The objectives of this study were to increase direct observation of residents' history taking, physical examination, and counseling skills; increase the actionable feedback that residents receive; and reduce the natural reluctance of faculty to participate in direct observation.

We performed a single center cohort study to evaluate the effect of a direct observation assessment system on changes in resident direct observation and faculty attitudes toward the system.

Our program encompasses 77 categorical residents. Approximately 40 residents rotate through a university-based ambulatory block each year, and the other 37 rotate through a Veterans Affairs ambulatory block. In this block rotation, residents see internal medicine outpatients 5 days per week for 1 to 3 months, at 3 participant sites, involving 14 faculty supervisors. Each resident interacts with 2 to 6 faculty physicians during the rotation, in which we implemented the new direct observation system. Each resident was required to obtain 3 observations for each of 3 clinical domains (history taking, physical examination, and counseling/decision making), totaling 9 observations during the block. To reduce faculty time and increase feasibility, observations focused on 1 domain at a time. Each resident entering the block received a 2- to 3-minute description of program goals and logistics via a phone call from a faculty investigator (J.S.), who then e-mailed the assessment forms, along with written reinforcement of the process.

Residents worked with their supervising faculty to select appropriate patients for observation; they gave their assessment form to the faculty prior to each observation, and also collected the form afterward. During the observation, faculty could opt to simply observe throughout, or could offer additional teaching and/or feedback. A reminder e-mail was sent midway through the rotation, and a final e-mail was sent at rotation end to collect the forms. Total time spent per month by the faculty investigator was approximately 2 hours, which was supported by the department. There were no additional costs incurred, and no staff support was required except to obtain the rotation schedule.

One-page assessment forms for each clinical domain (provided as online supplemental material) were adapted from several sources, including the American Board of Internal Medicine Mini–Clinical Evaluation Exercise (CEX),^16,17  the Arizona Clinical Interview Rating Scale,¹⁸  and literature on principles of informed decision making.¹⁹  The construct validity of mini-CEX use with residents is well demonstrated.^20–25  Consistent with the precepts of competency-based education, the assessments evaluate the presence or absence of observable behaviors, rather than using scales or judgments of relative skill. Drafts of the tools were e-mailed to the 14 faculty preceptors, who subsequently met to discuss revisions. The final forms assessed agreed-on specific behaviors and provided open-text space for additional feedback. Preceptors also underwent a 1-hour faculty development session focused on the rationale for direct observation, finding opportunities to directly observe (using the assessment form) and deliver feedback. Faculty members were instructed to complete the form while observing the resident, and to provide the resident with verbal feedback accompanied by the completed form. The 9 formative assessments replaced the existing summative rotation assessment by faculty, which was removed to increase feasibility, and because we questioned its validity as faculty had often based these summative assessments on very few resident interactions.

Residents completed a postrotation online survey; those who completed more than 1 rotation during the study period were surveyed only after their first block. The survey queried how often they had been directly observed prior to program initiation; the overall value of the direct observation experience (1, not at all, to 5, very); the clinical domain for which direct observation was most valuable (history taking, physical examination, or counseling/decision making); and if their skills had changed as a result of feedback, and if so, how they had changed.

Faculty completed an online survey 6 months after program initiation assessing how long the observations took, how many direct observations they conducted per year prior to implementation of the program and during the 6 months postprogram, and the clinical domain for which direct observation assessment and feedback was most valuable. The survey also queried the usefulness of direct observation (1, not at all, to 5, extremely); the likelihood of conducting direct observation in the future (1, not at all, to 5, much more); and how burdensome they anticipated the observations would be versus how burdensome they actually were (1, not at all, to 5, extremely). Finally, we asked if the new assessment system was better, worse, or the same as the prior single summative rotation assessment. Faculty and resident surveys were developed by the investigative team without further testing. Paired t tests were used to compare frequency of direct observation before and after the program and perceived burden versus actual burden of direct observation.

The University of Wisconsin Institutional Review Board declared this study exempt.

Fifty-seven residents rotated through the block from January 2013 through August 2015 (35 postgraduate year [PGY] 1, 15 PGY-2, and 11 PGY-3). The average years of practice of faculty evaluators was 18.4 (range 3–33). A total of 507 observations were documented on assessment forms: 169 history taking, 171 physical examination, and 167 counseling/decision making. Fifty-two of the 57 (91%) residents completed all 9 observations; the other 5 completed 7 or 8 (and were encouraged to complete the remaining observations in the subsequent month's continuity clinic).

All 57 residents completed the postrotation survey. Thirty-two (56%) reported 0 to 3 previous direct observations during residency, and 51 (89%) reported having changed aspects of their history taking, physical examination, or counseling, based on faculty feedback. Residents provided examples of changes in all 3 domains (history taking: “I now try to probe what it is they are anxious/scared about”; physical examination: “I changed my McMurray technique and the way I look for effusions”; and counseling/decision making: “I remember to counsel patients about risks and benefits of starting new medications”). Of these domains, 49 (86%) of residents felt that physical examination and counseling/decision making observations were more helpful than history taking (figure 1).

All 14 faculty members conducted observations using the assessment tools, and 12 (86%) completed the 6-month survey. Seven (58%) reported conducting direct observations fewer than 6 times per year prior to the new assessment program; 9 (75%) reported conducting at least 6 during the 6 months after program inception (P = .11); and 11 (92%) rated the utility of direct observations for evaluating and providing feedback as very/extremely useful. Four (33%) thought history taking was the most useful domain for observation (figure 1).

According to faculty, the observation process took on average 10.6 minutes (with a range of 5 to 15 minutes). Faculty reported significant improvement (P = .015) in how burdensome they anticipated the observations would be and how burdensome they actually were (figure 2). Eight (67%) reported that the new system made it more likely/much more likely that they would conduct direct observation in the future, and 10 (83%) preferred the multiple formative assessments to the single summative rotation assessment.

This novel assessment system successfully increased and maintained direct observation of residents' clinical skills, resulting in more than 500 encounters having been observed for the roughly 15 interns who complete the program yearly. Residents found the experience useful, with a large majority reporting changes in clinical skills.

Over half of residents reported undergoing observation fewer than 4 times during residency, prior to implementation of the system. This is consistent with the literature, which indicates that direct observation, though integral to competency-based education, is inadequately implemented. One study assessed 2-hour blocks of faculty-resident interaction time in emergency, inpatient, and outpatient settings, finding that only 1% to 6% of time involved direct observation.¹²  Another randomized controlled trial of an intervention to improve feedback for residents revealed that fewer than half reported being observed while they were conducting any portion of a physical examination.¹⁸  Another study revealed that 19% of interns had never been observed taking a history.¹⁷ 

Approximately 90% of our residents completed all 9 observations, and 100% were observed at least 7 times during the block rotation. Informal feedback from residents suggested that a key factor in achieving these high rates was the presence of a faculty member who provided a brief orientation to incoming residents, sent a midrotation e-mail reminder to complete observations, and collected assessment forms at rotation end. Indeed, recommendations have called for developing department champions to spearhead implementation efforts,²⁶  and our findings support the effectiveness of this idea. Another technique that may have contributed to success was making residents accountable for obtaining the mandatory observations. We feel that this is more likely to produce success than relying on supervising faculty to ensure observation completion.

A frequent concern about implementing ongoing direct observation is increased faculty burden.¹²  Faculty initially shared this concern, yet this concern diminished upon participation. The observation and feedback process was relatively quick, and the 9 observations per resident were spread among numerous faculty members over the course of 1 to 3 months. This may explain why faculty felt that the observations did not unduly add to their workday. In fact, faculty expressed a preference for this system, compared to the prior system, which did not mandate direct observation. Achieving success with a direct observation program may depend on getting faculty to commit to the program, and waiting to see if attitudes adjust as faculty participates in observations.

We noted a discrepancy between residents and faculty regarding which clinical domain was most useful for direct observation. Few residents endorsed history taking, but a third of faculty felt this domain was most useful. One interpretation is that residents may underestimate the nuances of skill necessary for optimal history taking, and overestimate their abilities. Another explanation is that residents perceive history taking as a skill learned in medical school, and therefore feel more uncomfortable being observed in this domain. Regardless, faculty should be encouraged to convey the benefits of assessment and feedback for history-taking skills.

There are several limitations to our study. Our assessment of the program's impact on changes to resident skills used self-report rather than objective measures. Our faculty sample was relatively small, and their improved attitudes toward direct observation may not be generalizable. The resident and faculty surveys were created by our team, and lack evidence of validity. We did not link our direct observations to a summative competency assessment. Our goal was to create a program that achieved ongoing direct observation, which is an important component of competency-based education. Now that this has been accomplished, next steps will include obtaining aggregate scores across a set of contexts, evaluators, and clinical scenarios, to document progress toward the achievement of milestones in these skills. Finally, it is unclear to what extent direct observation affects the resident-patient relationship; surveying patients would be an important area for further study.

We developed a novel direct observation system for residents in the ambulatory setting, replacing a single summative evaluation with multiple formative evaluations based on direct observation. The program, despite initial faculty skepticism, was successful in achieving high rates of observations and represents a sustainable approach for assessing residents in the ambulatory setting.