A Pilot Study of the Creation and Implementation of a Teaching Development Assessment Tool Open Access

The development of an individual's skills as a clinical teacher occurs over time and, similar to other skills in medicine, can be tracked through the achievement of discrete observable milestones. While there are well-accepted skills for effective clinical teachers,^1,2  previous studies of tools to assess these skills have not measured the longitudinal development of these skills.^3–6  Most of these tools have been studied in simulated teaching sessions that utilize physicians or physicians-in-training as raters.^7–11  Only a few were studied during live teaching encounters,^3,5,7  and none to date have used nonphysician raters, a valuable quality in an era of increasing demands on physicians' time. Additionally, many of these tools do not have established evidence of validity.

In 2011, a General Pediatric Master Educator Fellowship was established at Cincinnati Children's Hospital Medical Center (CCHMC) to formally train pediatricians in medical education.¹²  Our fellows, who are graduates of accredited pediatrics residencies, receive training in curriculum development, evaluation methodologies, and educational scholarship. They are also encouraged to develop their teaching skills in a variety of clinical settings for which we sought an assessment tool to document this development over time. However, we were unable to find an assessment tool with evidence of validity that tracked skill development longitudinally.

To assess the evolution of one's teaching skills on inpatient rounds we sought to construct a novel teaching development assessment tool (TDAT). This educational innovation had 2 aims: (1) to develop a tool based on rigorous descriptions of observable teaching behaviors (OTBs) along a continuum from beginner to mastery for 6 key teaching domains; and (2) to determine if this tool could reliably be used by trained nonphysician observers. We hypothesized that the TDAT would show evidence of validity and would prove to be a viable alternative to scarce faculty resources through reliable measurement by a trained nonphysician.

We describe the construction of the TDAT, and the assessment of its validity and interrater reliability for physician and trained nonphysician observers.

The Institutional Review Board at CCHMC approved this study as exempt.

Construction of the TDAT began with a review of the literature on best practices in teaching and evaluation tools to identify the core teaching domains for an inpatient attending physician.^1,2,4–6,13  This review yielded 12 potential teaching domains. Through an iterative process of in-person and phone meetings, using medical education experts locally and nationally, these domains were consolidated into 6 key teaching domains: (1) promoting clinical reasoning; (2) taking control of the inpatient learning environment; (3) using learner-centered teaching techniques; (4) having the ability to teach to multiple levels of learners; (5) instructing learners in physical examination techniques; and (6) providing feedback to learners.

Each of these key teaching domains was then broken down into discrete OTBs. Each OTB contained 3 distinct levels that demonstrate the pathway from novice to expert. The relationship between teaching domains, OTBs, and the continuum of levels of achievement are depicted in the online supplemental material.

During the development of the TDAT a trained physician spent 5 hours training a nonphysician research assistant (RA) on the components of effective teaching skills and the use of the TDAT. Together they piloted the tool during 6 inpatient rounding sessions, after which the RA completed another 4 sessions on her own, for a total of 16 hours of dedicated training. The RA then provided feedback on the OTBs to ensure they were easily identifiable to a nonphysician observer. The tool also underwent meticulous review by members of the CCHMC Medical Education Scholarship Team, the General Pediatric Master Educator Fellowship Oversight Committee, and mentors and scholars in the Academic Pediatric Association's Educational Scholars Program.¹⁴  Edits and revisions were made following each pilot use and review session.

Each of the inpatient rounding sessions included a team comprised of an attending physician, 1 to 2 senior residents, 2 to 3 interns, and 3 to 4 medical students for a time period of 1 to 3 hours (depending on the team census). The attending physicians were responsible for supervising and educating the various levels of learners, as well as overseeing the team's medical decision making. The attending physicians consisted of faculty or fellows from the Division of Hospital Medicine in the Department of Pediatrics at CCHMC. Only 1 attending physician was observed during each rounding session. At the end of an observed inpatient rounding session, consisting of 5 to 12 patient encounters on average, each observer selected 1 level for each OTB they observed the attending physician frequently executing. In scoring the tool, if a particular OTB was not observed, the observer was instructed to leave that OTB blank. Lack of rating did not imply performance at the novice level; rather it indicated that the attending physician did not demonstrate the particular OTB during a given rounding session. The tier where the majority of OTBs clustered determined the global rating assigned (ie, novice/beginner, competent/proficient, or expert) for that domain. During this pilot period, if there was a discrepancy between the 2 observers, they discussed their assessments and reconciled differences.

Two physicians (the fellowship director or associate director) with advanced training in educational theory and assessment and the previously trained nonphysician RA then spent 21½ hours jointly observing 11 inpatient rounding sessions over a 6-month period. During these observations, the 2 observers (1 physician and the RA) were instructed not to discuss their observations or scoring. All attending physicians being observed were informed of the study and provided verbal consent prior to the observation. The selection of the attending physicians was a convenience sample based on the observers' schedules.

Study data were managed with REDCap electronic data capture tools at CCHMC.¹⁵  Percent perfect agreement and interrater reliability using weighted κ value were calculated for each global teaching domain rating, as well as the percent perfect agreement for each corresponding OTB.

The TDAT (provided as online supplemental material) was reviewed and its content approved by 15 local and 10 national content experts over a 6-month period, conferring evidence of both content and construct validity.^16,17  For the 17 observations completed during the pilot and study phases, all had scores for at least 4 of the 6 skill domains on the TDAT. Variability in the scores was noted, indicating that not everyone received the same score. The ability of a trained nonphysician to easily observe and score attending physicians on 17 inpatient rounding sessions supports emerging evidence of viable validity. Chen¹⁸  describes viable validity (1 of the 3 components of integrative validity) as the degree to which intervention or assessment tool is practical, affordable, suitable, evaluable, and helpful in the real world. Our experiences in implementing the TDAT have shown that it meets most of those qualities. One issue in the broader implementation and affordability of the tool is that we relied on a paid RA for our study. However, if a nonphysician observer can be used in lieu of a physician, it enhances the feasibility and ability to implement the tool.

The percent perfect agreement and weighted κ for each of the 6 global teaching domain ratings and the 3 levels across the developmental continuum (novice/beginner, competent/proficient, or expert) are displayed in the table. Moderate to near-perfect interrater reliability was seen for the domains of promotion of clinical reasoning, control of the learning environment, ability to teach to multiple levels of learners, and provision of feedback.

Our TDAT is promising as a reliable assessment tool with evidence of validity. It holds potential for faculty developers who train clinical teachers and aim to track skill development over time. When compared to similar teaching assessment tools, our tool is novel for the following reasons. First, it provides a discrete framework consisting of OTBs that are easily identifiable and allows tracking of skill development over time. Second, it is an in vivo assessment tool constructed to provide feedback to teachers in a “live” educational setting. Third, a trained nonphysician can feasibly use the tool to observe teaching performance, allowing one to not solely rely on busy faculty members.

Our intervention has several limitations. Our pilot was conducted with a limited number of observations on a small sample of physicians, thus limiting generalizability. We also did not compare our results to other measures of teaching proficiency, such as learner evaluations. Finally, we did not investigate the tool's ability to detect progression of teaching skills over time. We are currently investigating the TDAT with cohorts of fellows and faculty to determine its ability to track teaching skills over time. While initial data are promising, further observations are needed to fully identify the reliability between the 2 observers for all teaching domains and OTBs.

This teaching development assessment tool has potential as a valid and reliable assessment tool to assess the development of an individual's teaching skills along the continuum from early development to mastery, and it is suitable for use by nonphysician observers.