As the Next Accreditation System (NAS) is rolled out, training programs will need to implement a process for continuous faculty development on the use of assessment tools. Reliable assessments of performance by multiple trained faculty are essential to demonstrate resident skill development over time. Yet, with increasing time demands on faculty, full- or half-day training sessions recommended by experts for greater efficacy of training are generally not feasible.1 Alternately, brief, continuous training, embedded in regularly scheduled meetings, can support ongoing faculty development to help meet NAS requirements.2
What Is Known
Assessment tools are only as good as the individuals who use them.3,4 A heightened awareness among faculty about the purpose, use, and interpretation of assessment tools will develop shared mental models and more reliable performance assessments over time.5 Combining the results of 6 to 10 observations with other assessment instruments yields a “composite reliability” on which progress in professional development and performance can be documented in a reliable manner.6 Thus, it is imperative that faculty receive regular training on each assessment instrument to understand (1) the purpose and use of the tool (Behavior Observation Training [BOT]), (2) the definitions and criteria for each performance dimension on the tool (Performance Dimension Training [PDT]), and (3) the need for rating accurately and consistently with the tool (Frame of Reference Training [FORT]).3
To report the Milestone evaluations, Clinical Competency Committees (CCCs) will rely on composite scores from faculty assessments of residents. Given the importance of each faculty assessment, medical educators have the professional responsibility to become trained raters on their program assessment tools.3
A Guide to Developing Mini-Sessions on Assessment for Faculty
Conduct a needs assessment with your CCC. Ask CCC members to identify assessment instruments and concomitant education needed to train faculty. For example, there might be concerns regarding the Milestone implementation, recent revisions to an assessment tool, or wide variation among faculty members' ratings resulting in inconsistent scoring. With the CCC, develop and prioritize a list of faculty training needs as the core content for the mini-session curriculum. Consider providing the CCC with a discussion resource, such as the Holmboe3 (p119–124) chapter on “Direct Observation by Faculty.”
Develop the mini-session curriculum. The overarching goal is for faculty to use program assessment tools with greater ease, higher frequency, and increased reliability. The number and order of sessions will depend on the identified list of assessment needs and will vary by program. Individual mini-sessions, based on the priority list of faculty training assessment needs above, might also include 1 or more of the following objectives:
Discuss the purpose, uses, and context of the assessment tool (BOT).
Review and discuss the definitions and criteria used in the assessment tool (PDT).
Elicit examples of resident behaviors that exemplify specific ratings to achieve greater interrater reliability (FORT).
Frame and deliver the mini-sessions. To support a continuous improvement cycle for faculty development, design a brief evaluation form for participants after each session. The form might include items on session relevancy, effectiveness, and usefulness (5-point Likert scale). Administer the form after every mini-session (t a b l e).
How You Can Start TODAY
Talk to your CCC for initial confirmation of faculty development needs and initiate preliminary planning.
With the CCC, prepare a brief proposal, including rationale, needs, goals, curriculum, and framework.
Secure a commitment from your chair/leader for 20 minutes during standing faculty meetings and retreats.
Designate interested faculty, including members of your CCC, to lead sessions.
Establish a framework that will work for your faculty to develop more shared mental models.
Design a simple evaluation form for the mini-sessions for continuous improvement.
What You Can Do LONG TERM
As part of ongoing needs assessment, survey faculty on issues regarding assessment in the NAS; design mini-sessions on assessments accordingly.
Gather baseline data on assessment tools to document temporal change (eg, rater variations, frequency).
Select outcome measures (eg, interrater reliability, faculty completion rate, survey responses).
Develop mini-sessions for all evaluation instruments or for other faculty development needs.
Build consensus among faculty over time for resident behaviors at each performance level, while simultaneously acknowledging the differing opinions and perceptions.
Work with their Clinical Competency Committee (CCC) to frame faculty development needs and goals by:
Identifying specific assessment tools and focus for targeted training;
Clarifying the purpose and use of assessment tools, defining criteria and behavior descriptions in the tools; and
Calibrating faculty for accuracy and consistency.
Secure 20 minutes of training time within standing faculty meetings/retreats.
Enlist CCC members to coplan the sessions.
References and Resources for Further Reading
All authors are at the University of Washington. Susan S. Johnston, EdD, is Director of Education, Graduate Medical Education; Isaac A. Bohannon, MD, is Assistant Professor and Program Director, Department of Otolaryngology; and Byron D. Joyner, MD, MPA, is Vice Dean for Graduate Medical Education and Designated Institutional Official (DIO), University of Washington School of Medicine.