The Challenge

As the Next Accreditation System (NAS) is rolled out, training programs will need to implement a process for continuous faculty development on the use of assessment tools. Reliable assessments of performance by multiple trained faculty are essential to demonstrate resident skill development over time. Yet, with increasing time demands on faculty, full- or half-day training sessions recommended by experts for greater efficacy of training are generally not feasible.1 Alternately, brief, continuous training, embedded in regularly scheduled meetings, can support ongoing faculty development to help meet NAS requirements.2 

What Is Known

Assessment tools are only as good as the individuals who use them.3,4 A heightened awareness among faculty about the purpose, use, and interpretation of assessment tools will develop shared mental models and more reliable performance assessments over time.5 Combining the results of 6 to 10 observations with other assessment instruments yields a “composite reliability” on which progress in professional development and performance can be documented in a reliable manner.6 Thus, it is imperative that faculty receive regular training on each assessment instrument to understand (1) the purpose and use of the tool (Behavior Observation Training [BOT]), (2) the definitions and criteria for each performance dimension on the tool (Performance Dimension Training [PDT]), and (3) the need for rating accurately and consistently with the tool (Frame of Reference Training [FORT]).3 

To report the Milestone evaluations, Clinical Competency Committees (CCCs) will rely on composite scores from faculty assessments of residents. Given the importance of each faculty assessment, medical educators have the professional responsibility to become trained raters on their program assessment tools.3 

A Guide to Developing Mini-Sessions on Assessment for Faculty

  1. Conduct a needs assessment with your CCC. Ask CCC members to identify assessment instruments and concomitant education needed to train faculty. For example, there might be concerns regarding the Milestone implementation, recent revisions to an assessment tool, or wide variation among faculty members' ratings resulting in inconsistent scoring. With the CCC, develop and prioritize a list of faculty training needs as the core content for the mini-session curriculum. Consider providing the CCC with a discussion resource, such as the Holmboe3 (p119–124) chapter on “Direct Observation by Faculty.”

  2. Develop the mini-session curriculum. The overarching goal is for faculty to use program assessment tools with greater ease, higher frequency, and increased reliability. The number and order of sessions will depend on the identified list of assessment needs and will vary by program. Individual mini-sessions, based on the priority list of faculty training assessment needs above, might also include 1 or more of the following objectives:

    • Discuss the purpose, uses, and context of the assessment tool (BOT).

    • Review and discuss the definitions and criteria used in the assessment tool (PDT).

    • Elicit examples of resident behaviors that exemplify specific ratings to achieve greater interrater reliability (FORT).

  3. Frame and deliver the mini-sessions. To support a continuous improvement cycle for faculty development, design a brief evaluation form for participants after each session. The form might include items on session relevancy, effectiveness, and usefulness (5-point Likert scale). Administer the form after every mini-session (t a b l e).

How You Can Start TODAY

  1. Talk to your CCC for initial confirmation of faculty development needs and initiate preliminary planning.

  2. With the CCC, prepare a brief proposal, including rationale, needs, goals, curriculum, and framework.

  3. Secure a commitment from your chair/leader for 20 minutes during standing faculty meetings and retreats.

  4. Designate interested faculty, including members of your CCC, to lead sessions.

  5. Establish a framework that will work for your faculty to develop more shared mental models.

  6. Design a simple evaluation form for the mini-sessions for continuous improvement.

What You Can Do LONG TERM

  1. As part of ongoing needs assessment, survey faculty on issues regarding assessment in the NAS; design mini-sessions on assessments accordingly.

  2. Gather baseline data on assessment tools to document temporal change (eg, rater variations, frequency).

  3. Select outcome measures (eg, interrater reliability, faculty completion rate, survey responses).

  4. Develop mini-sessions for all evaluation instruments or for other faculty development needs.

  5. Build consensus among faculty over time for resident behaviors at each performance level, while simultaneously acknowledging the differing opinions and perceptions.

Rip Out Action Items

Programs should:

  1. Work with their Clinical Competency Committee (CCC) to frame faculty development needs and goals by:

    • Identifying specific assessment tools and focus for targeted training;

    • Clarifying the purpose and use of assessment tools, defining criteria and behavior descriptions in the tools; and

    • Calibrating faculty for accuracy and consistency.

  2. Secure 20 minutes of training time within standing faculty meetings/retreats.

  3. Enlist CCC members to coplan the sessions.

References and Resources for Further Reading

References and Resources for Further Reading
1.
Holmboe
ES
,
Ward
DS
,
Reznick
RK
,
Katsufrakis
PJ
,
Leslie
KM
,
Patel
VL
,
et al.
.
Faculty development in assessment: the missing link in competency-based medical education
.
Acad Med
.
2011
;
86
(
4
):
460
467
.
2.
Bar-on
ME
,
and
Konopasek
L
.
Snippets: an innovative method for efficient, effective faculty development
.
J Grad Med Educ
.
2014
;
6
(
2
):
207
210
.
3.
Holmboe
ES
,
and
Hawkins
RE
.
Practical Guide to the Evaluation of Clinical Competence
.
Philadelphia, PA
:
Mosby Elsevier
;
2008
.
4.
van der Vleuten
CP
,
Schuwirth
LW
,
Driessen
EW
,
Dijkstra
J
,
Tigelaar
D
,
Baartman
LK
,
et al.
.
A model for programmatic assessment fit for purpose
.
Med Teach
.
2012
;
34
(
3
):
205
214
.
5.
Kogan
JR
,
Conforti
L
,
Bernabeo
E
,
Iobst
W
,
and
Holmboe
E
.
Opening the black box of clinical skills assessment via observation: a conceptual model
.
Med Educ
.
2011
;
45
(
10
):
1048
1060
.
6.
Moonen-van Loon
JM
,
Overeem
K
,
Donkers
HH
,
van der Vleuten
CP
,
and
Driessen
EW
.
Composite reliability of a workplace-based assessment toolbox for postgraduate medical education
.
Adv Health Sci Educ Theory Pract
.
2013
;
18
(
5
):
1087
1020
.

Author notes

All authors are at the University of Washington. Susan S. Johnston, EdD, is Director of Education, Graduate Medical Education; Isaac A. Bohannon, MD, is Assistant Professor and Program Director, Department of Otolaryngology; and Byron D. Joyner, MD, MPA, is Vice Dean for Graduate Medical Education and Designated Institutional Official (DIO), University of Washington School of Medicine.