The Challenge

In the Next Accreditation System, the Accreditation Council for Graduate Medical Education requires training programs to develop a clinical competency committee (CCC) to monitor and record the progress of their residents on predefined Milestones. In many programs, the current practice is for the program director to weigh the evaluation data from multiple sources and make decisions about each trainee's progress with or without discussion with program faculty. Some programs, however, already use a committee for promotion decisions. The committee monitors resident growth and makes recommendations, such as specific educational or remediation plans. A few specialties such as anesthesiology have used competency committees. The challenge now is for each residency program to formally develop a CCC and leverage or discontinue existing processes as needed.

What Is Known

The Purpose of the CCC

Each program must create a document that describes the responsibilities and membership of the CCC. The CCC should be designed to triangulate the evaluation process; the CCC judges each resident's progress using the results from multiple assessment tools, taking into consideration tool reliability and validity. The CCC is expected to meet at least twice annually to assess each resident's progress in acquiring the relevant reporting Milestones for the resident's level of training. The CCC must make recommendations to the program director (PD) regarding resident progress, including promotion, remediation, and dismissal. Therefore, CCC recommendations will be part of an early warning system for residents who may not be progressing as expected.

Rip Out action items

Programs must:

  1. 1..

    Identify CCC members, including a chair.

  2. 2..

    Set CCC meeting dates and times.

  3. 3..

    Educate committee members about the CCC process.

  4. 4..

    Review current assessment tools.

  5. 5..

    Map Milestones to objectives and existing assessment tools.

  6. 6..

    Implement faculty development for all program faculty members.

  7. Establish a process for CCC's own continuous improvement.

Value of Group Decision Making for High-Stakes Assessment

No single assessment can provide all the information necessary for a competency appraisal. Experts suggest that 8 to 10 samples provide enough information for a reliable assessment.1 Qualitative assessments using narratives, such as comments from direct observation or work-based assessments, have been found to provide important information on resident performance.2,3 Studies show that group input and committee discussions can provide valuable information on resident performance and identify deficiencies not obtained from individual assessments.4,5 

A Guide to Developing a CCC

  • 1.

    Membership: CCCs should have a minimum of 3 faculty members but may have more. Membership can include faculty, assessment specialists, a medical director or service chief, faculty from other programs, nurses, and nonphysician members of the medical team.

  • Chief residents who have completed core residency programs in their specialty disciplines, possess a faculty appointment from the program, and are eligible for specialty board certification may attend the CCC meetings and provide input to CCC deliberations. They cannot be members of the CCC.

  • Residents in the accredited years of training who do not meet the above criteria may not attend CCC meetings. This ensures that resident peers do not have direct input into promotion and graduation decisions, or are involved in recommendations for remediation or disciplinary actions. Input from resident peers may be sought through the evaluation system.

  • Coordinators may attend CCC meetings to provide administrative support and help document CCC deliberations and decisions, but may not serve as CCC members.

  • 2.

    Role and responsibility of CCC members: The CCC members are expected to provide honest, thoughtful evaluations of each resident and participate in consensus decisions about the trainee's competency level. These decisions must be based on multisource input, not personal opinion alone. The CCC members will determine the schedule of meetings. The CCC should meet at least semiannually; CCCs for larger programs may need to meet more frequently.

  • 3.

    CCC chair: The chair guides the committee in its work to provide a consensus recommendation for reporting Milestones. Some program requirements state that the PD cannot serve as CCC chair. In programs where residency review committees are flexible on this issue, the PD may opt not to serve as chair to allow the CCC to provide independent recommendations, though the PD would continue to have ultimate decision-making authority.

  • 4.

    Faculty development: The following points should be considered in terms of faculty development:

    • Committee members must be oriented to each assessment tool and its relationship to the reporting Milestones. The CCC will need to decide how many assessments are needed for any given Milestone, how to aggregate data across tools, and how to verify data quality.

    • All program faculty must be trained regarding the reporting Milestones and their associated assessment tools to enhance rating consistency and accuracy. Training should include discussion of the Milestone levels and should establish agreement on the meaning assigned to each tool's rating anchors. The CCC's approach to aggregating and interpreting assessment data from each resident will need to be discussed with and agreed on by all faculty members, in an ongoing fashion, to ensure consistency in evaluation decisions.

  • 5.

    The review process: The data review process may vary by specialty and program. Potential strategies include (a) 2 faculty members review a resident's file and then prepare a joint recommendation to the committee; (b) 1 faculty group reviews the junior residents and another group reviews the senior resident files; (c) each faculty member is assigned a specific Milestone(s) and reviews all files for that Milestone (eg, 1 member reviews all residents' Milestone achievements for systems-based practice); and (d) a subgroup of CCC members reviews resident files and provides the recommendations for residents specifically needing remediation or probation. Currently, there is no consensus regarding involvement of the resident's faculty adviser in the file review process.

How You Can Start TODAY

  1. Write a description of the CCC's responsibilities.

  2. Select CCC members and ensure that they are familiar with the program's Milestones and assessment tools.

  3. Map current rotation objectives and overall program educational goals to the Milestones.

  4. Map current assessment tools to the Milestones to identify unnecessary redundancies and opportunities for further tool development to meet gaps. Before creating a new tool, consider if minor revisions to existing tools would meet your needs, or review existing tools developed by other programs.

What You Can Do LONG TERM

The CCC should institute a process for continuous improvement. The CCC data can be used to inform the annual program evaluation. Examples include the following:

  1. In the CCC minutes, include a section for questions to be addressed during annual program review. For example, do all residents have the opportunity to achieve all levels of competency in a Milestone within the current program structure?

  2. Document how the Milestone consensus was reached and determine whether the appropriate assessment tools are available for all Milestones. Does the program gather sufficient data for each Milestone? Over time, the CCC will likely identify gaps and new assessment tools that should be developed by the program.

Resources

Resources
1
van der Vleuten
CP
.,
Schurwirth
LW
,
Scheele
F
,
Driessen
EW
,
Hodges
B
.
The assessment of professional competence: building blocks for theory development
,
Best Pract Res Clin Obstet Gynaecol
,
2010
;
24
(
6)
):
703
719
.
2
Regehr
G
.,
Ginsburg
S
,
Herold
J
,
Hatala
R
,
Eva
K
,
Oulanova
O
.
Using standardized narratives to explore new ways to represent faculty opinions of resident performance
,
Acad Med
,
2012
;
87
(
4
):
419
427
.
3
Govaerts
MJ
.,
Van de Wiel
MW
,
Schuwirth
LW
,
Van der Vleuten
CP
,
Muijtjens
AM
.
Workplace-based assessment: raters' performance theories and constructs
,
Adv Health Sci Educ Theory Pract
,
2013
;
18
(
3
):
375
396
.
4
Schwind
CJ
.,
Williams
RG
,
Boehler
ML
,
Dunnington
GL
.
Do individual attendings' post-rotation performance ratings detect residents' clinical performance deficiencies
?
Acad Med
,
2004
;
79
(
5
):
453
457
.
5
Gaglione
MM
.,
Moores
L
,
Pangaro
L
,
Hemmer
PA
.
Does group discussion of student clerkship performance at an education committee affect an individual committee member's decisions
?
Acad Med
,
2005
;
80
(
suppl 10
):
55
58
.

Author notes

Susan B. Promes, MD, MBA, is Program Director and Vice Chair for Education and Professor, Department of Emergency Medicine, University of California, San Francisco; and Mary Jo Wagner, MD, is Program Director, Chief, and Professor, Emergency Medicine, Central Michigan University College of Medicine.