Introduction
To move toward a competency-based residency curriculum, the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project will use educational Milestones to assess residents' acquisition of competence for unsupervised practice, and will require measures of educational program effectiveness by assessing whether residents achieve certain specialty-specific Milestones in the 6 ACGME competencies. In January 2012, the American Board of Thoracic Surgery (ABTS) and the ACGME jointly formed a working group and charged it with developing the Thoracic Surgery Milestones. The development of the Milestones come at an important time, as the curricular and training paradigms in cardiothoracic surgery (CTS) have changed in recent years, and the Milestones will also be used to assess the success of these important changes.
Milestone Development History
In addition to appointment of the Milestone Working Group, an advisory group was created drawing representatives from the ACGME and the ABTS (Drs Baumgartner, Calhoon, Fullerton, and Wood; 2a1box). Dr Carolyn Reed was charged by the ABTS to select the working group (2a1box), which included the authors of this article (Dr Nguyen represented the Thoracic Surgery Residents Association). Members of the group represented the specialty areas of adult cardiac, general thoracic, and pediatric congenital surgery. After Dr Reed's death, Dr Merrill assumed the role of chair.
Box Members of the Thoracic Surgery Milestone Working and Advisory Groups
Working Group
Walter Merrill, MD, Vanderbilt University Medical Center, Chair
Edward Bove, MD, University of Michigan
Ajay Carpenter, MD, University of Texas, Health Science Center at San Antonio
Robert Higgins, MD, The Ohio State University Medical Center
Richard Lee, MD, St. Louis University
Thomas Nguyen, MD, University of Texas Health Science Center at Houston
Carolyn Reed†, MD, Medical University of South Carolina
Ara Vaporciyan, MD, University of Texas MD Anderson Cancer Center
Thomas Varghese, MD, University of Washington
Edward Verrier, MD, University of Washington
Cameron Wright, MD, Massachusetts General Hospital
Stephen C. Yang, MD, Johns Hopkins Medical Institutions
Advisory Group
William Baumgartner, MD, The Johns Hopkins Hospital
Timothy Brigham, MDiv, PhD, Accreditation Council for Graduate Medical Education (ACGME)
John Calhoon, MD, University of Texas Health Science Center
David Fullerton, MD, University of Colorado Anschutz Medical Campus
John Potts, MD, ACGME
Peggy Simpson, EdD, ACGME
Douglas Wood, MD, University of Washington Medical Center
The working group was charged with designing Thoracic Surgery Milestones specifically with integrated 6-year CTS training programs in mind. Traditional 2- and 3-year programs and programs with a 4/3 training model would use the higher outcomes to assess their trainees. The primary goals were to (1) further define and categorize the 6 core competencies into the context of CTS; (2) develop a manageable number of Milestones (less than 36) to ensure the burden of implementing the Milestones would not outweigh the benefits; and (3) use the newly developed ABTS curriculum outline to determine Milestone anchors.1
The working group began the development of the Milestones in March 2012 and held 2 in-person meetings to develop the Milestones for the Medical Knowledge (MK) and Patient Care/Technical Skills (PC/TS) competencies. Once these Milestones had been finalized, the working group completed the remaining 4 competencies and developed assessment tools specifically for CTS. Weekly conference calls were held until the completion of the project in early December 2012.
The completed Thoracic Surgery Milestones were presented by the Thoracic Surgery Directors Association at the Society for Thoracic Surgery and the American Association of Thoracic Surgeons 2013 meetings.2
General Features of the Specialty Milestones
The working group first determined that the MK and PCTS competencies would be categorized according to organ systems to be consistent with the organization of the ABTS curriculum.1 A total of 25 Milestones were developed across the 6 competencies (table). For the cross-cutting competencies (Interpersonal and Communication Skills, Professionalism, Practice-Based Learning and Improvement, Systems-Based Practice), the group chose to follow a structure developed by an ACGME expert panel. The working group expected that residents demonstrate a commitment to carrying out professional responsibilities and adhere to ethical principles. To respond to assessment needs arising out of this, a number of tools were developed specifically for CTS in the areas of communication, professionalism, and teaching.
Establishing Milestone Validity, Utility, and Practicality
The integrity, validity, and utility of the Milestones in thoracic surgery, as is the case with Milestones in other specialties, will need rigorous study. Some of the major threats to rigorous performance assessment validity include few performance cases and rater bias, especially since the year of training is known to faculty raters. To promote the quality and validity of ratings, it will be important that all faculty raters be trained in how to use the assessment tools to ensure consistent and appropriate evaluation. Assessment tools and the collection of data should undergo beta-testing to ensure that final adjustments and implementation feedback can be done prior to the full implementation of the Milestones (with the first submission of Milestone data due to the ACGME in January 2015).
Initial beta-testing of the CTS Milestones was carried out during the spring of 2013 at selected institutions nationally. In general, the majority of pilot participants agreed that the Milestones discriminate among trainees by competency and that the subcompetencies reflected what was needed to complete training. It was emphasized that the Milestones mainly are a reporting tool, that choosing and implementing assessment tools is done at the level of the individual program, and that placement of individual trainees within a specific Milestone often requires multiple ways of assessing them.
Envisioned Practical Use in Evaluating Residents
It is expected that each thoracic surgery program will prepare a summary Milestone evaluation for each resident. These evaluations will be submitted to the ACGME on a semiannual basis to be used in an annual review of program performance for accreditation purposes. Assignment of Milestone levels (1–5) is expected to be based on objective data and evaluations. Consistent with the ACGME Milestones Project for all specialties, achievement of the Thoracic Surgery Milestones is not intended to be time-dependent, though. Level 4 is the knowledge, skills, and behavior expected of a graduating CTS resident, and it aligns with the ABTS content for certification and generally should be accomplished by the end of residency. For residents whose progress appears delayed, areas of weakness should be identified and a plan for remediation instituted, especially if lack of progress or a regression is noted or delayed acquisition of competency is noted when a trainee is compared to others at the same training level within a program.
Recommendations for Competency Committee Composition and Functioning
It is expected that each training program form a Clinical Competency Committee (CCC) that will include core teaching faculty and representatives from the different disciplines and institutions unique to the given program. The role of the CCC encompasses:
determining Milestone achievement levels for each resident;
reviewing, if possible developing, and implementing new assessment tools;
gathering information about a resident's performance at many points to minimize the impact of variation of assessments by different evaluators;
reviewing all available assessment data for each trainee; and
reaching consensus for all Milestone level assignments.
Conclusion
The Thoracic Surgery Milestones will be used to assess and provide feedback to residents in areas highly relevant to unsupervised practice in the specialty, and will also be used as part of the ACGME's annual assessment of programs. These uses of the Milestones will also provide the data for research on their validity and utility, and on their value to individual assessments and to the accreditation process.
References
†deceased.
