Developing the Otolaryngology Milestones Open Access

Under the guidance of the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Otolaryngology (ABOto), a working group of educators, program directors, residency review committee (RRC) members, experts in the specialty, and a resident representative developed the Otolaryngology Milestones with the dual aims of (1) allowing programs to track the educational progress of their residents, culminating in readiness to enter unsupervised practice; and (2) using Milestone data aggregated to the program level to report on otolaryngology programs' educational effectiveness for accreditation purposes.

The ACGME RRC and the ABOto leadership appointed a Milestone Working Group and a Milestone Advisory Group. Both groups were led by a Milestone chair jointly selected by the RRC and ABOto. The educational leadership of the major otolaryngology societies nominated individuals for membership in these groups.

The advisory group, whose membership was made up of senior educational leaders from these societies, provided oversight of the effort, including guidance on the organizing format for the Otolaryngology Milestones. The charge to the advisory group also included assistance with communicating and disseminating information about the development of the Otolaryngology Milestones.

The working group was responsible for Milestone content development. Its members had a demonstrated commitment to otolaryngology education at the national level, and represented all subspecialty areas of otolaryngology (facial plastic and reconstructive surgery, head and neck, otology/neurotology, rhinology/allergy, laryngology, sleep medicine/pediatric otolaryngology).

After review of Milestone sets developed by other RRCs, the advisory group approved a “disease-based” format for the Patient Care (PC) and Medical Knowledge (MK) Milestones. Other organizing formats considered and ultimately not selected included organization by anatomic site and by skills for specific contexts and encounters (ie, consultation).

The working group identified 19 disease processes spread across otolaryngology subspecialty areas to ensure Milestone representation across the spectrum of care. These were derived from the ABOto Scope of Knowledge Study Summary Report,1 the Otolaryngology-Head and Neck Surgery Comprehensive Core Curriculum2 and Core Surgical Procedures,3 and the Otolaryngology Program Requirements and Key Indicator Procedures. Milestone sets are organized under subcompetency headings that relate to a disease process, such as salivary disease (Table 1). Delineation into either a PC or a MK Milestone/subcompetency set was based on the importance of procedural and patient care skills versus medical knowledge for each disease process.

The advisory group and RRC narrowed the 19 candidate Milestone disease processes down to 12 Milestone sets for seminal subcompetencies (8 for PC and 4 for MK). An ACGME expert panel developed drafts of Systems-Based Practice, Practice-Based Learning and Improvement, Professionalism, and Interpersonal and Communication Skills Milestone sets. The working group narrowed these down to the 5 subcompetencies/Milestone sets most applicable to otolaryngology.

In the Milestone sets for PC and MK, each of the 6 subspecialty areas of Otolaryngology has 2 Milestones. These 12 Milestone sets, and the 5 Milestone sets for the other competences comprise the 17 subcompetencies that will be used to assess the performance of otolaryngology residents and residency programs.

All Milestone sets are scored from Level 1 through Level 5, listed horizontally (Table 1). Importantly, these scores do not represent postgraduate year (PGY) levels. Level 1 represents the expectations for a “novice” or graduating medical student. Level 4 represents a targeted competency level expected of a graduating resident, while Level 5 represents an aspirational or “stretch” goal that will be achieved by a subset of the residents during their training years.

Within the subcompetencies, the Milestones represent the general skills, attitudes, attributes, and knowledge that correspond to each Milestone component and level. These components differ in content and order for the PC and MK Milestones (Table 2).

Given the need to measure early resident competency with higher fidelity and to help identify potential underperformers at the earliest possible time, the set includes 6 “early” Milestones and 6 “late” Milestones. Although the timing of Milestone competency is variable among programs and individual residents, some of the subcompetencies were developed to allow the majority of residents to progress to Level 4 at a more junior PGY level. Optimally, graduating residents should obtain at least a Level 4 on each of the subcompetencies.

After final review by the advisory committee, the RRC, and the ABOto Educational Council, the Otolaryngology Milestones were presented at the 2012 Society of University Otolaryngologists/Otolaryngology Program Directors Organization Annual Meeting.

The Milestones were subsequently piloted in 14% of the otolaryngology programs with a 56% to 93% favorable response to the pilot survey questions. Some additional changes suggested by pilot participants were included in the final version. The pilot demonstrated completing the Otolaryngology Milestone evaluations takes an average of 10 to 12 minutes per resident.

The RRC for Otolaryngology will use the Milestone evaluations to assess global program outcomes as a measure of program performance. The Milestones also will be useful for program directors. They are intended to describe residents' progress from novice to competent/proficient based on observable measures, which is superior to the traditional “operative numbers” methodology. The Milestones will give program leaders new tools to objectively document residents' performance.

The individual Milestones within the grid facilitate accurate documentation of the performance level for each subcompetency. This detail also affords ability to provide focused formative feedback to the resident and can finely define areas of resident deficiency for focused study. The transparency of the Milestone approach may encourage resident self-assessment and self-directed learning, as well as help guide program curricular development.

With experience, each program can begin to understand the training level during which the residents typically obtain competency. Thresholds for acceptable program progress can then be ascertained on a PGY level basis. This would be different for each program as each program's training paradigm is unique. This can be used to gauge progress of their subsequent residents and help direct annual summative evaluations for resident promotion. Aggregated to the program level, the Milestone data also are useful for the identification of areas for improvement within the curriculum. The appropriate actions to be taken if a resident does not achieve Level 4 competency are a critical next area of study.

In addition to the 12 Milestones for which data are reported to the ACGME, the working group developed an additional 7 supplemental Milestone sets that will be made available on the webpage of the RRC for Otolaryngology for use by programs in identifying and remediating residents with deficiencies in the areas of MK or PC. Programs will not submit Milestone data for these added subcompetencies to the ACGME.

The ACGME has charged each program's Clinical Competency Committee (CCC) with semiannually completing Milestone assessments for all residents. The CCC needs to make sure the evaluations are useful and effective for resident evaluation and program improvement. The intent is to have the CCC review all assessments and evaluations completed during the previous 6 months and determine individual resident and aggregate program levels on the Milestones.

This review can be done several ways. First, residents could be evaluated by the CCC to complete the Milestone scoring at a set time every 6 months. However, a downside of this approach is the risk of potential limited exposure of the CCC membership or resident to the specific Milestone disease processes during the assessment period. Second, subspecialty faculty that educate and supervise residents in a particular Milestone disease process can provide the CCC with information on resident performance on particular subcompetencies. Third, the CCC could develop an evaluation form that parallels the Milestones to be used in addition to the existing evaluations. The residents could be evaluated real time by involved specialty-specific faculty. These scores can be collated to develop a consensus score by the CCC. Programs that piloted the Milestones used direct observation with immediate written assessment completed on a rating form or checklist, and end-of-rotation evaluations as the 2 most effective tools to assess the Milestones.

All Milestone sets require reassessment by the CCC throughout the training, as competency can change (in either direction) depending on a particular resident's experience.

The RRC for Otolaryngology is committed to making Milestone implementation an iterative process. The validity of the Milestones to determine competency has yet to be determined, and will be studied in the coming years. Part of this process could include the correlation of the Milestone competency determination with data being collected from program directors by the ABOto on procedural competency timing. As the field evolves, the Milestones may need to be updated to reflect changes in the specialty in the modalities of care and in the approach to education.