Stages of Milestones Implementation: A Template Analysis of 16 Programs Across 4 Specialties

In 2001, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) published the 6 general competency domains for evaluation of resident performance: patient care, medical knowledge, practice-based learning and improvement, systems-based practice, interpersonal and communication skills, and professionalism. In 2012, the ACGME introduced the Next Accreditation System (NAS).¹  An integral component of NAS was the introduction of a competency-based, developmental framework called the Milestones to inform resident and fellow assessment and progression. Milestones sets articulated specialty-specific subcompetencies within each of the 6 competency domains.²  These Milestones were implemented nationally in 2013 for emergency medicine (EM), internal medicine (IM), and pediatrics.³  The family medicine (FM) Milestones were implemented in 2014.⁴ 

The ACGME also required all programs to create clinical competency committees (CCCs) as part of the NAS. CCCs consist of 3 or more members of the faculty and convene at least twice per year to review resident assessments and make recommendations to program directors on resident subcompetency Milestone ratings and resident progress. Programs are required to share documented performance with residents semiannually. This may occur as an in-person conversation between a faculty educational leader and the resident or fellow. Finally, program directors are required to submit the Milestone ratings semiannually to the ACGME.

The use of the Milestones is part of a larger movement toward competency-based medical education (CBME), with a specific goal of facilitating a transformation from a process-bound system of graduate medical education (GME) accreditation to one that focuses on educational and clinical outcomes. This focus on outcomes serves to prepare physicians for a changing health care system, emphasizing the functional capabilities of graduating residents and ensuring they match patient and health care system needs.

Challenges to the full realization of this vision have been identified in the literature, including concerns about assessment, CCC processes, and effective use of Milestones data for meaningful feedback to residents. While the Milestones define developmental progressions of residents in narrative terms, they are often used as a simple numeric scale in practice, leading to criticisms of reductionism.⁵  This has led to calls for greater faculty development efforts to make better use of the narrative descriptors, both for reflective assessment and feedback to residents.⁶  Another concern involves the methods employed by CCCs to inform their judgments about resident progression during their semi-annual meetings.⁷  Residency programs vary widely in how they provide feedback to residents and in terms of whether the Milestones are used to provide this feedback at all.⁸ 

While Milestone reporting has been required since 2014, effective practices to implement and use Milestones across specialties is not fully known. One previous qualitative study exploring the early use of Milestones in neurological surgery programs found that Milestones helped to build a shared understanding of the competencies among faculty and that CCCs were helpful in assessing residents, while resident and faculty involvement in the Milestones was variable.⁹  Data collected from focus groups suggested Milestones are most effective when residents are introduced to their specialty set early in training, perform self-assessments prior to CCC meetings, compare their self-assessments with CCC feedback in person with a trusted faculty member, and create an individual learning plan.¹⁰  A deeper understanding of Milestone implementation and impact, especially in the large specialties of EM, IM, pediatrics, and FM is needed to better inform effective practice. Lessons from these 4 specialties can guide implementation and effective practice with Milestones as well as help inform the Milestones 2.0 revision and implementation process.¹¹ 

In this study, Milestone implementation was defined as the set of program-specific processes that inform the professional development of medical residents, with the goal of improving process efficiency and the engagement and knowledge of faculty and residents. For this qualitative interview study, we were interested in finding themes, processes, moderating forces, and outcomes attributable to the Milestone implementation efforts of participating programs across the 4 specialties.

This study, using transcript data obtained from interviewing program directors (PDs), CCC chairs, and residents in EM, IM, pediatrics, and FM, served to determine how strategies and environmental contexts affected programmatic assessment, CCC meetings, and feedback processes. Using template analysis to build on previous insights across these 3 Milestone domains, we sought to explore what works (or not), for whom, in what circumstances, and why.¹²  Drawing from previous Milestone-based research and experience, investigators with qualitative content and methodological expertise from the 4 specialties and ACGME research staff created interview protocols and an initial thematic template to guide the coding of transcripts.^9,10,13  The American Institutes of Research Institutional Review Board approved this study after an expedited review.

Using purposeful sampling based on program size, geographic location, and type of sponsoring institution in order to query programs with varying Milestone experiences, 63 programs were identified for invitation to participate in the study. In 2017 and 2018, expert panel members from the 4 specialties sent recruitment emails to program directors, with an a priori goal of obtaining 24 programs (6 from each of the 4 specialties) to participate. A study investigator then contacted these 63 program directors to assess their interest in participating in the study. Upon consenting to participate, PDs provided study staff with contact information for CCC chairs and program coordinators. Program coordinators, in turn, provided contact information for second- and third-year residents who could provide thoughtful responses to interview questions. All subjects who ultimately participated in the interviews verbally consented to participate.

For each participating program, either the program director or the CCC chair was asked to complete a pre-interview survey (available as online supplementary data), querying faculty development and CCC processes within the program. An interview guide was developed and tested by the lead author (N.A.Y.) and contained questions regarding the implementation of the Milestones, experiences using them, and resident and faculty perceptions. All interviews were conducted by telephone. Most program director and CCC chair interviews were conducted individually, but some opted to be interviewed together. A maximum of 2 residents were interviewed per program. When more than one resident in a program was participating, they were given the option to be interviewed together or separately. All interviews were recorded and transcribed. Transcriptions were edited for accuracy, and all identifiable information was anonymized prior to any coding. Any program for which at least one interview was conducted was included in the study.

The interview transcripts were analyzed using template analysis, a form of thematic analysis that emphasizes the use of hierarchical coding by means of a coding template to represent themes identified in the data.¹⁴  The online supplementary data includes more information on the template analysis. The initial thematic template was derived from a previous qualitative study that sought to determine the effect of the ACGME Milestones on the assessment of neurological surgery residents and focus groups conducted by ACGME staff at educational meetings.^9,10,13 

A detailed description of the evolution of the template is included as online supplementary data. The panel held iterative consensus discussions about the template, and the second iteration of the template consisted of 3 overarching domains: processes specific to the CCCs, assessment and resident feedback, and the reported effects of Milestone implementation on the faculty, residents, and culture of residency programs. Originally, all members of the panel employed the template to code 2 interview transcripts from the same program. They were given instructions to evaluate the template and its usability for accurately representing themes and phenomena reported by the interviewees. Specifically, coders were limited to affirming or supporting existing themes, inserting new themes, deleting, or changing the scope of themes, and modifying the hierarchical classification. Coded transcripts and annotated templates were collected and collated, and the template was revised (third iteration) according to the recommendations of the panel.

For feasibility and timeliness of continued textual analysis, 2 coders (N.A.Y., E.C.B.) from the original 8 proceeded with coding additional interviews. The additional transcript coding and template development led to more template iterations. As a result of the second round of coding, a separate codebook was developed to account for the different resident perspectives, emphasizing their understanding of the subcompetencies and their perceptions of the validity and scope of their assessments. The penultimate template then structured the themes in terms of individual program implementation processes, highlighting the variation and commonalities across programs in terms of their assessment, CCC meeting, and feedback processes. The final template integrated the perceptions and attitudes of residents, faculty, and program leadership into these 3 processes, treating these stakeholder perspectives as moderating forces that affected both the processes themselves as well as the resultant outcomes.

In finalizing the template, members of the panel were consulted to ensure that the themes and descriptions were representative of the interview data. This final approval process included representatives from each of the 4 specialties.

In total, 44 individuals from 16 programs participated in the study. Participants were from 5 EM, 3 FM, 4 IM, and 5 pediatrics programs. Details of the participating programs and the interviewees within each program are shown in Table 1. Fifteen of the 16 programs completed the preinterview survey. Table 2 shows each program's responses to selected survey items.

The final template emphasized 3 domains of Milestone implementation: assessments, CCCs, and feedback. This template accounts for program variation in processes, contextual moderators, and reported outcomes for each of the 3 domains. Tables 3, 4, and 5 describe some of the subprocesses, moderating factors, and outcomes within each domain. Brief descriptions of early experiences through the transition implementation stages and later stages of implementation are outlined in the tables and presented alongside illustrative quotes.

For all 3 domains, the early stage of implementation was defined by programmatic challenges, including large resource investments without returns as well as low levels of engagement among residents and faculty. The transition stage of implementation was characterized by modest positive outcomes in terms of resident development, assessments, and CCC meeting efficiencies, as well as increased engagement. The later stage of implementation corresponded with routine use of the specialty-specific Milestones subcompetencies by faculty and CCCs. This included increased faculty skill and consistency in assessing residents and providing feedback. Programs also adopted continuous quality improvement approaches to their assessment, CCC meeting, and feedback processes in this stage.

Table 3 describes assessment process themes, moderating forces, and outcomes reported by study participants. Programmatic processes included assessment approaches, tools, and learner self-assessment. Faculty engagement levels, attitudes of program leadership, and perceived valuation of the Milestones were identified as forces that moderated the effectiveness of the identified processes. Outcomes of assessment system implementation included differences in coverage of the 6 general competencies as well as the varying resource burdens experienced by faculty and other assessors.

The early implementation stage of program assessments was defined by difficulties transitioning to a competency-based framework of learner development, burdens on faculty assessors, and negative perceptions of the Milestones by program leadership and faculty. The transition stage was characterized by increasing engagement of assessors and emergence of a common language to describe learner development along with the subcompetency levels. The later implementation stage involved the development and fine-tuning of assessment tools targeted to the expertise and clinical observation opportunities of assessors, reduced faculty burden, and positive attitudes toward the specialty Milestones by faculty. In this later stage, as reported by PDs and CCC chairs, faculty realized the value of using the developmental subcompetency levels to catch struggling residents early, and to provide more objective descriptions of professionalism and communication skills than were available before the Milestones.

Table 4 highlights the CCC process themes, forces, and outcomes. Programs reported differing strategies for their pre-meeting preparation, data review, and decision processes. The reported role of the CCCs also varied among programs, as did CCC membership and the training provided to CCC members. The available institutional support and existing data management systems moderated the effectiveness of these CCC processes. In terms of outcomes attributable to CCC meetings, themes mirrored the assessment outcomes and included program resource expenditure and the extent to which CCCs uncovered specific problems in resident development.

The early stage of CCC meeting implementation was characterized by an absence of pre-meeting preparations, little or no training of CCC members, and perceptions by program leadership that gaps in evaluation data were an inconvenience instead of an opportunity for curriculum or evaluation improvement. In the transition stage, program leaders implemented CCC member training, distributed resident evaluation data before the meeting for member review, and devised visual representation of learner data to increase meeting efficiency. In the later stage, program leaders preassigned resident ratings to anchor learner-specific discussions, and CCC deliberations were summarized to provide context for the feedback given to residents.

The number of residents in specific programs also affected the CCC processes, but program strategies did not fit along an implementation continuum. One CCC chair from a small IM program commented on how the CCC addressed coverage difficulties when a resident required time away from providing clinical care. Similarly, frequency, duration, and setting of CCC meetings differed across programs as a result of contextual differences, and different implementation stages were not apparent.

Feedback processes, forces, and outcomes are described in Table 5. The feedback received by residents differed across programs in terms of content, timing, and directionality. Learner perceptions of feedback validity as well as individual receptivity were found to influence the effects of received feedback. Resident development and reported instances of individual behavior changes emerged as outcomes of the feedback processes.

In the early implementation stage, resident feedback was inconsistent, devalued by both faculty and residents, and generally not actionable. Feedback in the transition stage was better received and resulted in individual behavior changes among residents. The later stage of program implementation was characterized by consistent, frequent, objective, and actionable feedback to residents, where faculty and program leaders utilized the Milestones to provide context to enable residents to understand and take ownership of their developmental trajectory.

Generally, when examples were given from the early implementation stage, interviewees were describing past experiences or outcomes, either in the first year of Milestones reporting or when they first joined the residency program. A few programs reported persistent implementation challenges at the time of the interviews.

The transitional stages outlined in the tables signaled increasing familiarity with the new competency-based framework. Certain programs adopted practices that accelerated this familiarity, including consistent and brief reintroductions of the subcompetencies at the beginning of CCC meetings as well as concise reviews of the purpose of the Milestones and the developmental trajectory before delivering feedback to residents.

The later stage of implementation was often described by program directors, CCC chairs, and residents in aspirational terms. Many interviewees, including residents, responded to the interview questions with explanations of the long-term goals for their programs or for their specialty communities. Several programs did report specific successes, where certain processes were understood, valued, and executed by most of the faculty and the residents within the program.

Resident interview transcripts proved invaluable in illustrating the learner experience across the assessment, CCC, and feedback domains. Most resident interviewees did not report observing the ways in which assessment data reach the CCC meeting and how these data are used by CCC members to make Milestone ratings and resident progression decisions. Still, many voiced a desire to be more involved in the design and implementation of the assessment and feedback processes. Others reported an appreciation for the structured framework to guide their development and even allow them to advocate for teaching or observation germane to specific subcompetencies. “Seeing where my weaker points were, and then also getting affirmed in where my stronger points were, being able to ask specifically at the start of a rotation, ‘I need to work on my procedural-based competencies in this area,' or ‘I need to work on this specific thing, do you mind setting time aside to help me with this area over the next 2 weeks?'” –IM PGY-3 Resident

In this study of 16 residency programs, varying programmatic assessment, CCC, and feedback strategies were placed along implementation continua. The individual process components, moderating factors, and outcomes were organized as to whether they were characteristic of the resource-intensive early stage, the emerging efficiencies of the transition stage, or the fine-tuning activity of the later stages of implementation.

In our sample, certain programmatic implementation strategies were associated with reported positive outcomes. As discussed in many implementation frameworks, programs that accounted for contextual factors, such as program size and institutional support, and employed an iterative quality improvement approach to implementation also reported relatively low resource burdens, good resident and faculty engagement, and increased efficiency across each of the implementation processes.^15–19  For example, programs that tailored assessment forms according to the expertise and observation opportunities of raters across clinical rotations also reported improved completion and accuracy. Further, programs that iteratively checked in with faculty and CCC members periodically also reported increased faculty engagement and an emerging sense of faculty ownership over assessment and feedback processes.

We believe that the descriptions of the implementation processes can inform residency program leaders as to the various stages they will experience and help them progress more efficiently and effectively toward the later stages described in the continuum. Unfortunately, many attending physicians and residents remain skeptical of the validity and value of utilizing the competency-based Milestones framework to provide context for residency training. Residency program leadership, in concert with national leaders in medical education and within the specialty communities, must communicate the benefits of competency-based medical education and disseminate strategies and practices that both foster faculty and resident engagement as well as decrease program and assessor burden.

Future Milestone research should apply this implementation continua or other implementation frameworks across a more comprehensive set of processes and could query the possible differences among medical, surgical, and hospital-based specialties.

There are several limitations to this study. While we discovered new themes for the thematic template, we did not reach our a priori goal of 24 programs. The results are likely influenced by non-response bias, with additional themes potentially missed. We did interview 44 subjects, however, involving program directors, CCC chairs, and residents. The recruitment occurred over a longer period than anticipated; meaning programs were likely at different stages of Milestones implementation when they were being interviewed.

Another limitation of this analysis is that it focuses solely on the perceptions and experiences of the interview participants. While respondents were asked about their perceptions of the experiences of faculty members and non-interviewed residents, their own experiences likely influenced their responses.²⁰  Also, the self-reported survey data collected prior to the interviews were limited by the fact that they were not independently verified and were subject to self-reporting bias.²¹ 

Using template analysis to analyze program director, CCC chair, and resident interview transcripts, we found that the implementation of assessment, CCC meeting, and feedback processes were moderated by contextual forces and resulted in varying outcomes across EM, IM, pediatrics, and FM programs. These processes, moderating forces, and outcomes can be characterized across 3 distinct stages on an implementation continuum. Based on interview transcripts, we characterized the early stage of implementation by resource challenges, the transition stage by increased efficiency and engagement, and the later stage by skillful stakeholder execution and iterative efforts to fine-tune assessment, CCC, and feedback processes.

The authors would like to thank Lisa Conforti, MPH, for her contributions to this manuscript.