Milestones Next Steps: Results of a Feedback Questionnaire

The Milestones Development team at the Accreditation Council for Graduate Medical Education (ACGME) has periodically surveyed the graduate medical education (GME) community and volunteers, during and after development of the specialty-specific Milestones 2.0, to check with members of each specialty on the implementation of the new Milestones. As we approached the conclusion of Milestones 2.0 development, it was decided to have broader survey dissemination for all specialties. Although still only 10 questions, 2 major changes were made to the quality assurance feedback questionnaire: (1) It solicited comments and impressions from the approximately 26 000-member email distribution list that receives the ACGME e-communications rather than only those who volunteer to be respondents; and (2) It had a more summative focus meant to be the start of the upcoming program evaluation stage of Milestones, rather than soliciting formative input for any further changes to Milestones 2.0. The survey was open for 6 weeks in the fall of 2022 and resulted in 215 responses, which, after clearing the data of collection errors and detectable duplications, produced 211 valid responses from 27 specialties across the GME community. At the time of distribution, email and survey tracking metrics were not available to produce a meaningful response rate. Respondents provided general descriptions of their specialties and specific roles, their program enrollment, and the membership of their Clinical Competency Committees (CCCs). They were then asked open-ended questions that allowed them to describe any challenges with Milestones 2.0 and to make suggestions for the program as a whole. The intent of this 10-item questionnaire was to collect data that will direct conversations on change and help guide this first step, as well as further program evaluation projects that explore the relationships between Milestones and the GME community. The results of the qualitative analysis of this questionnaire are based on those who chose to respond to the ACGME e-communication request for feedback. The subsequent findings should not be generalized to the GME community or be made to represent thoughts or reactions of any subpopulation therein.

Of 211 valid responses, roughly half (n=105, 49.8%) identified as program directors (Table 1) from a dropdown list of options (Question [Q]3), and 46 (21.8%) indicated that they were their CCC chair (Q4). The most represented ACGME core specialties (Q1) were internal medicine (n=46, 21.8%), family medicine (n=35, 16.6%), pediatrics (n=24, 11.4%), and surgery (n=17, 8.1%). Additionally, 80 (37.9%) respondents self-identified a subspecialty (Q2) in the open text question following the specialty question.

The responses to (Q5) “Indicate the number of residents or fellows you are currently assessing in your program using the Milestones” ranged from 2 to 170. This question was asked to obtain an approximate number of residents and fellows being assessed, and it was beyond the purview of this project to check a respondent's accuracy regarding this data point.

When asked to describe the roles within their current CCC membership (Q6), 181 (85.8%) of the respondents listed their committee as having a program director, 162 (76.8%) indicated a program coordinator, 161 (76.3%) listed an assistant or associate program director, and 199 (94.3%) specified a core faculty member (Table 2). While the survey question had a “public member” option, no respondents indicated as such.

Two open-ended questions guided the major focus of the survey: (Q8) “Indicate any challenges you have experienced with Milestones 2.0, or related concerns,” and (Q9) “What changes or resources would you like to see in the Milestones?” Finally, respondents could self-identify under (Q10) “Enter your email and full name to be notified of future Milestones feedback opportunities,” which produced a list of volunteers for future research projects.

Researchers from the Milestones department used emergent, in vivo coding to categorize the open-ended responses describing challenges and providing suggestions. While responses reflected specific and individual experiences, some patterns emerged from similar data during analysis that helped paint a picture of the challenges facing the GME community around Milestones. Guided by a heuristic inquiry methodology, the research team performed the coding in the MAXQDA software (VERBI Software GmbH) to label the open-ended responses and look for the patterns using a constant comparative analysis approach.¹  The most prevalent challenges and suggestions are discussed below with exact verbiage lifted from the responses.

The first prominent idea uncovered in the response data involved faculty use and interpretation of the Milestones. Particularly, many respondents mentioned faculty not being fully invested in the Milestones as an assessment tool and that faculty may consider the goals of the Milestones as too lofty or unachievable.

• “…getting enough evaluations from faculty members to meaningfully complete them.” (internal medicine–critical care)

• “Progress in the clinical skills rated in the Milestones is highly non-linear, and nuanced. It is hard to accurately rate trainees.” (neurology–vascular neurology)

• “Teaching faculty to embrace the milestones.” (osteopathic neuromusculoskeletal medicine)

Another common topic from the data highlighted issues with specific subcompetencies, namely those that measure advocacy or well-being. Some respondents called out a few Milestones that were difficult to observe as well as directly assess, particularly those speaking to system management and the physician role in the health care system. Similarly, there were comments that described the Milestones as being too vague or subjective and too difficult to transfer to the provided scale. One comment stated: “Based on how these are worded many of our residents hit a level 3 at PGY-2 and then stay there despite continuing to improve in their abilities.” (internal medicine)

Data also reflected issues with the amount of time required to adequately assess residents and fellows on each Milestones subcomeptency. This may be related to other comments where respondents felt there were too many Milestones to assess or that the Milestones were too long or wordy. “Too many Milestones! Takes way too long to complete them twice a year for all of our residents.” (diagnostic radiology)

Some respondents described challenges with transitioning from Milestones 1.0 to 2.0, such as making adjustments to their assessment practices and software tools to reflect the changes. One comment pointed out that it would get easier: “There are some that don't correlate perfectly with the new milestones, so this made tracking resident improvement over time difficult. This should get easier once we use the 2.0 version for all 3 resident years.” (pediatrics)

There were also a few respondents who felt as though the Milestones were not applicable to actual clinical practices. For example: “I find the wording of the Milestones 2.0 incredibly hard to apply to real world resident performance.” (internal medicine) and “The verbiage is still extremely convoluted and often does not apply to real life.” (family medicine) Others indicated that the Milestones do not capture the context of the resident's “rotation experience.” (family medicine)

Similarly, data from the survey described issues with few opportunities for residents to be assessed or actually observed within specific Milestones contexts. They also spoke of other challenges: “Bias from prior milestone evaluations. Some CCC members may evaluate residents that they have not personally worked with over the past 6 months.” (surgery–surgical oncology) and “Finding ways for residents to achieve, and for us to observe and document, some milestones elements (ex. SBP2, SBP3, PROF1, ICS1), especially those in the upper levels.” (pathology–surgical pathology)

There remained some responses that did not fall into a particular pattern or theme but still highlighted concerns worth mentioning:

• Integration with curriculum management systems

• Alignment with curriculum

• Comfort level with Milestones 2.0

• Difficulty in separating inpatient and outpatient assessments

• The burden put on the coordinator

• Issues with scheduling the CCC

The survey respondents' suggestions were just as varied as their challenges, but some themes were evident. The most prominent area for suggestions centered around the need for clearer applications and more examples for the Milestones levels, either beyond the existing supplemental guidebooks or an updated version of them.

• “The supplemental guide was indispensable when trying to figure out the observable behavior that maps to the milestones...more examples of how residents can fulfill aspects of the milestones sub-competencies, curricular tools, etc.” (family medicine)

• “More examples of situations that meet the milestones would be appreciated.” (family medicine)

• “Update of specialty supplemental guides to include more assessment examples of hard to assess subjective content.” (pathology-cytopathology)

Another common thread among the suggested changes was the inclusion of either additional Milestones or a new focus reflected within Milestones. Particularly, these responses spoke to skills development in leadership, teaching, and scholarly activities.

• “Address teaching. Address areas of improvement.” (family medicine)

• “Milestones that incorporate leadership skills sets.” (internal medicine–pediatrics)

• “Scholarship and research would be fantastic to include.” (psychiatry)

Respondents also focused their suggestions on language simplification and applying more realistic language as well as condensing Milestones, making them shorter or even reducing the overall number of them.

• “Simplify the wording, make them easy to apply to real world observations, get more input from faculty members and APDs/PDs using them on a regular basis.” (internal medicine)

• “Make each step in progress tied to very defined, objective, and measurable standards.” (diagnostic radiology)

• “Shorten them. For years in other specialties I have worked in, faculty constantly complain about length of milestones.” (obstetrics and gynecology)

Another theme identified the need for more faculty development and education that address “goals, changes, and interventions to address failure to progress.” (surgery–acute care surgery) Responses also spoke to the need for evaluation forms that would be easy to use and implement into the learner's record automatically. Some respondents suggested using the Milestones as the assessment tool: “Evaluation tools would be helpful, not just the milestones but also some suggested way to assess. Or just let us use the milestones as assessment tools, because the rubrics are built out well enough now that they could be used as direct assessments.” (anesthesiology)

Individual comments spoke to a wide variety of suggestions, such as the following:

• Using entrustable professional activities with Milestones

• Having automatic or online processes to record Milestones achievements

• Accesssing residents' past [Milestones] scores more easily

• Providing resources specifically for new program directors and associate program directors

• Standardizing how to assess Milestones across specialties

This survey was developed to collect general, voluntary feedback with only 2 focused questions for challenges and suggestions as an early look informing program evaluation for Milestones. While the data are not representative of any group or subgroup within GME, the broad and organic analysis shows that the GME community feels that the Milestones program should continue its focus on providing a transparent and concise assessment tool that does not overburden faculty, program directors, or the residents and fellows.^2-5  The survey results also indicate the need to reiterate the validity of the Milestones assessments in the harmonized competencies as well as in patient care and medical knowledge.^6,7  There is evidence supporting more emphasis on developing a concerted effort by faculty and CCCs to utilize the tools and resources that the Milestones program provides. Some survey respondents also expressed the need for Milestones to continue efforts for developing subcompetencies that could integrate with other specialty-specific assessment tools.

The Milestones department has already researched and taken steps to provide GME faculty with access to resources, such as guidebooks (Milestones, CCCs, assessment, etc) and faculty development courses,^8,9  and currently offers periodic courses to address faculty understanding and usage of the Milestones assessments. The department also has developed supplemental resources and has explained the rigor that goes into developing the validity and reliability of the tools.¹⁰  The data also spoke to the importance of keeping current and future resources visible, available, and accessible to the GME community.

The ACGME Milestones team would like to thank those who took part in the survey and provided their valuable feedback. The resulting suggestions and conversations will give the Milestones department the foundations to better inform their upcoming evaluation research on the Milestones program. Over the next few years an intensive review of the Milestones will occur. This research will delve into the development process, content, and implementation within programs. Data will be collected through focus groups, interviews, content analyses, and targeted surveys. If you are interested in sharing your opinion, please send an email with your name, specialty, and/or subspecialty to [email protected].