Aligning In-Service Training Examinations in Plastic Surgery and Orthopaedic Surgery With Competency-Based Education

The 6 competencies were implemented by the Accreditation Council for Graduate Medical Education (ACGME) in 1999, followed by the development of the Next Accreditation System and the adoption of competency-based assessments in the Milestone Project in 2013 and 2014.^1,2 

The milestones for each specialty were developed with input from key stakeholders, including Review Committees and the member boards of the American Board of Medical Specialties, faculty, and residents. The milestones reflect the knowledge and skill competencies for trainees in each specialty. Validity evidence and reliability have been demonstrated for some of the core specialties.^3,4 

An important evaluation tool used across specialties is the annual in-service training examination (ITE). These multiple-choice examinations are advantageous in that they assess specialty content areas, have high reliability, and are graded through an automated system.^5–7  Yet, studies have shown variable results in the correlation of performance on ITEs with clinical performance and certifying examinations.^8–11  Assessment through direct observation, clinical simulation, and multi-source assessments like the milestones provide important complementary information beyond the ITE.^12–14 

With the implementation of milestones and competencies, ITEs now are accompanied by competency-based systems that consolidate resident assessments that add to a more complete picture of resident performance. However, while residents receive formative evaluations of both ITEs and the milestone-based assessments, there is no existing method that correlates performance on ITEs to competency-based assessments.

We aimed to determine the distribution of milestones and competencies on ITEs, and integrate and align ITEs within the framework of competency-based education.

We identified ACGME specialties that publish the entire content of their ITEs, including answer keys and explanations. Although several specialties provided partial information, only plastic surgery (the Plastic Surgery In-service Training Examination, PSITE) and orthopaedic surgery (the Orthopaedic In-training Examination, OITE) make the whole examinations available.

We independently reviewed and performed content analysis to define themes in question stems, and used rules to assign milestones within this framework.^15,16  As each milestone has an associated competency, the following rules were used to assign a competency: (1) questions that required management of patient disease, procedural complications, decision-making, or interpreting clinical and diagnostic information fell under the competency of patient care; (2) questions that required anatomic knowledge, guidelines and classification systems knowledge, or genetic associations and syndromic constellations were considered medical knowledge; and (3) for the remaining competencies, milestones were specific enough to determine the appropriate assignment based on the question stem. Application of our rules for coding resulted in no ITE question appearing to have more than 1 associated milestone.

Questions on the PSITE and OITE were assigned a milestone from the 36 milestones in plastic surgery and the 41 milestones in orthopaedic surgery.^17,18  The 2016 PSITE consisted of 250 questions, and the 2014 and 2015 PSITEs had 200 questions. Ten questions on the 2014 OITE were not published and were excluded from scoring. Given the disease-specific nature of the Orthopaedic Surgery Milestones, a number of ITE questions could not be coded to a milestone because they tested a pathology not documented by the milestones, and were coded as other.

This study was determined exempt from requirement for Institutional Review Board approval.

There was an uneven distribution of milestones across both ITEs (provided as online supplemental material). Of the 36 Plastic Surgery Milestones, 9 were represented in 52% (341 of 650) of examination questions. On the OITE, 7 of 41 Orthopaedic Surgery Milestones were represented in 51% (201 of 394) of questions, and 5 milestones were not represented. Detailed information for both specialties is provided as online supplemental material.

Of the competencies, patient care was tested by 62% (403 of 650) of PSITE and 59% (233 of 394) of OITE questions; medical knowledge was tested in 34% (222 of 650) of PSITE and 31% (124 of 394) of OITE questions. Interpersonal and communication skills had the lowest representation on both the PSITE and the OITE (0.3% and 0.5%, respectively). While patient care and medical knowledge were equally distributed, the other competencies varied significantly (figure 1).

Our analysis found that the ITEs for plastic surgery and orthopaedic surgery focus primarily on patient care and medical knowledge. Combined, the 2 competencies represented 96% (625 of 650) of PSITE and 91% (357 of 394) of OITE questions. In this study, the OITEs included significantly more questions emphasizing the other 4 competencies (OITE = 9%; PSITE = 4%), in part through a dedicated section covering professionalism and systems-based practice in 2013.

Our analysis also revealed an uneven distribution of milestones in the ITEs, with a quarter of the milestones representing more than 50% of examination questions, while some milestones were not tested. Less than half (48%, 394 of 815) of OITE questions mapped to any Orthopaedic Surgery Milestone (figure 2).

Formulating ITEs for their representation of the 6 competencies may enhance their utility as evaluative metrics, and aligning ITE domains with milestones may improve integration of these formative tests with competency-based education. Considering the labor-intensive process to develop and evaluate the milestones for evidence of validity, they serve as a useful blueprint for writing ITEs.^19–24  By using such a framework, ITE scoring may be better integrated with clinical evaluations using the milestones, and it may be useful to compare faculty-rated milestone performance with objective ITE results. A step toward this goal would be to crosswalk milestones and competencies with ITE questions.

This study has limitations. First, it represents a review of 3 years of ITEs in 2 available specialties. Second, as reviewers were not blinded to the study questions, there may be inherent subjectivity in assigning milestones to ITE questions.

To better define the role of ITEs in competency-based education, future studies could compare resident milestone evaluations to milestone-coded performance on the ITE. Referencing a milestone and competency for each ITE question would be a useful addition to see how performance on ITEs fits into a competency-based framework.

Despite the shift toward competency-based evaluation in residency training, ITEs for plastic surgery and orthopaedic surgery are not well integrated with ACGME competency-based assessment domains. The ITEs for orthopaedic surgery and plastic surgery tested a minority of the milestones for each specialty, and focused predominantly on patient care and medical knowledge.