Winning Paper
Early Detection of Residents at Risk of Failure Using a Keyword Specific Algorithm
Introduction: Literature suggests that specific keywords included in summative rotation assessments might be an early indicator of abnormal progression or failure.
Objective: This study aims to determine the possible relationship between specific keywords in rotation assessment forms and subsequent failure or abnormal progression. The ultimate goal was to create a functional algorithm to identify residents at risk.
Methods: A database of 41,618 rotation assessment forms from 3292 residents was used. Since the database contained low occurrence of residents with training difficulties, a classification rule was constructed by recursive partitioning using CART methods, and a loss function to optimize sensitivity. Family medicine residents (1129) were analyzed separately from specialties (2163).
Results: The 2 analyses yielded similar results. Sensitivity specificity, positive predictive value, and negative predictive value were approximately 75%, 95%, 50%, and 99%, respectively, for both analyses. Approximately 75% of residents with failure to progress were detected for both family medicine and medical specialties.
Conclusions: Rotation evaluation keywords can be used to identify residents with training difficulties. The low positive predictive values may be a reflection of supervisors documenting poor performance but unwilling to mark the end-of-rotation assessment either as “difficult progression” or “failure.” Classification and regression trees were helpful to identify pertinent keywords, and the algorithm from the CART model can be implemented in our electronic assessment system to detect future residents at risk. A prospective analysis will be needed to test the validity of the algorithm in our current resident cohort.
G. Tremblay, M. Gregoire, J. Maziade
Université Laval, Québec, QC
An Empirical Examination of Consistency in Programs' Milestones Rating Over Time Using a Novel Random Coefficients Model
Introduction: Since December 2013, the ACGME has accumulated milestones data on residents' performance from programs. Programs must include Clinical Competency Committee (CCC) judgement as part of the assessment process. The validity of the longitudinal data depends on how consistently programs apply their rating standards over time.
Objective: This study investigated the consistency of milestone judgements.
Methods: We examined December milestone ratings of first-year residents from a single specialty (for illustrative purposes). Radiology data from 2013 (1170 residents from 179 programs), 2014 (1185 from 182), 2015 (1228 from 185), and 2016 (1228 from 191) were used. The program-level average milestone rating was regressed on year of assessment (TIME) using a random coefficients model for each of the 12 subcompetencies. The TIME coefficient was considered an indicator of whether programs maintained consistency in rating or became more lenient (positive coefficient)/stringent (negative coefficient) over years.
Results: No significant positive TIME slope was found for any subcompetency. The TIME slope values were significantly negative for 6 of the 12 subcompetencies (50%).
Conclusions: The findings indicate that, overall, programs maintain their rating consistency or tend to become more stringent in rating the first-year residents over time. This could be an indication that programs and CCCs become more comfortable in making milestone ratings with experience (ie, no evidence of “grade inflation”). The analytic model developed in this study could be used to (1) provide information for improving rating processes; (2) modify any subcompetencies with problematic language; and (3) investigate the consistency in CCCs' ratings in other specialties.
K. Yamazaki, E. Holmboe, S. J. Hamstra
Accreditation Council for Graduate Medical Education, Chicago, IL
Sharing and Shifting Responsibility: Clinician Educators' Strategies for Coping With Underperformance and Failure
Introduction: Moments of underperformance or failure are inherent in medical practice: sustaining a career in medicine relies on being resilient. To promote wellness, we need to understand how resiliency develops in medical education and practice. If we can understand what enables seasoned clinicians to successfully navigate struggle, we may reveal strategies for supporting struggling learners.
Methods: Using constructivist grounded theory, we interviewed 28 specialist consultants about their experiences with underperformance or failure. We used constant comparative analysis to identify themes.
Results: Participants' experiences with struggle ranged from catastrophic patient errors and academic failures to frequent, smaller moments of interpersonal conflict and work-life imbalance. In telling their stories, participants sometimes either shared personal responsibility for underperformance with external factors or shifted accountability to patients or institutions. In some instances, sharing and shifting seemed to deflect blame. More often, however, participants seemed to accept personal responsibility while simultaneously sharing and shifting accountability to make sense of underperformance or failure. Paradoxically, participants perceived learners who used this strategy as lacking in insight.
Conclusions: Participants demonstrated the protective and functional value of distributing responsibility for underperformance and failure. Sharing and shifting may be an element of reflection and resilience; recognizing external factors may provide a way to gain perspective and to preserve the self. However, this strategy challenges educators' assumptions that learners who deflect are avoiding personal responsibility. Our findings raise questions about what it means to be resilient, and how assumptions about learners' responses to failure may affect remediation for underperforming learners.
K. A. LaDonna1, S. Ginsburg2, C. Watling1
1Western University, London, ON; 2University of Toronto, Toronto, ON
Editor's Note: The following are the Top 5 Resident Papers selected by the JGME and the Royal College of Physicians and Surgeons of Canada for the 2016 International Conference on Residency Education meeting in Québec City, Canada. A full listing of submitted abstracts appears online (http://www.jgme.org/page/ICREAbstracts). Underlined author names indicate presenting author at the conference.
Winning Paper
Evaluation of a Resident-Led Teaching Program for the MRCPCH Clinical Examination in Wales, United Kingdom
Introduction: Pediatric training in the United Kingdom requires all residents to complete the membership of the Royal College of Paediatrics and Child Health (MRCPCH) clinical examination. Wales had the lowest MRCPCH clinical examination pass rate among residents when compared with other regions in the United Kingdom.
Objective: This resident-led quality improvement project sought to establish a structured, sustainable program for residents undertaking the MRCPCH clinical examination. Outcome measures included residents' self-reported learning experience and a successful outcome in the examination.
Methods: The program was implemented in September 2012, coordinated in each round (3 per year) by a lead resident. PDSA #1: 6-week consultant-led teaching program developed. PDSA #2: Senior residents invited to teach on the program. PDSA #3: Number of teaching sessions increased from 12 to 25. PDSA #4: WhatsApp messaging set up to communicate with residents. An evaluation questionnaire was sent to all residents who attended teaching in each examination sitting.
Results: A total of 218 teaching sessions were delivered by 32 residents and 30 consultants between September 2012 and October 2016. Qualitative analysis of participants' comments showed that all residents found the teaching useful, and that it effectively met their learning needs. Following the introduction of the program, pass rates for the MRCPCH clinical examination in Wales improved dramatically: from 37% in 2013, 47% in 2014, 74% in 2015, and 64% in 2016. Pass rates for residents in Wales are now above the national average.
Conclusions: Coordinated resident-led teaching programs represent a powerful quality improvement tool for residents undertaking postgraduate clinical examinations.
S. Bihari1, K. Greenwood1, Z. Howard2
1Morriston Hospital, Swansea, Wales; 2University Hospital Wales, Cardiff, Wales
Communications Skills of Residents Applying in the PGY-4 Canadian Medicine Subspecialty Match: What Do Reference Letters Actually Tell Us?
Introduction: Having strong communication skills is an important quality in prospective subspecialty trainees. Referees for applicants in the Canadian Medicine Subspecialty Match (MSM) for internal medicine residents are specifically asked to comment on a candidate's communication skills. Our research has demonstrated that reference letters are indiscriminate, and do not provide accurate portrayals of candidates.
Objective: We sought to determine what exactly referees were commenting on when they wrote about communication skills. To do this, we qualitatively analyzed the comments related to communication skills from 730 reference letters.
Methods: An independent research assistant reviewed 730 reference letters from Canadian Resident Matching Service (CaRMS) MSM (identifying information redacted), and extracted comments related to communication skills. A thematic analysis of these comments was performed to look for themes, patterns, and gaps.
Results: In the 730 letters, 575 comments regarding communication skills were found. Two researchers examined these comments for themes with very good agreement. Referees referred to the following areas (in descending order of frequency): interactions with patients (41%), clinical presentations (37%), personal qualities (29%), formal presentations (9%), and ability to be concise (8%).
Conclusions: Our examination of comments related to communication skills demonstrated a significant variation in the areas emphasized by referees. Interestingly, emphasis on interactions with patients was not universal, and even fewer focused on the ability to synthesize information. This reiterates that further guidance should be given to physicians writing letters for candidates, especially when commenting on communication skills.
D. Chopra, M. Joneja, L. Hookey
Queen's University, Kingston, ON
Development and Evaluation of a PGY-1 Competency-Based Anatomy Rotation for Radiology Residents
Introduction: As medical schools reduce the hours of anatomy teaching, residents in anatomy-intensive residency programs, like radiology, must independently acquire the anatomy knowledge they need to achieve competency as early as possible in their training.
Objective: The purpose of this educational initiative was to develop and evaluate a 4-week competency-based, self-directed anatomy rotation for junior residents.
Methods: Seven postgraduate year 1 (PGY-1) radiology residents completed a 4-week rotation of radiological anatomy. The objectives were developed from standards, senior residents, and expert opinion; the competency-based curriculum included self-directed modules. Pre- and postcourse tests were administered and test scores were compared using an unpaired t test. In addition, PGY-1 residents completed a course evaluation and survey regarding their anatomy knowledge and anatomy exposure prior to completing the course.
Results: Out of the 25 points available, the average pretest score was 10.79 ± 2.78 (range, 8–16.5) and the average posttest score was 21.64 ± 2.23 (range, 18.5–25). This difference was statistically significant (P < .0001). On average, PGY-1 residents reported receiving less than 10% of dedicated radiological anatomy teaching prior to residency, and felt unprepared for the anatomy required in residency. Overall, residents felt more confident in looking at images after completing the self-directed radiological anatomy course.
Conclusions: This study demonstrates the need for dedicated radiology training for radiology residents beyond what is offered in medical school. In addition, this study reveals how it is possible to create a simple self-directed course for radiology residents that significantly improves their anatomy knowledge.
K. E. Darras, R. Spouge, C. Mar, C. Hague, B. Forster, S. Chang
University of British Columbia, Vancouver, BC
Are They Ready? Organizational Readiness for Change in Clinical Teaching Teams Analyzed by Multilevel Modeling
Introduction: The field of postgraduate medical education (PGME) is continuously changing as a result of social demands and advancing educational insights. Change experts contend that organizational readiness for change is a critical precursor for successful implementation of change initiatives.
Objective: Our aim was to assess readiness for change in clinical teaching teams in regards to a recent curriculum change, the implementation of competency-based medical education.
Methods: Specialty Training's Organizational Readiness for curriculum Change (STORC), a questionnaire to measure organizational readiness for change in educational teams was administered among hospitals in the Netherlands. Additionally, change-related behavior was measured using the “behavioral support-for-change” measure. Results were analyzed using multi-level modeling.
Results: In total, 836 clinical teaching team members were included in this study: 288 (34%) trainees, 307 (37%) clinical staff members, and 241 (29%) program directors. Of all respondents, program directors generally had the highest scores on readiness for change, and showed more supportive behavior (P < .05). Overall, items regarding whether the program director had the authority to lead scored higher compared to the other scales. On the other end, the subscales “management support and leadership,” “project recourses,” and “implementation plan” showed the lowest scores in all respondent groups.
Conclusions: Change in PGME is mainly coordinated by the program director with relatively little guidance from an appropriate change model and implementation strategies. The results of this study reinforce the need for change management support in change processes in PGME in order to enhance efficiency in the process itself as well as to improve chances for success.
L. Bank1, M. Jippes2, J. Leppink3, C. den Rooijen4, S. mi. van Luijk3, A. Scherpbier3, F. Scheele1
1OLVG, Amsterdam; 2Erasmus MC, Rotterdam; 3Maastricht University, Maastricht; 4KNMG, Utrecht
How Changing Quality Management Influenced PGME Accreditation: A Focus on Decentralization and Quality Improvement
Introduction: Evaluating quality of postgraduate medical education (PGME) through accreditation is common practice worldwide. An appropriate accreditation system is important, since accreditation may have substantial consequences.
Objective: This study aims to map out how changing views on educational quality and quality management have affected the design of the Dutch PGME accreditation system.
Methods: To chart the historical development of the Dutch PGME accreditation system we conducted a document analysis of accreditation documents spanning the past 50 years, and a vision document outlining the future system. Template analysis technique was used to identify the main elements of the accreditation system.
Results: Four themes in the Dutch PGME accreditation system were identified: (1) objectives of accreditation; (2) PGME quality domains; (3) quality management approach; and (4) actors' responsibilities. The major shifts have taken place in decentralization of actors' responsibilities, and quality improvement in the quality management approach. Quality improvement originated in the current system, and the emphasis on quality improvement will further increase in the future. The formal accreditation documents of the past 50 years expanded enormously, which led to increased bureaucracy. Therefore, the future system is intended to decrease the amount of standards, and to focus on measurable quality output.
Conclusions: The 4 themes could enhance international comparison and exchange of ideas for the design of accreditation systems. There is an urgent need for international perspectives and evidence about the effectiveness of the distinct elements of accreditation.
N. Akdemir1, K. Lombarts2, E. Paternotte2, B. Schreuder3, F. Scheele2
1OLVG Teaching Hospital, Amsterdam, Netherlands; 2Academic Medical Center, Amsterdam, Netherlands; 3The Royal Dutch Medical Association (KNMG/RGS), Utrecht, Netherlands
Author notes
Editor's Note: The following are the Top 3 Research in Residency Education Papers selected by the JGME and the Royal College of Physicians and Surgeons of Canada for the 2016 International Conference on Residency Education meeting in Québec City, Canada. A full listing of submitted abstracts appears online (http://www.jgme.org/page/ICREAbstracts). Underlined author names indicate presenting author at the conference.