ICRE Top Research Papers Open Access

Introduction: Interviewer gender is said to influence scoring in interview situations. We investigated the relationship between candidate and interviewer gender and interview scores in multiple mini-interviews for the 2018 AOA 21 Australian Orthopaedic Training Program selection process.

Methods: Selection for the AOA 21 Orthopaedic Training Program in Australia is a national process, and 2018 was the first year in which similar numbers of male and female interviewers took part in the national interview process for selection. At least one member of each 3-person interview panel was female. Deidentified results for the multiple mini-interview component of the selection process were reviewed for all candidates. Results were compared for candidate gender and interviewer gender differences.

Results: Two hundred nineteen junior physicians (181 male and 38 female) applied for orthopaedic training in 2018. A total of 144 candidates were interviewed: 123 male (85%) and 21 female (15%). One hundred forty-four interviewers participated in the multiple mini-interviews across 5 states: 79 male (55%) and 65 female (45%). Female and male interviewer scores for each candidate had high correlation (r = 0.83). Female interviewers scored the same male candidates (P < .001) and female candidates (P < .01) significantly lower than male interviewers. Female candidates scored higher than male candidates across all interview stations (P < .001).

Conclusions: There is a high correlation among interviewers' scores for selection for individual candidates, with gender differences identifiable in interviewers' scores. Female interviewers score both male and female candidates significantly lower than male interviewers.

I.W. Incoll, B. Balhatchet

Australian Orthopaedic Association, Sydney, NSW, Australia

Introduction: Programmatic assessment has at its core the ongoing production, review, and feedback of multiple low-stakes assessments. In postgraduate medical education, clinical competency committees (CCCs) serve as the locus of these processes. Despite the proliferation of recent scholarship on CCCs much remains unknown about their inner workings. The aim of this constructivist grounded theory study was to explore the internal processes of CCCs.

Methods: Our sample consisted of 18 CCC meetings in 8 postgraduate programs, selected purposively to represent a range of postgraduate programs. Data were collected through nonparticipant observations and semistructured interviews (n = 18) and analyzed iteratively using a constant comparative method.

Results: CCCs' work extends beyond the review and collation of assessment data. Considerable additional work was required to ensure that data are interpreted appropriately, and that they drive learning in the way programmatic assessment intends. The invisible work of CCCs is organized into 4 types: recalibration of their program of assessment (ie, revision of forms); facilitating trainee exposure to required experiences (ie, purposeful scheduling); tracking and analyzing trends in data (eg, identifying areas of difficulty); and providing individualized coaching to trainees (eg, targeted feedback). Despite the extensive work that CCC members engaged in participants reported finding value in the formalization of CCCs as an assessment strategy.

Conclusions: Our ongoing research suggests that CCCs engage in extensive invisible work that is integral to the successful implementation of programmatic assessment. The resourcing of CCCs may have profound implications for translating programmatic assessment theory into practice.

R. Pack, S. Cristancho, C. Watling, S. Chahine, L. Lingard

Western University, London, ON, Canada

Introduction: Although not recommended, the ACGME Milestones reporting forms are often used by individual faculty to rate residents' competence. Used in this way, the ratings may be more susceptible to interpersonal biases than when combined with other faculty ratings during a clinical competency committee (CCC) meeting. A recent study of 8 emergency medicine (EM) programs reported gender differences in individual faculty milestone ratings that implied systemic misjudgment of competence that was equated to extended training for women.

Methods: The present study investigated national milestones data generated by CCCs in EM (n = 1340 residents), internal medicine (IM; n = 7062), pediatrics (n = 2634), and diagnostic radiology (DR; n = 846). A multilevel spline regression model was used to determine whether gender was predictive of milestone rating trajectories over the course of residency. Gender differences in rating at time of graduation were also compared.

Results: In all 4 specialties, male and female resident trajectories were almost identical on all subcompetencies: maximum mean difference across all specialties = 0.10 milestone units (out of 5.0). At time of graduation, 4 of 22 subcompetencies were statistically significantly higher for men in IM, and 1 of 12 subcompetencies in DR, while in pediatrics, women were rated higher than men on 6 of 14 subcompetencies. No significant gender difference was found in EM on any subcompetency.

Conclusions: It appears the mediating effect of the CCC group process may reduce the probability for gender bias in milestone decisions regarding progression or graduation. The educational implications require ongoing follow-up, given the importance of this issue.

S.J. Hamstra, K. Yamazaki, E. Holmboe

Accreditation Council for Graduate Medical Education, Chicago, IL, United States

Editor's Note: The following are the Top Resident Research Papers selected by the JGME and the Royal College of Physicians and Surgeons of Canada for the 2019 International Conference on Residency Education meeting in Ottawa, Canada. A full listing of submitted abstracts appears online (http://www.jgme.org/page/ICREAbstracts). Underlined author names indicate presenting author at the conference.

Introduction: It is commonplace for North American trainees to interrupt their surgical residency programs to complete 2 or more years of research training. The impact of this practice on surgical education is unknown. As the University of Toronto has both the largest general surgery and surgeon scientist training programs, we have the opportunity to evaluate performance of residents in both clinical and research streams serially on annual in-training clinical aptitude examinations.

Methods: We collected anonymized scores obtained at both written and oral annual in-training aptitude examinations by all general surgery residents at the University of Toronto from 2011 to 2016, inclusively. We compared performance of residents prior to, during, and following their research training both to themselves and to their peers in clinical training.

Results: At the junior resident level, future enrollment in research training was associated with higher examination performance (P = .003). Annual scores plateaued during research training (P = .50), while scores of residents who continued in clinical training improved year over year (P = .009). Research stream resident examination scores remain stagnant after 1 year then improve in the second year back in clinical training (P = .90 and P = .00007, respectively). Scores obtained in the final year of residency training do not significantly differ between the 2 groups of residents.

Conclusions: We demonstrate that interruption of clinical activities for research training results in “hibernation” of aptitudes on annual in-training examinations, with an eventual catching up after at least 2 years of resumed clinical training. This may inform residency program designs.

A. Mikhail¹, A. Connor¹, N. Ahmed²

Introduction: Frequent workplace-based assessments (WBAs) are critical to the success of competency-based medical education (CBME). After transitioning to a CBME-based training program in 2017, we discovered a large variation in numbers of learner-triggered WBAs between residents. Current evidence is inconclusive regarding whether WBAs lead to better resident performance. The purpose of our study was to explore the relationship between the number of assessments a resident obtains and performance of entrustable professional activities.

Methods: Data included all WBAs obtained by first-year internal medicine residents between July 1, 2018 and February 20, 2019 (22 residents, 516 assessments). Residents were divided into 3 groups based on numbers of assessments obtained (low 0–20, medium 21–30, high 31). Global entrustment scores were compared among the groups using analysis of variance.

Results: Mean global entrustment scores for the low, medium, and high assessment number groups were 4.03 (CI 3.89–4.17), 4.25 (CI 4.17–4.34), and 4.27 (CI 4.2–4.34), respectively. Residents with low assessment numbers had a significantly lower global rating compared to residents in the medium and high groups (P < .01). The difference in ratings between the medium and high groups was not significant.

Conclusions: Our results demonstrate a correlation between the number of WBAs and mean global entrustment scores. We identify a number of hypotheses regarding why this may be the case and correlate them with existing evidence on feedback-seeking behavior and WBA effectiveness. Understanding these relationships advances our understanding of how to improve assessment seeking among residents, thus enhancing the efficacy of assessment in CBME.

S. Madan, D. Taylor, S. Gauthier

Queen's University, Kingston, ON, Canada

Introduction: Competence committees (CCs) struggle with incorporating professionalism issues into resident progression decisions. This study examined how professionalism concerns influence individual faculty decisions about resident progression using simulated CC reviews.

Methods: In 2017, the investigators conducted a survey of 25 program directors of Royal College emergency medicine residency training programs in Canada and those faculty members who are members of the CCs (or equivalent) at their home institution. The survey contained 12 resident portfolios, each containing formative and summative information available to a CC for making progression decisions. Six portfolios outlined residents progressing as expected and 6 were not progressing as expected. Further, a professionalism variable was added to 6 portfolios, evenly split between those residents progressing as expected and not. Participants were asked to make progression decisions based on each portfolio.

Results: Raters were able to consistently identify a resident needing an educational intervention versus those who did not. When a professionalism variable was added, the consistency among raters decreased by 34.2% in those residents progressing as expected, versus increasing by 3.8% in those not progressing as expected (P = .01).

Conclusions: When using an unstructured review of a simulated resident portfolio, individual reviewers can better discriminate between trainees progressing as expected when professionalism concerns are added. Considering this, educators using a CC in a competency-based medical education program must have a system to acquire and document professionalism issues to make appropriate progress decisions.

S. Odorizzi

University of Ottawa, Ottawa, ON, Canada

Introduction: Postgraduate medical programs in Russia have been accepting citizens or medical graduates from culturally related backgrounds. These programs have mostly been structured around building physician capabilities in clinical specialties. Kazan State Medical University's (KSMU) newly established residency program in internal medicine adopted a comprehensive CanMEDS competency-based curriculum, which accepts foreign citizen residents. We assessed the difference in patient-physician communication skills between native Russian and foreign origin residents.

Methods: A survey assessed communication skills of 28 native Russian and 11 foreign residents involving 190 cognitively intact patients at KSMU participating hospitals. Patients participated in the study, if managed 3 or more days by a trainee during their first year of training. Communication skills were scored by patients utilizing CanMEDS recommended assessment tools. Data were analyzed by SPSS 23.0, using the Mann-Whitney U test (P ≤ .05) to assess rank differences.

Results: We identified significant differences in 13 of 14 patient-physician communication aspects where foreign residents ranked higher than native in greeting, demonstrating respect, listening attentively, in achieving shared understanding, encouraging raising questions, fostering shared decision-making, planning next steps, demonstrating responsibility, demonstrating sincerity, spending sufficient time, empathy, eye contact (P < .05), and overall perceived medical care (P < .05).

Conclusions: Overall, foreign origin residents scored significantly higher than native Russian residents on patient-physician communication skills in 13 of 14 tested aspects and perceived medical care. Diversity in medical residency is potentially beneficial in enhancing communication skills. Further assessment is required to identify associated factors of patient-physician communication and perceived medical care differences.

M.C. Browne, S. Alaskari, A.N. Maksudova, T.V. Ryabova

Kazan State Medical University, Kazan, Russia

Introduction: Entrustable professional activities (EPAs) are assessed through instruments that produce both quantitative and qualitative data. Promotion decisions require group decision-making by clinical competency committees (CCCs). The volume of raw assessment data may be too great for CCCs to review in its entirety for each promotion decision. To facilitate interpretation, data must be collated, analyzed, and displayed in reports that facilitate decision-making. This study seeks to understand how assessment data can be reported to support CCCs in making valid promotion decisions.

Methods: Using a design-based research methodology, template assessment reports were developed using Messick's validity framework. Mock reports were created to represent well-performing, borderline, and poorly performing residents. Through semistructured individual interviews, reports were presented to 10 CCC members from 2 training programs at the University of Toronto. Interviews sought to understand how CCC members interact with reports and identify which elements are used to support valid promotion decisions. Data were analyzed using a framework analysis using Messick's validity framework.

Results: Reports designed using Messick's validity framework provide CCC members with evidence to support promotion decisions. While analysis is ongoing, data from 5 interviews indicate that CCC members use contextual information and ensure narrative comments are congruent with numerical scores. Detailed analyses from the complete dataset will be presented to provide insights into how CCC members interpret validity evidence and help inform valid report design.

Conclusions: Results from this study will inform development of assessment reports and contribute to understanding of how CCCs use validity evidence to support promotion decisions.

R. Luther¹, R. Cavalcanti¹, S. Heeneman²

Editor's Note: The following are the Top Quality Improvement Papers selected by the JGME and the Royal College of Physicians and Surgeons of Canada for the 2019 International Conference on Residency Education meeting in Ottawa, Canada. A full listing of submitted abstracts appears online (http://www.jgme.org/page/ICREAbstracts). Underlined author names indicate presenting author at the conference.

Introduction: Evidence-based guidelines recommend promoting sleep in the intensive care unit (ICU), yet many patients experience poor sleep quality. We identified poor sleep quality for patients in the ICU at Kingston Health Sciences Centre (KHSC), measured by the Richards-Campbell Sleep Questionnaire (RCSQ). We aimed to improve ICU patient sleep quality from 53.7 to 70, measured by the RCSQ, by June 2019.

Methods: Patients included were > 18 years old, admitted to the KHSC ICU, with a Richards Agitation-Sedation Scale (RASS) ≥ -2. The study followed an interrupted time-series framework of quality improvement. Root cause analysis utilized qualitative descriptive analysis. Two interventions were developed: inclusion of sleep quality discussion in morning nursing report and a patient doorway poster. The primary outcome measure was RCSQ score, measured by bedside nurse. Process measures included intervention adherence and quantitative measurements of light and sound intensity. Balancing measures included number of overnight codes as a marker for unrecognized clinical decompensation and nurse-reported sleep medication use.

Results: The initiative is ongoing at the time of abstract submission; significant improvements in RCSQ have not yet been appreciated. Baseline data collection (December 2018) revealed a mean RCSQ score of 53.7. Major local sleep barriers include nursing stigma associated with less actively managing a patient and delayed downgrading of patient illness severity.

Conclusions: Interventions targeted to improving sleep quality are ongoing. A rigorous root cause analysis has identified novel barriers to sleep promotion. This underscores the importance of understanding local culture in the design of quality improvement initiatives.

S. Douglas, H. Hobbs, G. Digby

Queen's University, Kingston, ON, Canada

Introduction: Scrotal doppler ultrasound (DUS) is an adjunct for the diagnosis of testicular torsion (TT) when clinical assessment is equivocal. Our group identified that acquiring a DUS results in a 48-minute delay. Point-of-care ultrasound (POCUS) may be used to negate this delay. The purpose of this study was to develop and evaluate a scrotal POCUS curriculum for urology and emergency medicine (EM) residents.

Methods: Experts from urology, EM, and radiology collaborated in the Delphi method to design a practical and didactic curriculum for scrotal POCUS. The study followed a pre-post design. The OASUS scale was used to evaluate for competency in scrotal POCUS skills. Residents were also asked to rate their comfort and confidence with scrotal POCUS before and after the curriculum.

Results: Twenty-four urology (n = 12) and EM (n = 12) residents participated in a scrotal POCUS curriculum. Pre-post testing showed significant improvements in knowledge (6.3 versus 8.0, P < .001) among the residents. Residents were more comfortable (pre 0.6 versus post 3.6, P < .001) and confident (pre 1.0 versus post 2.1, P < .001) utilizing scrotal POCUS to assess for TT after the curriculum (5-point Likert scale). Lastly, 23 out of the 24 residents were rated as competent at performing scrotal POCUS.

Conclusions: Our scrotal POCUS curriculum was effective and acceptable to both urology and EM residents. This skill may potentially reduce delays in diagnosing TT and improve testicular salvage rates.

L. Stringer¹, E. Chan¹, F. Myslik¹, H. Razvi¹, S. Dave¹, P. Wang¹, A. Jiang², S. Cocco², G. Brahm³

Introduction: Physician burnout peaks during residency, eroding new physicians' wellness. Despite this harmful endemic, programs and institutions are falling behind, though ideally we should be forming the frontlines, creating a culture that promotes well-being. The RESPITE initiative (Resilience in the Era of Sustainable Physicians: An International Training Endeavour) is a voluntary curriculum, founded by McMaster's psychiatry program, that works to lead this cultural shift.

Methods: RESPITE encompasses 3 components: an e-curriculum, peer-support rounds, and quarterly newsletters. The e-curriculum integrates 2 core learning dimensions: Know Yourself and Integrate New Lifestyles, which focus on building awareness and providing resilience tools. The rounds provide a confidential space for learners to debrief and process difficult topics related to life as a resident. The newsletters offer wellness strategies and serve as a reminder that physicians are not alone in their experiences of burnout.

Results: RESPITE is a pilot project, launched in 2019. The short- and medium-term educational goals are evaluated by outcome measures obtained for 12 months. Data are obtained through residents' engagement and feedback, and will be utilized to improve the initiative to address wellness needs, understand how best to engage learners, and determine the efficacy of the current methods aimed at improving resilience and well-being of the physician-in-training.

Conclusions: While further trial of RESPITE will determine its utility and role, this project has the potential to improve the quality of life for the resident physician and cultivate a systematic training model that is more attuned to the well-being and sustainability of its physicians.

T. Riddell¹, A. Hategan², K. Saperson², S. Harms²

Introduction: Prior to 2017, core internal medicine residents at the University of Alberta did not have a standardized quality improvement (QI) educational curriculum. Our goal was to use QI principles to develop and implement a QI curriculum by providing internal medicine residents the Evidence-based Practice for Improving Quality (EPIQ) training course.

Methods: Two plan-do-study-act (PDSA) cycles, 1 year apart, consisted of EPIQ course delivery to postgraduate year 1 (PGY-1) to PGY-3 cohorts (110 residents–PDSA1) and then to the next PGY-1 cohort (27 residents–PDSA2). Residents were grouped into teams to work through a QI issue that was subsequently presented at an academic day for evaluation. Residents completed pre- and post-course surveys in PDSA1, as well as pre- and post-course tests in PDSA2, to evaluate knowledge acquisition and curriculum satisfaction.

Results: In PDSA1, 98% of residents felt they had acquired an understanding of QI principles (56% increase) and 94% of PGY-2 and PGY-3 residents preferred this method of learning QI to previous years. During PDSA2, test scores improved from 77.6% to 80%. One hundred percent of residents felt they had acquired an understanding of QI principles (76% increase).

Conclusions: Evaluations and verbal feedback from residents have been overwhelmingly positive. Developing a QI curriculum using validated QI tools highlighted areas of change opportunity enhancing change acceptance and sustainment, further supporting a QI culture within our hospitals and health care system. As more cycles of EPIQ are delivered and more residents become facilitators, it is our aim to have this curriculum sustained by future residents.

C. Collins, P. Mathura, N. Kassam, A. Tapardel

University of Alberta, Edmonton, AB, Canada

Introduction: Pressure ulcers are a serious health care problem for residents in long-term care facilities and also a key quality metric for regulators of these facilities. Three initiatives were introduced at a 128-bed, long-term care facility to improve pressure ulcer prevention.

Methods: Firstly, a quality assurance and performance improvement project and a root cause analysis was conducted to improve the facility's wound care program. Secondly, a digital wound care management solution was adopted and implemented to track wound healing progression and management. Thirdly, the formation of the role a skin integrity coordinator was created to act as a central point of accountability for these wound care-related activities and related performance metrics.

Results: Improvements in pressure ulcer prevention were monitored using publically available quality metrics, as a regulatory requirement, specifically (1) the percentage of long-stay high-risk residents with pressure ulcers, and (2) the percentage of short-stay residents with pressure ulcers that are new or worsened. The pressure ulcer prevalence for long-stay high-risk residents was 12.9%, and upon implementation of these initiatives, the facility saw continued reductions in pressure ulcer prevalence to as low as 2.9% after 1 year, while pressure ulcers for short-stay residents were maintained at 0 throughout this period.

Conclusions: This study highlights the power of effective management combined with real-time data analytics, as enabled by the innovation of digital wound care management, to make significant improvements in the quality of health care delivery.

S. Wang¹, Y. Au², A. Cassata², M. Holbrook³, A. Skeens³, J. Painter³

Introduction: Translating clinical evidence into practice in critical care teaching environments is notoriously difficult, with the combined challenges of 24-hour rosters, frequent staff rotation, and high patient throughput impeding departments' ability to deliver consistent messaging or timely, targeted feedback.

Methods: We demonstrate a novel approach to implementation science in a busy emergency department in Queensland, Australia. Project aims were (1) to reduce intravenous (IV) prescription of metronidazole where oral administration would be appropriate, and (2) to reduce administration of aDT vaccinations to trauma victims where it was not indicated. Outcome measures include analysis of prescriptions, nursing time, expenditure on medications, and patient satisfaction. Our strategy involved 3 components: (1) implementation of a leading practice onboarding, orientation and change management mobile application; (2) an appreciative inquiry approach to clinical education; and (3) real-time automated analytics of department performance against set targets. In this model, the traditional phases of quality improvement and plan-do-study-act, can be undertaken simultaneously on an ongoing basis.

Results: Preliminary results from this ongoing trial (completed by May 2019) show excellent results, with a 21% shift from IV to oral administration of metronidazole within the first 10 weeks of the intervention, improved patient satisfaction as well as marked savings in nursing time and cost of medications.

Conclusions: Preliminary results suggest that this combination of content delivery through a mobile application, an appreciative inquiry approach to staff education, and real-time feedback of analytics is a highly effective and scalable approach to change management in tertiary critical care environments.

R. Pearlman, C. Weston, T. Collins

MedApps, Sydney, NSW, Australia

Introduction: Diagnosis and follow-up of inflammatory bowel disease (IBD) in patients younger than 50 years of age is a widely accepted indication for magnetic resonance enterography (MRE). However, at our institution (St. Joseph's Healthcare Hamilton [SJH]), there has been a gradual increase in the average age of referred patients and in inappropriate indications such as anemia, diarrhea, and abdominal pain. The purpose of this quality assurance study was to determine the referral practices for MRE at SJH and to educate the referring physicians about the appropriateness of the test.

Methods: A total of 150 consecutive MRE examinations from July to October 2017 were retrospectively reviewed. The study date, patient age, clinical indication, and quality of the study were recorded. Indications were categorized based on the pathology in question. A repeat analysis was performed on 150 cases from July to November 2018 post-intervention. An educational letter was drafted to the top 20 referring clinicians to clarify the appropriateness criteria for MRE.

Results: A total of 52 patients (35%) were older than 50 years pre-intervention, compared to 32 (21%) post-intervention (P < .05). A total of 127 studies (85%) were related to IBD, compared to 117 studies (78%) post-intervention (P > .05).

Conclusions: The educational letter as the intervention led to desired reduction in the mean age of the referred patients for MRE. While the IBD-related indication rate was initially high, there was no significant increase post-intervention.

D. Odedra¹, A. Alabousi²

Introduction: The prevalence rate of contrast media (CMs) allergy decreased with the introduction of nonionic CMs and premedication for high-risk patients. Severe adverse reactions were 0.04% in the nonionic CMs examinations and death still occurred. One station regarding CMs allergy was implanted in the postgraduate year 1 (PGY-1) objective structured clinical examination (OSCE) from 2011 in Taichung Veterans General Hospital, and we identified a major gap in anaphylaxis recognition and management. A high-fidelity OSCE based on anaphylaxis guideline was then introduced to improve emergent medication administration and airway management.

Methods: Interprofessional high-fidelity OSCE content was developed through focus group discussion, which included ADR consultants, intensive care unit (ICU) physicians, and radiologists. Each trainee was assessed by 2 assessors with 3 major domains: anaphylaxis medication and airway management by senior physician and leadership by senior nurse. A debrief was conducted immediate after simulation. Residents who were classified as “need direct supervision” in overall performance had to attend an extended course. In-hospital CMs allergy was reported to ADR team and managed independently.

Results: Three hundred thirty-four PGY-1, 110 internal medicine residents, and 19 radiology residents attended the OSCE. Trainee satisfaction to course content was 93%. Radiology residents had better performance in identification of anaphylaxis than internal medicine residents in airway management. In 2016, 130 out of 15,029 patients had CMs allergy. Thirty-seven patients had anaphylaxis and 15 patients were admitted to the ICU. None of them died or had severe morbidity.

Conclusions: Integration of evidence-based practice and high-fidelity OSCE were effective in residents' training and could reduce major morbidity and mortality rate of CMs allergy.

W. Hung, T. Hsieh, Y. Chen

Taichung Veterans General Hospital, Taichung, Taiwan