Pediatrics Residents' Confidence and Performance Following a Longitudinal Quality Improvement Curriculum Open Access

Quality improvement (QI) training during residency allows trainees to develop necessary skills to deliver high-quality patient care. Recognizing the importance of these skills, the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements¹  include the competencies of practice-based learning and improvement and systems-based practice. Although residents are required to complete QI training, few studies address the optimal curriculum for physician training²  and the elements that prepare physicians to incorporate QI learning into future practice.

Few longitudinal resident QI curricula include first-year residents^3,4  or embed a QI coach. Many residency programs incorporate QI education during a 1-month ambulatory rotation^5–7  or an elective clinical rotation,⁸  include only second-year and third-year residents,^9,10  or utilize longitudinal projects with a series of residents on monthly rotations.¹¹ 

The Center for Advancing Pediatric Excellence (CAPE) developed a longitudinal curriculum employing a multimodal format to improve pediatrics residents' confidence in their QI knowledge and skills and competence in QI methodology. The curriculum provided experiential learning, including conducting successive personal and practice-based QI projects, and culminated in the joining/leading of a multidisciplinary QI team, which is consistent with the ACGME's Clinical Learning Environment Review expectations for quality and safety improvements.¹²  The purpose of this study was to evaluate the curriculum components involved in developing residents' confidence in QI knowledge and skills and to examine its relationship to QI competence.

Participants were 3 successive cohorts of pediatrics residents (N = 36) who completed the program's QI requirement from 2009 through 2014.

The CAPE curriculum, a longitudinal experience based on the Model for Improvement^13,14  and the adult learning theory emphasizing experiential learning,¹⁵  is provided as online supplemental material. Requirements included a personal improvement project (PIP)¹⁶  designed to teach QI basics with application to a personal, nonclinical activity; a reflective practice project (RPP) to identify and address gaps in clinical practice; resident membership in an ongoing inpatient or ambulatory QI project team; and leadership of a component of the QI project. Resident projects were chosen using both institutional strategic planning and resident interest.

The CAPE staffing included physicians and nonphysicians and involved pediatrics faculty and 36 pediatrics residents. Support staff positions and job descriptions were adapted from existing positions and pay scales within the organization. An overview of CAPE staffing, including job titles, full-time equivalent status, education/experience, and job functions, is provided as online supplemental material. A QI coach and data analyst provided the coaching, with interactions individualized to each resident, as well as education, data presentation, data analysis, and project management tailored to suit project needs. Material costs were primarily associated with obtaining QI publications for training and QI resources.

To examine the curriculum's effects on confidence in QI knowledge and skills, residents completed surveys both at program entry and completion. At the end of postgraduate year (PGY) 3, residents rated their ability to understand and use the Model for Improvement before and after the PIP and RPP. Survey components are available as online supplemental material. At the PGY-3 assessment, residents ranked the educational value of individual curricular components.

The Quality Improvement Knowledge Application Tool (QIKAT),⁶  a commonly used, adult-focused method for QI curricular evaluation, may have limitations in pediatrics settings as the adult case content may not resonate with pediatrics residents. The CAPE staff developed the CAPE pediatric QI assessment scenario (PQIAS) and scoring, based on the QIKAT,^6,17  to assess pediatrics residents' competence in applying QI methodology (available as online supplemental material). After assessing the QIKAT for suitability, we created the PQIAS to emphasize demonstrating competence with familiar content and piloted it for attending and chief resident feedback. The PQIAS describes 1 outpatient scenario that prompts residents to formulate a QI project with aims, goals, measures, and 1 change idea or plan-do-study-act cycle. The PQIAS was administered at program completion. The PQIAS was evaluated by 2 blinded raters with QI expertise (C.C., L.N.). Residents received points for correct answers in individual PQIAS sections, which were summed for a total score.

The Carolinas HealthCare System provided Institutional Research Board approval.

Ratings on individual scale items were summed and averaged for total scale scores for confidence in QI knowledge and skills. Paired t tests were used to compare residents' precurriculum and postcurriculum scores and pre-post PIP and RPP experiences. Analyses of residents' modal responses were conducted on rankings of the educational value of specified curricular components. Kendall tau-b was used to examine the interrater reliability of PQIAS scores. Following a median split of PQIAS scores, residents above and below the median were compared by independent t tests on confidence in QI knowledge and QI skill scales. All statistical analyses were conducted using SPSS Statistics version 21 (IBM Corp, Armonk, NY).

Comparisons of confidence in QI knowledge and skills ratings were available for 31 residents. Four residents did not complete the preassessment, and 1 transferred to another program before the postassessment. All other comparisons were based on 35 residents who completed all surveys. Residents' perceived confidence in QI knowledge improved from a mean of 2.7 (standard error of the mean [SEM] = 0.19) to 7.5 (SEM = 0.23, P < .001). Residents' perceived confidence in QI tools improved from a mean of 2.3 (SEM = 0.22) to 6.3 (SEM = 0.30, P < .001; table 1A). Comparisons of pre-post PIP and RPP ratings showed that residents reported significantly greater understanding of the Model for Improvement and abilities to use QI skills and tools (table 1B).

Total scores on the PQIAS ranged from 8 to 20 with a median of 15.5; this also reflected the 80% mark of total possible points. The 2 independent raters showed high interrater reliability on this measure (tau-b = 0.893, P < .001). Compared to residents with scores below the median (n = 17), residents with scores above the median (n = 18) had significantly greater confidence in QI skills (above median: mean = 6.9, SEM = 0.37; below median: mean = 5.7, SEM = 0.40; t(33) = 2.20; P < .035), but not in QI knowledge (above median: mean = 7.8, SEM = 0.27; below median: mean = 7.2, SEM = 0.33; t(33) = 1.54; P < .13). Postcurriculum confidence scores were higher for QI knowledge than QI skills for residents in both the lower competence (t(16) = 3.83, P < .001) and the higher competence (t(17) = 3.50, P < .003) groups.

Individual coaching consistently received a high modal ranking across the 3 years. Conducting a PIP and being a team member were also ranked as having high perceived educational value for PGY-1 and PGY-2 residents, respectively. Leading a team received the highest modal rating of educational experiences for PGY-3 residents. Didactics, Institute for Healthcare Improvement modules, and suggested readings were consistently ranked as having much lower perceived educational value.

The CAPE curriculum significantly improved residents' perceived confidence in QI knowledge and skills. The 2 experiential learning projects imbedded in this curriculum, the PIP and RPP, were rated as significantly improving residents' perceived understanding of specific aspects of QI methodology. A unique component consistently ranked highly for its educational value was the QI coach's ability to individualize education and provide support for team activities.

The longitudinal nature of the curriculum allowed for the benefit of reflection on learning and the development of confidence in QI skills and knowledge over time. The goal of the CAPE curriculum was to incorporate QI into daily resident work.^4,18  Several challenges arose regarding integration of a longitudinal program into a monthly schedule. Residents spent approximately 40 hours over the 3 years on QI projects. As there was no protected time for QI work, time was integrated into the overall block schedule via individualized coaching sessions at resident requested times, typically during elective or outpatient months and before or after the noon conference. Team meetings were arranged around resident schedules, and project management was coordinated by the QI coach. All project teams were multidisciplinary. Although residents were involved in generating plan-do-study-act cycles, much of the testing was performed by front-line staff in the patient care setting.

Faculty QI expertise was initially limited, with only 5 faculty members having formal QI training. These faculty members invested approximately 30 hours per resident project over the course of 3 years. A core group of faculty, self-identified as QI champions, embraced the CAPE curriculum and learned QI methodology side by side with residents. As learners themselves, most faculty functioned as content experts, not QI experts, on resident projects. Resident projects (table 2) often were part of larger, organizationally driven projects. A few residents successfully generated an independent project. Despite time limitations, residents and faculty rated participation in the CAPE curriculum as a positive experience.

An important component of the evaluation of this curriculum was whether the educational components and confidence generated by the QI curriculum resulted in demonstrable QI competence, as measured by the PQIAS. Half of the residents obtained at least 80% of the possible PQIAS total points at program conclusion, suggesting that they were able to translate their perceived confidence into an ability to develop a QI project. Half of the residents did not achieve the 80% mark of success, suggesting that improved confidence does not necessarily signify competence. Residents above the PQIAS median reported significantly greater confidence in QI skills than residents below the median; no difference was found between resident groups in confidence of QI knowledge when related to QI competence. Resident competence thus appears associated with confidence in using such skills as creating a data collection sheet and/or generating a run chart or histogram, not confidence in knowledge, per se. Of interest, the lowest-rated components of the curriculum were those associated with didactic and nonexperiential learning or knowledge.

Curricular elements that were ranked as having the highest educational value were the interactive learning experiences, which required guided, active participation. While residents indicated that they had more confidence in their QI knowledge than QI skills, the latter scale reflecting active learning experiences was the one associated with demonstrated competence on the PQIAS.

Limitations of this educational innovation include the PQIAS scenario's sole focus on ambulatory care. It also is possible that residents' confidence improvements were affected by other factors, such as other experiences that may enhance confidence. However, in our institution, there are few opportunities for residents to participate in QI besides the CAPE curriculum.

Obtaining institutional funding for a comprehensive effort may be a challenge. In our case, a Duke Endowment grant provided funding to develop the CAPE QI curriculum and hire staff. Based on success in starting and sustaining improvement projects aligned with institutional goals, the Department of Pediatrics currently supports the CAPE staff. Residents have participated in more than 25 projects to date, and many projects have advanced to sustainability. In other settings, institutional support may hinge on the magnitude of the positive effects from resident participation on QI teams.

QI coaching support may not be available in all settings, but existing resources may be secured for projects strategically aligned with the institutional mission. Several residency programs at our institution have adapted and implemented components of the CAPE curriculum to align with available resources.

Areas for future research include whether guided resident involvement on QI projects affected outcomes, understanding how QI coaching can affect physicians in training and postresidency QI efforts, expansion of scenarios to assess QI competence, and the relationship to the acquisition of QI knowledge and skills at the completion of residency. Further studies should evaluate applicability of the curriculum to other residency training programs and other health professions training.

A 3-year longitudinal, experiential QI curriculum increased pediatrics residents' confidence in QI knowledge and skills. Demonstration of QI competence was not related to confidence, except in the area of acquisition of QI skills. The QI coaching and team leadership received the most favorable resident ratings. Our innovative approach to QI training is a feasible solution to developing QI leadership skills and embedding QI training into daily work.¹⁹