Background

Quality improvement and patient safety (QIPS) curricula are critical in graduate medical education, yet barriers limit the educational experience and project outcomes.

Objective

To explore the impact of QIPS curricular enhancements and integration of continuous improvement specialists (CIS) by examining the A3 document, the primary project product and surrogate for project quality.

Methods

Since 2009, University of Michigan internal medicine and medicine-pediatric residents participate in a QIPS curriculum, which includes a 4-week group project. In 2016, residency leaders collaborated with CIS staff, non-clinical experts in QIPS with backgrounds in engineering and business, to improve the curriculum. Informed by a needs assessment, the intervention was implemented in 2017 and consisted of a set of enhancements including integration of CIS staff into groups as co-facilitators. In this retrospective cohort study, a blinded reviewer evaluated all available A3 documents before and after the intervention using a quantitative analysis tool.

Results

All residents participated in the curriculum during the pre-intervention (July 2009 to June 2016, n=351) and post-intervention (July 2017 to June 2020, n=148) periods. A total of 23 of 84 (27%) pre-intervention and 31 of 34 (91%) post-intervention A3 documents were available for review. Scores improved significantly for 17 of 23 (74%) A3 items and for 7 of 8 (88%) sections. Mean A3 total scores increased from 29.0 to 47.0 (95% CI 12.6-23.4; P<.001) out of a possible 69.0.

Conclusions

Embedding CIS experts into residency QIPS curricula is associated with improved A3 document quality.

As frontline providers, resident physicians are positioned to recognize opportunities for improvements in patient safety and quality.1,2  The Accreditation Council for Graduate Medical Education (ACGME) mandates resident quality improvement and patient safety (QIPS) education and involvement in health systems improvement efforts within the practice-based learning and improvement (PBLI) and systems-based practice (SBP) competencies.1,3,4 

Graduate medical education programs have implemented QIPS curricula, but barriers remain.5-8  The optimal facilitator type and role to lead resident QIPS projects is undefined.7,9,10  Limited access to institutional data results in underdeveloped projects.7,8  The ACGME Clinical Learning Environment Review site visits have identified uncertainty of residents' role in institutional QIPS efforts.11,12  Lack of alignment between resident projects and institutional priorities leads to fragmented QIPS efforts and less engaged learners.13,14 

In 2016, the University of Michigan internal medicine (IM) and combined internal medicine and pediatric (Med-Peds) residency programs sought to mitigate these barriers by partnering with the Department of Internal Medicine Quality & Innovation Program's continuous improvement specialists (CIS), non-clinical experts in health care QIPS, to evaluate and improve the existing QIPS curriculum. In this retrospective cohort study, we describe the role of the CIS team and specific curricular enhancements, examine the effect of our interventions by comparing residents' pre- and post-intervention A3 documents (one-page reports that summarize QIPS projects) using an A3 assessment tool,15  and share residents' intervention and scholarly project outcomes. We discuss how other institutions may be able to adapt a similar intervention at the local level and examine meaningful outcomes from QIPS training.

Objectives

To explore the impact of embedding continuous improvement specialists (CIS), non-clinical health care improvement experts, into residency quality improvement and patient safety (QIPS) group projects by examining the A3 document, the primary project product, and a surrogate for project quality.

Findings

In this cohort study in which a blinded reviewer used a quantitative analysis tool for A3 scoring, we show that A3 documents improved significantly following the intervention.

Limitations

The retrospective nature of the study may lead to selection bias, and the institutional support required for the intervention may limit generalizability.

Bottom Line

Partnering with non-clinical CIS staff to enhance QIPS curriculum and embed into resident group projects enhances the QIPS education of residents and is associated with improvement in project quality.

Setting, Participants, and Design

The University of Michigan IM and Med-Peds residency programs train approximately 157 and 32 residents each year, respectively, at a suburban tertiary care hospital. The programs' QIPS curriculum was established in 2007 to equip residents with fundamental QIPS concepts.16  Since 2009, the required curriculum has included a foundational didactic series consisting of two 1.5-hour interactive lectures in the first postgraduate year (PGY) of training. This series is followed by a 4-week group project during protected time (approximately 58 hours) on an ambulatory rotation during PGY-2. Each year, approximately 45 IM and 8 Med-Peds PGY-2 residents work in groups of 4 to 5 per 4-week rotation. Residents may voluntarily participate in further QIPS work throughout their training. The longitudinal QIPS curriculum is depicted in the Figure.

Figure

Overview of QIPS Curriculum With Focus on PGY-2 Content

Abbreviations: QIPS, quality improvement and patient safety; PGY, postgraduate year.

Figure

Overview of QIPS Curriculum With Focus on PGY-2 Content

Abbreviations: QIPS, quality improvement and patient safety; PGY, postgraduate year.

Close modal

The focus of this study is the PGY-2 group project experience, which challenges residents to identify a real-world problem and apply foundational QIPS knowledge and skills, through A3 thinking, to this problem. A3 thinking is a structured problem-solving methodology in which a complex problem is analyzed and presented on a single sheet of A3-sized paper (11”×17”). Pioneered by Toyota as part of the Toyota Production System, the A3 is a key component of lean methodology and is commonly used in health care.17  The A3 report includes the problem, background, current state, root cause analysis, improvement recommendations, and implementation plan.18-20  The process of creating an A3 involves observations, interviews, data review, identification of specific performance gaps, and development of recommendations that address root causes of the problem. The 4-week rotation culminates in a presentation of the A3 report to peers, program leadership, and institutional stakeholders. An IM or Med-Peds faculty physician with QIPS experience and training facilitates each project by reinforcing QIPS principles, offering project guidance and feedback, and assessing residents based on ACGME PBLI and SBP Milestones. Each faculty assists 2 resident groups annually, and each has 0.05 full-time equivalent protected time for this ongoing role.

Intervention

The intervention in this study focused on integration of 3 CIS staff into the PGY-2 portion of the curriculum. CIS are business and industrial engineering professionals with experience in health care QI and advanced degrees, training, and certifications in lean methodologies, A3 thinking, quality improvement, and data analysis. CIS staff are funded by the IM department and support the Clinical Experience and Quality program, providing coaching for QIPS projects and facilitating data requests across 13 medical specialties.

Current State Assessment and Pilot Phase

In July 2016, the Quality & Innovation Program partnered with IM and Med-Peds residency leadership and faculty facilitators to evaluate and improve the QIPS curriculum. This initiative followed a QI approach using lean methodologies. A current state assessment included a survey of residents who had completed rotations in the previous 2 years along with direct observations of 7 teams from July 2016 through January 2017.

Faculty facilitators, lacking standardized training, had variable emphases in their teaching and expectations for results. Projects selected by residents tended not to align with institutional priorities.

Several interventions were developed to address the variability in pedagogical methods and expectations, dearth of objective data, and lack of project alignment within the institution. These interventions involved 2 domains: (1) reforming curricular content and resources, and (2) embedding CIS staff into resident groups as co-facilitators alongside faculty. The interventions were piloted with 5 groups in the February through June 2017 rotations. During this pilot, curricular leaders, faculty, and CIS staff reflected on effects of these changes, gathered informal resident feedback, and implemented rapid iterative changes to ensure interventions were achieving program goals and satisfactory to residents.

Post-Intervention Phase

In July 2017, curricular changes were finalized. CIS staff were integrated into groups as outlined in the Box. One CIS staff member co-facilitates each group, requiring 10% effort during the rotation. In conjunction with faculty facilitators, the CIS team uses formal resident feedback and annual stakeholder discussions to inform plan-do-check-act cycles for ongoing curricular enhancements. These include refining the resident feedback survey instrument and revising educational modules and other program materials based on that feedback.

Outcomes

A3 Document Quality

We performed a retrospective review of all available A3 documents since archiving began in 2009. Project documents were included if they resembled an A3 with sections on background, current state, goal, analysis, countermeasures, action plan, and follow-up plan.18-20  The pre-intervention phase spanned rotations from July 2009 to June 2016. The current state assessment and pilot phase were excluded from analysis. The post-intervention phase included project rotations from July 2017 to June 2020, excluding April and May 2020 when the curriculum was paused due to the COVID-19 pandemic.

A member of the study team (D.B.P.) retrospectively reviewed each A3 document and assigned scores using a previously studied 23-item A3 assessment tool, which has good evidence of validity (content validity, construct validity, and high interrater reliability).15  This external, publicly available resource provides a self-instruction package on assessment of A3s and allows practice assessments using example A3s, with the ability to compare assessments against a standard.21  A single individual who has undergone requisite training provides reliable assessment scores for A3s developed by health care professionals across a variety of clinical and operational topics15 ; thus, no additional reviewers were used for A3 scoring in this study. Our reviewer is a CIS staff member from a separate department who has expertise in QI principles and a background in developing A3 documents and teaching QIPS. The reviewer was not a member of any resident groups and was not aware of specific curricular changes or intervention dates. The reviewer was provided A3 documents in a randomized order and was blinded to pre- or post-intervention status and reason for A3 assessment. The A3s were de-identified of names, dates, and terms that would allow for project dating.

Project Outcomes and Dissemination

In the post-intervention phase, CIS staff follow up with resident groups and faculty at the end of each academic year to determine if projects were disseminated as scholarly products (eg, presentation, abstract, journal article). CIS staff also track which groups implemented countermeasures and which projects are connected to departmental initiatives or health system strategic priorities. While outcomes and dissemination are formally checked for the preceding academic year's projects every July, CIS and faculty continually share and track updates on outcomes or dissemination as they arise. Information on outcomes and dissemination is not available for pre-intervention projects, and thus these data cannot be compared.

Analysis

The results of A3 items are summarized as means and standard deviations. Due to the small sample size and non-normality of the data, we compared the pre- and post-intervention A3 results using nonparametric Wilcoxon rank-sum tests. A 2-sided P value of <.05 was considered statistically significant. All statistical analyses were done in SAS 9.4 (SAS Institute Inc).

This study was determined to be exempt by the University of Michigan Institutional Review Board.

All PGY-2 residents participated in the QIPS curriculum. During the pre-intervention period, 351 (100%) residents participated in 84 distinct group projects. During the post-intervention period, 148 (100%) residents participated in 34 distinct projects.

A3 Document Quality

A total of 23 of 84 (27%) and 31 of 34 (91%) project A3 documents were available for review from the pre- and post-intervention periods, respectively (Table). Seventeen of 23 (74%) distinct A3 items showed significant score increases after the intervention (individual score changes shown in the Table). All A3 sections demonstrated significant increases in aggregate scores except for “background.” Following the intervention, mean individual item scores increased from 1.3 to 2.0 (95% CI 0.6-1.0; P<.001) out of a possible 3.0. Mean total A3 scores also increased from 29.0 to 47.0 (95% CI 12.6-23.4; P<.001) out of a possible total of 69.0.

Table

A3 Document Quality as Evaluated by the A3 Assessment Tool

A3 Document Quality as Evaluated by the A3 Assessment Tool
A3 Document Quality as Evaluated by the A3 Assessment Tool

Project Outcomes and Dissemination

Within the post-intervention group, 53 of 147 residents (36%) from 11 of 34 distinct group projects (32%) implemented at least one countermeasure. An example of an implemented countermeasure was a change to electronic health record alerts for updates to blood culture results to facilitate rapid tailoring of antimicrobial agents to specific microorganisms. Additionally, 10 of 34 projects (29%) were connected to ongoing larger initiatives or health care system strategic priorities. For example, one resident group conducted telephone interviews with patients who were recently admitted to the hospital for hepatic encephalopathy. These perspectives informed a separate institutional workgroup that had been tasked with decreasing readmission rates for this patient population. Twenty-two of 147 residents (15%) from 5 of 34 distinct projects (15%) submitted their work to a local, regional, or national conference, and 5 of 147 residents (3%), all within a single project, together submitted one manuscript for publication.

To our knowledge, this retrospective study spanning 11 years is the largest evaluation of a QIPS curriculum to date. In this study, we demonstrate that CIS staff add value when integrated as partners into experiential QIPS resident group projects. We show an association between our intervention and improved A3 products, surrogates for project quality. Our study provides valuable data that inform the ways in which these kinds of curricular interventions can affect the quality of QIPS group project outcomes.

This study affirms and builds on QIPS and educational literature in important ways. Many publications outline keys to successful QIPS curricula that our curricular enhancements have addressed.22-25  These include ensuring foundational education for all residents and providing trained faculty facilitators and adequate and dedicated time for this education. Embedding CIS staff into group projects further enhances these keys to success by deepening project mentors' QIPS knowledge, standardizing educational content, improving access to data for projects, aligning projects with institutional priorities, and providing institutional memory of past and current QIPS efforts. While there are numerous studies that suggest guidance for high-quality curricula, literature describing methods for reliably and objectively evaluating quality of QIPS curricula and projects is lacking, in part because of the uniqueness of each individual curriculum, context, and project. Prior studies involving QIPS curricular evaluation focus primarily on learner satisfaction and knowledge outcomes,6,25-32  while some expand to include future effects on learners (eg, frequency of choosing a career with a QIPS component)28,30  and ability to implement meaningful change.33-36  Our study builds on current literature by moving from Kirkpatrick Level 1 (learner attitudes) to focusing on objective project outcomes including A3 quality (Kirkpatrick Level 2: learning transfer) and countermeasure implementation and dissemination of scholarly products (Kirkpatrick Level 3: impact).37  In the future, it would be helpful to quantify impact on patient health outcomes.

It is notable that all A3 sections except for “background” showed significant improvement after the intervention, and there are several possible explanations. In the pre-intervention period, groups spent a lot of time choosing a topic and researching its background, leaving little time for actionable work. The inclusion of CIS staff allowed residents to engage more efficiently and effectively in structured scientific problem-solving and thus enabled rapid progress throughout the rotation. Additionally, limited access to patient data in the pre-intervention period led groups to more often rely on provider attitude surveys to inform current state assessments. CIS staff helped access and extract electronic health record data, which led to better informed and more robust assessments. CIS staff assisted with project scoping, helped define goals, and provided links to institutional stakeholders, all of which led to higher quality products and alignment with established initiatives. Faculty facilitators often have competing priorities outside of QIPS education, may work within narrow clinical or divisional domains, and may not be attuned to broader QIPS strategies and resources. These factors could contribute to inconsistency in QIPS facilitator teaching. Finally, while “action plan” and “follow-up” sections improved statistically, the magnitude of change was less pronounced than other sections. This could be explained by the fact that implementation of countermeasures is not required and hence it is possible that residents devoted less time to these sections.

Our study has several limitations. The study is retrospective in nature and thus prone to selection bias. Fewer projects were consistently archived in the pre-intervention period, and projects with a concrete product were more likely to be archived than those without one. It is likely that recall bias and social desirability affected the retrospective and subjective report of outcomes and dissemination, particularly since institutional initiatives and priorities were loosely defined. Additionally, we do not have direct comparisons of pre- and post-intervention data on implementation and dissemination. The A3 assessment tool does not assess the higher level outcome of patient impact.15  Institutional resources to support CIS staff and faculty effort may not be feasible for other programs, which may present a barrier to curriculum implementation. Finally, integration of CIS staff into groups resulted in a myriad of concurrent curricular changes, and it is not possible to determine if a specific aspect, several aspects, or the collective intervention impacted the chosen outcomes.

While there is employment cost for CIS staff, this study demonstrates that the investment appears to be beneficial to resident education and is associated with higher quality project products. Programs lacking resources to support CIS staff may consider creating standardized educational sessions and establishing a mechanism for residents to obtain health record data for QIPS projects. Future research should focus on assessing the impact on patient health outcomes.

After partnering with non-clinical CIS staff to enhance the QIPS curriculum and embedding them into resident group projects, we demonstrated positive effects on A3 document quality.

Box Continuous Improvement Specialist Roles in QIPS Curricular Changes and Active Engagement in Resident Group Projects

Curricular Changes Following CIS Staff Involvement

  • Revised course syllabus, educational objectives, and course content around the essential question: How can physicians contribute to high quality and safe patient care?

  • Provided standardized calendar to deliver the content and clarify expectations across the 4 weeks

  • Provided curricular reference materials including A3 templates, survey design tips, and data collection strategies

  • Created central electronic repository that stored past project A3 examples for reference throughout the rotation

  • Standardized the didactic content and alignment with A3 problem-solving thinking by creating educational modules, delivered weekly via flipped classroom model:

    • Module 1: Syllabus review, patient safety culture, problem selection, project scoping

    • Module 2: A3 overview, background, current state, problem statement, goal development

    • Module 3: Root cause analysis, countermeasures, recommendations, implementation

    • Module 4: Project stakeholders, project sustainability, plan-do-check-act, lessons learned

  • Curated and maintained a list of venues in which residents may disseminate their work

  • Co-created a departmental grand rounds series to highlight resident group project findings

Active Engagement in Resident Group Projects

  • Assist with appropriate project scoping to help ensure it can be tackled in the rotation time frame

  • Assist in aligning or connecting resident projects with ongoing departmental and institutional projects and priorities

  • Connect with data specialists for access to patient data and institutional metrics

  • Connect resident team members with representatives from related institutional projects

  • Assist with keeping the group on task throughout the rotation

  • Assist with A3 document development

  • Deliver feedback to resident teams throughout the rotation

  • Administer surveys to residents regarding project satisfaction, evaluation, and feedback

  • Track post-project activity including implementation of countermeasures and scholarly work

Abbreviations: QIPS, quality improvement and patient safety; CIS, continuous improvement specialists.

The authors would like to thank Matthew Johnson (former team member who assisted in early phases of this work), Linda Bashaw (provided data support to resident projects), Drs Jennifer Stojan, Sandro Cinti, David Stewart, and John Gosbee (additional past and present faculty facilitators for the group projects), Drs John Del Valle and Michael Lukela (program directors for internal medicine and medicine-pediatrics residency programs, respectively), Drs Scott Flanders (Vice Chair, Internal Medicine Quality & Innovation Program, 2015-2019) and Roma Gianchandani (Associate Vice Chair, Internal Medicine Quality & Innovation Program, 2019-present), and the internal medicine and medicine-pediatrics residents past, present, and future at the University of Michigan.

1. 
Nasca
TJ,
Philibert
I,
Brigham
T,
Flynn
TC.
The next GME accreditation system—rationale and benefits
.
N Engl J Med
.
2012
;
366
(11)
:
1051
-
1056
.
2. 
Conway
PH,
Clancy
C.
Transformation of health care at the front line
.
JAMA
.
2009
;
301
(7)
:
763
-
765
.
3. 
Weiss
KB,
Bagian
JP,
Wagner
R.
CLER pathways to excellence: expectations for an optimal clinical learning environment (executive summary)
.
J Grad Med Educ
.
2014
;
6
(3)
:
610
-
611
.
4. 
Accreditation Council for Graduate Medical Education.
Common program requirements.
5. 
Djuricich
AM,
Ciccarelli
M,
Swigonski
NL.
A continuous quality improvement curriculum for residents: addressing core competency, improving systems
.
Acad Med
.
2004
;
79
(suppl 10)
:
65
-
67
.
6. 
Ogrinc
G,
Headrick
LA,
Morrison
LJ,
Foster
T.
Teaching and assessing resident competence in practice-based learning and improvement
.
J Gen Intern Med
.
2004
;
19
(5 Pt 2)
:
496
-
500
.
7. 
Weingart
SN,
Tess
A,
Driver
J,
Aronson
MD,
Sands
K.
Creating a quality improvement elective for medical house officers
.
J Gen Intern Med
.
2004
;
19
(8)
:
861
-
867
.
8. 
Coleman
MT,
Nasraty
S,
Ostapchuk
M,
Wheeler
S,
Looney
S,
Rhodes
S.
Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum
.
Jt Comm J Qual Saf
.
2003
;
29
(5)
:
238
-
247
.
9. 
Patow
CA,
Karpovich
K,
Riesenberg
LA,
et al
Residents' engagement in quality improvement: a systematic review of the literature
.
Acad Med
.
2009
;
84
(12)
:
1757
-
1764
.
10. 
Myers
JS,
Bellini
LM.
Leveraging the continuum: a novel approach to meeting quality improvement and patient safety competency requirements across a large department of medicine
.
Acad Med
.
2018
;
93
(9)
:
1321
-
1325
.
11. 
Weiss
KB,
Wagner
R,
Nasca
TJ.
Development, testing, and implementation of the ACGME Clinical Learning Environment Review (CLER) program
.
J Grad Med Educ
.
2012
;
4
(3)
:
396
-
398
.
12. 
Weiss
KB,
Bagian
JP,
Nasca
TJ.
The clinical learning environment: the foundation of graduate medical education
.
JAMA
.
2013
;
309
(16)
:
1687
-
1688
.
13. 
Wong
BM,
Baum
KD,
Headrick
LA,
et al
Building the bridge to quality: an urgent call to integrate quality improvement and patient safety education with clinical care
.
Acad Med
.
2020
;
95
(1)
:
59
-
68
.
14. 
Tess
A,
Vidyarthi
A,
Yang
J,
Myers
JS.
Bridging the gap: a framework and strategies for integrating the quality and safety mission of teaching hospitals and graduate medical education
.
Acad Med
.
2015
;
90
(9)
:
1251
-
1257
.
15. 
Myers
JS,
Kin
JM,
Billi
JE,
Burke
KG,
Harrison
RV.
Development and validation of an A3 problem-solving assessment tool and self-instructional package for teachers of quality improvement in healthcare
.
BMJ Qual Saf
.
2022
;
31
(4)
:
287
-
296
.
16. 
Kim
CS,
Lukela
MP,
Parekh
VI,
et al
Teaching internal medicine residents quality improvement and patient safety: a lean thinking approach
.
Am J Med Qual
.
2010
;
25
(3)
:
211
-
217
.
17. 
Toussaint
JS,
Conway
PH,
Shortell
SM.
The Toyota production system: what does it mean, and what does it mean for health care? Health Affairs Forefront. Published April 6, 2016. Accessed March 1, 2023.
18. 
Lean Enterprise Institute.
A3 problem-solving—a resource guide.
19. 
Ghosh
M.
A3 process: a pragmatic problem-solving technique for process improvement in health care
.
J Health Manag
.
2012
;
14
(1)
:
1
-
11
.
20. 
Shook
J.
Managing to Learn: Using the A3 Management Process. Lean Enterprise Institute Inc;
2008
.
21. 
University of Pennsylvania Perelman School of Medicine Center for Healthcare Improvement & Patient Safety.
A3 problem solving.
22. 
Philibert
I,
Gonzalez del Rey
JA,
Lannon
C,
Lieh-Lai
M,
Weiss
KB.
Quality improvement skills for pediatric residents: from lecture to implementation and sustainability
.
Acad Pediatr
.
2014
;
14
(1)
:
40
-
46
.
23. 
Jones
AC,
Shipman
SA,
Ogrinc
G.
Key characteristics of successful quality improvement curricula in physician education: a realist review
.
BMJ Qual Saf
.
2015
;
24
(1)
:
77
-
88
.
24. 
Massagli
TL,
Zumsteg
JM,
Osorio
MB.
Quality improvement education in residency training: a review
.
Am J Phys Med Rehabil
.
2018
;
97
(9)
:
673
-
678
.
25. 
Wong
BM,
Etchells
EE,
Kuper
A,
Levinson
W,
Shojania
KG.
Teaching quality improvement and patient safety to trainees: a systematic review
.
Acad Med
.
2010
;
85
(9)
:
1425
-
1439
.
26. 
Starr
SR,
Kautz
JM,
Sorita
A,
et al
Quality improvement education for health professionals: a systematic review
.
Am J Med Qual
.
2016
;
31
(3)
:
209
-
216
.
27. 
Boonyasai
RT,
Windish
DM,
Chakraborti
C,
Feldman
LS,
Rubin
HR,
Bass
EB.
Effectiveness of teaching quality improvement to clinicians: a systematic review
.
JAMA
.
2007
;
298
(9)
:
1023
-
1037
.
28. 
O'Leary
KJ,
Fant
AL,
Thurk
J,
et al
Immediate and long-term effects of a team-based quality improvement training programme
.
BMJ Qual Saf
.
2019
;
28
(5)
:
366
-
373
.
29. 
Oyler
J,
Vinci
L,
Arora
V,
Johnson
J.
Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules
.
J Gen Intern Med
.
2008
;
23
(7)
:
927
-
930
.
30. 
Duello
K,
Louh
I,
Greig
H,
Dawson
N.
Residents' knowledge of quality improvement: the impact of using a group project curriculum
.
Postgrad Med J
.
2015
;
91
(1078)
:
431
-
435
.
31. 
Aboumatar
HJ,
Thompson
D,
Wu
A,
et al
Development and evaluation of a 3-day patient safety curriculum to advance knowledge, self-efficacy and system thinking among medical students
.
BMJ Qual Saf
.
2012
;
21
(5)
:
416
-
422
.
32. 
Tartaglia
KM,
Walker
C.
Effectiveness of a quality improvement curriculum for medical students
.
Med Educ Online
.
2015
;
20
(1)
:
27133
.
33. 
Rosenbluth
G,
Burman
NJ,
Ranji
SR,
Boscardin
CK.
Development of a multi-domain assessment tool for quality improvement projects
.
J Grad Med Educ
.
2017
;
9
(4)
:
473
-
478
.
34. 
Leenstra
JL,
Beckman
TJ,
Reed
DA,
et al
Validation of a method for assessing resident physicians' quality improvement proposals
.
J Gen Intern Med
.
2007
;
22
(9)
:
1330
-
1334
.
35. 
Peiris-John
R,
Selak
V,
Robb
G,
et al
The state of quality improvement teaching in medical schools: a systematic review
.
J Surg Educ
.
2020
;
77
(4)
:
889
-
904
.
36. 
Kiger
ME,
Bertagnoli
T.
A project-based, resident-led quality improvement curriculum within a pediatric continuity clinic
.
MedEdPORTAL
.
2018
;
14
:
10738
.
37. 
Frye
AW,
Hemmer
PA.
Program evaluation models and related theories: AMEE guide no. 67
.
Med Teach
.
2012
;
34
(5)
:
e288
-
e299
.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.