Adapting SQUIRE 2.0 to Create a Quality Improvement Evidence-Based Medicine Critical Appraisal Tool (QI-EBM-CAT) for Graduate Medical Education Trainees

Evidence-based medicine (EBM), broadly defined as the judicious integration of current best evidence into the delivery of medical care, is a common component of graduate medical education (GME) curricula.¹  The Accreditation Council for Graduate Medical Education (ACGME) endorses Practice-Based Learning and Improvement as 1 of 6 core competencies in their requirements for GME programs; teaching EBM practices to residents is an assessment milestone within this domain.²  Critical appraisal of clinical research literature, including randomized controlled trials (RCTs), meta-analyses, and systematic reviews, is a central skill in EBM and is routinely taught in the context of a traditional journal club, guided by well-defined best practices and instructional frameworks.^3-5  In contrast, teaching critical appraisal of quality improvement (QI) literature to residents lacks similarly robust approaches and tools to inform instructional design and is further complicated by the incompatibility of applying clinical research literature appraisal methods to a QI manuscript.^6-9  While tools for QI manuscript development have been established, most notably SQUIRE 2.0,¹⁰  these have yet to be translated into an educational context.

At our institution, internal medicine residents engage in a longitudinal EBM curriculum in a journal club setting that focuses on clinical research literature appraisal and developing skills to interpret therapeutic and diagnostic evidence. In 2018, our QI faculty were asked by EBM curricular leads to develop an approach that integrates QI literature appraisal into the EBM curriculum. In the planning process, we identified a notable gap in the ability of residents to interpret QI literature during our residents' concurrent QI course. Faculty noted that residents did not understand the design and methodology differences between QI and clinical research papers and lacked skills to assess a QI paper's strengths and limitations.

Given these findings, as well as the ACGME's shift toward preparation of graduates as systems thinkers,¹¹  we created an educational intervention for residents comprised of a systematic QI literature assessment tool. The intervention was designed to achieve several fundamental objectives in teaching residents: (1) Distinguish the differing approaches to QI and clinical research literature appraisal; (2) Teach basic QI study design, methodology, and interpretation skills; and (3) Provide a framework for assessing the strengths and weaknesses of QI interventions. Here, we report on the iterative development of a QI-EBM critical appraisal tool (QI-EBM-CAT), how residents rated the usefulness of the tool, and changes in resident confidence in appraising QI literature.

This study was conducted within the internal medicine residency program (∼100 residents) at Oregon Health & Science University (OHSU), a metropolitan academic health center located in the Pacific Northwest, with approximately 100 residents. Residents at OHSU participate in a year-long, 150-minute, monthly, longitudinal, health systems science curriculum through which the basics of QI methodology are taught in addition to other systems thinking skills. Thus, all residents involved in this study, except interns who had not yet completed the curriculum, had basic QI knowledge at the start of the study.

The residency program has an additional required, monthly, 90-minute, research EBM curriculum that exists separate from the health systems science course. It was into the EBM curriculum that the QI-EBM-CAT was introduced. The QI-EBM-CAT replaced 1 to 2 sessions annually of the research EBM curriculum from 2018 to 2020 (2018-2019 in-person; 2020 virtual format). The sessions included 20 to 30 participants, a mixture of postgraduate year (PGY)-1 to PGY-3 residents, and were taught over 4 consecutive weeks to capture all learners in the program during their respective weeks of continuity clinic. The QI-EBM-CAT and a single QI article were emailed to the residents 1 week prior to the session. From 2018 to 2020 each session featured a different QI article chosen by the Chief Resident in Quality and Patient Safety (CRQS) from the affiliated Portland Veterans Affairs (VA). The QI-EBM-CAT tool was then introduced to and used by residents to actively guide their appraisal of the selected QI paper.

Four faculty, consisting of 3 hospitalists, a primary care physician, and the VA CRQS, served as facilitators for all QI-EBM-CAT sessions. All facilitators also teach the longitudinal health systems science curriculum and are experienced instructors at the intersection of QI and medical education. No additional funding was provided for faculty to develop or teach this course.

We conducted 3 plan-do-study-act (PDSA) cycles to refine the QI-EBM-CAT (Table 1).

During PDSA cycle 1 (2018), the initial iteration of the QI-EBM-CAT was designed to provide organizational structure for residents to easily use during the QI-EBM teaching session. We adapted a list of key questions for appraising QI studies from the Journal of the American Medical Association (JAMA) User's Guide to the Medical Literature.¹²  Selected questions were then arranged under the broad headings of Background, Validity, and Results to create the basic structure of the tool. Blank space was provided for note-taking as they read the manuscript (the final QI-EBM-CAT tool is provided as online supplementary data). The session was divided into 3 parts: 20 minutes for article review, 30 minutes for small group appraisal (each group assigned one of Background, Validity, or Results) and completion of the QI-EBM-CAT, and 20 to 30 minutes for large group discussion of each group's answers with faculty facilitation of the debrief.

During PDSA cycle 2 in 2019, subjective faculty evaluations of the QI-EBM teaching sessions and informal participant feedback from PDSA cycle 1 were compiled, and they both suggested the tool did not fully meet the aim of residents being able to discern between clinical research and QI literature. Thus, we added a 3- to 5-minute introductory didactic called the QI-EBM primer, in which faculty described salient structural and interpretive differences between clinical research and QI papers (Table 2).¹³  The remainder of the intervention remained unchanged from PDSA cycle 1.

During PDSA cycle 3 (2020), we revised the QI-EBM-CAT to enable residents to translate QI-EBM appraisal knowledge and skills into future QI project participation and publication. We used the revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) to rework the guiding questions and further emphasize understanding methodology.¹⁰  The SQUIRE 2.0 guidelines were closely followed, except for educational additions to highlight the differences between QI and research manuscripts, and minor deletions of subsections for the sake of time efficiency. The QI-EBM-CAT deviated most notably from SQUIRE 2.0 in the results section, with additions to help trainees distinguish important differences from research papers, including how results are presented, strengths and weaknesses of presentation methods, and follow-up to ensure change is maintained across time. SQUIRE 2.0 language was modified into directed questions that encourage residents to seek specific answers from the manuscript to foster better understanding. Structural headings for the QI-EBM-CAT were rearranged into Introduction, Methods, Results, and Discussion, and subheadings were added to frame the rationale underlying each question. During this cycle, residents were divided into 4 small groups rather than 3, but the remainder of the session was unchanged from PDSA cycles 1 and 2.

We designed a 6-item survey to assess usefulness (1 item) of the QI-EBM-CAT session and confidence (5 items) in evaluating a QI manuscript before and after completing the QI-EBM workshop. A Likert-type scale was used to assess participants responses. Face validity was established after several revisions, based on expert review. The survey was then tested on chief residents who had previously taken the QI-EBM course to assess question clarity and order. The survey is available as online supplementary data.

Descriptive statistics, including frequencies and percentiles, were calculated for each survey variable. An overall confidence score was calculated by summing the numbers in each response category for each variable and calculating the percent based on the total in the pre- vs post-session survey. This overall score reflects a summary of confidence in the study population. Differences in survey findings according to PGY level were assessed using chi-square tests. The Wilcoxon signed rank test was used to compare participants' pre- and post-session responses due to the non-normal distribution of data. All tests were 2-tailed with the alpha level set at <.05 for determining statistical significance. IBM SPSS Statistics, version 28 was used to conduct analyses. OHSU's Institutional Review Board reviewed all study activities and considered them exempt due to their QI focus.

Sixty-six of 74 participants (89.2%) responded to the survey and reported their training year and were included in an unlinked analysis. Of these, 21 (31.8%) were PGY-1, 23 (34.8%) were PGY-2, and 22 (33.3%) were PGY-3. The majority of participants (85.1%) rated the usefulness of the session for providing a framework to appraise QI studies as somewhat to tremendously useful—somewhat useful (28.4%), quite a bit useful (43.2%), or tremendously useful (13.5%). No statistical differences were found between training years (P values >.50).

Overall changes in confidence for the 5 variables assessed between pre- and post-session are shown in Table 3. The summary confidence score in QI manuscript critical appraisal improved from a 64% rating of moderately to very confident in the pre-period to 94.6% in the post period (P<.001). In the pre-session period, the majority of participants rated themselves as being slightly to moderately confident (slightly confident range of 27.3%-34.8%; moderately confident range of 33.3%-42.4%). In the post-session period, the majority of participants reported being very confident (range of 47.5%-57.6%) and the P values for all 5 variables were <.001, indicating all increases in confidence were statistically significant. Again, no statistical differences were found between training years (P values for all 5 variables were >.60).

To our knowledge, this is the first study to adapt existing QI literature guidelines into an educational QI-EBM critical appraisal tool to help trainees analyze QI manuscripts. Our QI-EBM session using the QI-EBM-CAT was perceived as moderately to very useful by trainees and resulted in a statistical improvement in all confidence items assessed as well as in the summary score for confidence related to analyzing a QI manuscript. ACGME's expanding guidelines around teaching QI to trainees and continued emphasis on utilizing evidence-based medicine makes these findings important.

The QI-EBM-CAT fills an important gap in the QI appraisal literature, as other contemporary QI appraisal tools do not focus on physician training. For example, the Quality Improvement Minimum Quality Criteria Set (QI-MQCS) provides guidelines for reviewers rather than learners.¹⁴  SQUIRE 2.0 is designed to provide guidelines for authorship.¹⁵  The QI-EBM-CAT is specifically a teaching tool in QI manuscript assessment and a framework for identifying strengths, weaknesses, methodology, and the differences between QI and research.

Our pre-course survey showed no differences in confidence scores when evaluating a QI paper across PGYs, suggesting that residents do not learn how to evaluate QI papers from their traditional EBM education or basic QI education, highlighting the importance of this tool. Our survey suggests residents require specific training in QI-EBM appraisal to improve their confidence. Improving confidence has been linked to improved self-efficacy and ultimately authentic performance. When viewed through the lens of self-determination theory,¹⁶  the improved confidence our residents demonstrated after completing the QI-EBM course is an important first step of actualizing learned behaviors. Future research should focus on how training residents to interpret QI literature changes attitudes toward QI projects and involvement in QI initiatives. Although we only studied the utility and impact of this tool for trainees, the use of this tool could be generalizable to other medical education settings (such as undergraduate and continuing medical education) and should also be studied in those settings.

The limitations of this study include completion of the study at a single site with only internal medicine residents. In addition, the survey assessed residents' perceptions, a subjective measure. It should be noted that grouping the top 3 options on the 5-point Likert scale during our analysis may skew results positively. We also conducted an unlinked analysis; it would have been optimal to code the pre-post surveys so we could have linked survey responses in the pre- and post-periods to the same resident. Our sessions were taught by clinical educators with QI expertise, which could create context-specific advantages. Finally, the n dropped by 10.8% from pre- to post-assessment, which could affect survey outcomes.

We adapted SQUIRE 2.0 guidelines to create the first QI-specific EBM critical appraisal tool to train residents to critically assess a QI paper. We found statistically significant improvement in confidence with QI manuscript analysis among internal medicine residents.

The authors gratefully acknowledge Hayden Oldham for contributions to the QI-EBM-CAT creation.