The RISE Assessment: A Tool to Evaluate Competencies in Patient Safety and Quality Improvement Open Access

It has been a quarter-century since the Institute of Medicine’s seminal report “To Err is Human” served as a call to action for the health care industry.¹  Patient safety and quality improvement (PSQI) have subsequently become fundamental tenets in health care delivery, so it is imperative that the next generations of physicians are able to take up the mantle. Medical schools and residency training programs have increasingly established formal education and standardized curricula in this evolving and expanding field of medicine.^2-4  However, the ability to assess PSQI competencies remains a challenge.

Tools have been developed previously to assess resident competencies in PSQI. A systematic review performed to identify and evaluate tools to assess patient safety competencies for health care professionals in a hospital setting found 34 tools, of which 20 were for medical professionals, 9 were for nursing professionals, and 5 were for both.⁵  The authors noted that the tools described were predominantly used in specific clinical situations and may not be generalizable to other environments. The Quality Improvement Knowledge Application Tool (QIKAT) has been used to evaluate resident core learning and the effectiveness of QI education for medical students.^6,7  The Quality Improvement Knowledge Application Tool-Revised (QIKAT-R) was subsequently designed with a more user-friendly scoring rubric.⁸  The Systems Quality Improvement Training and Assessment Tool (SQI TAT) measures application skills, self-efficacy, and knowledge using a questionnaire and scoring system.⁹  However, no assessment tool is currently being used universally, and a standardized tool that encompasses a broader scope of content with more detailed competency levels would be helpful. The University of Rochester Medical Center (URMC) therefore developed the Rochester Improvement and Safety Education (RISE) Assessment.

The purpose of this article is to describe the development, structure, and implementation of the RISE Assessment for residents and fellows, and provide preliminary validity evidence, including feasibility of use. We hypothesized that the RISE Assessment would allow us to measure changes in proficiency with years of postgraduate training.

URMC is a regional academic medical center in upstate New York. Strong Memorial Hospital, a quaternary-care facility with 886 licensed beds within URMC, is the primary clinical learning site for more than 860 trainees in 26 residency and 69 fellowship programs.

URMC formalized graduate medical education (GME) training in PSQI in 2018, consistent with the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements. This emerged from our institution’s participation in the ACGME Pursuing Excellence Initiative.¹⁰  This endeavor sought to integrate GME trainees as active members in their sponsoring institution’s PSQI efforts. In alignment with this initiative, URMC experts in patient safety, clinical quality, performance improvement, health care delivery, nursing practice, medical education, professional development, organizational leadership, hospital operations, and interprofessional teaming developed a 2-phase curriculum. Phase 1 includes didactic and experiential learning with a “flipped classroom” approach. Trainees have been required to complete 13 online courses from the Institute for Healthcare Improvement (IHI) and obtain their Basic Certificate in Quality and Safety.¹¹  Residency and fellowship programs have then provided forums for the trainees to apply this knowledge. Examples of such sessions include development of aim statements to address specialty-specific care gaps, simulated and actual root cause analyses, and workshops on error disclosure. The format and frequency of these forums have varied by program. For example, our general surgery residency program implemented a 2-year curriculum cycle with 5 sessions per academic year. During these sessions, core content is reinforced in a didactic style for about 15 minutes after which learners have an opportunity to apply their knowledge in 45-minute small-group activities.

Phase 2 is active participation in an actual quality improvement initiative. To satisfy this requirement, trainees must play a meaningful role in an interprofessional team’s effort. “Meaningful role” includes but is not limited to analyzing baseline data collaboratively with the team, then developing, implementing, and analyzing the impact of at least one test of change.

The RISE Assessment matrix (provided as online supplementary data) was developed by the lead author (M.S.L.). A list of more than 100 content areas was initially compiled, each representing a key concept, principle, tool, or other aspect of PSQI. The content areas were selected based on priorities and commonalities within our institution and across multiple regulatory, accreditation, certification, and health care safety and quality organizations, including the Association of American Medical Colleges, the American Board of Medical Specialties, the American Board of Pediatrics, the ACGME, the Agency for Healthcare Research and Quality, the Centers for Medicare & Medicaid Services, the IHI, the Institute for Safe Medication Practices, the National Healthcare Safety Network, the National Patient Safety Foundation (now merged with IHI), and Quality and Safety Education for Nurses. The list of content areas was reviewed by an interprofessional team at URMC with the same fields of expertise as those who developed the curriculum. The team narrowed the number of content areas by removing or combining those that were similar. Content areas were then grouped into domains and subdomains based on interrelatedness and application in the clinical environment. This consolidation resulted in the 4 domains: patient safety, performance improvement, quality assessment, and team effectiveness. Team effectiveness was included as a domain given the integral and critical role high-functioning teams play in PSQI.^12,13  These 4 domains were divided into 2 subdomains each, with ultimately a total of 52 content areas in the first axis.

The second axis is a rubric developed to establish common expectations for each proficiency level. Descriptors were adapted from the NIH Proficiency Scale.¹⁴  Specific competencies were designed in an ACGME Milestone-based format which leveraged trainee familiarity with the model used by the ACGME for medical and surgical specialties.¹⁵  For each content area, the lead author (M.S.L.) applied the rubric shown to create 5 milestones. The team of subject matter experts then reviewed and revised these milestones. A level zero was subsequently added which was consistently “Learner is unfamiliar with the content area” to create the current tool.

The RISE Assessment was built in REDCap, hosted at the University of Rochester, to enable online completion.¹⁶  Approximately 250 incoming and 250 graduating trainees completed it through the GME office as a required self-assessment during the onboarding and offboarding processes, respectively, from 2018 through 2022. It was also periodically used to assess members of interprofessional teams who participated in the Pursuing Excellence Initiative, and by some training programs to assess residents at varying intervals.

The RISE score for a content area is the learner’s proficiency level for that content area. The RISE score for a domain or subdomain is the mean score of the items in that domain or subdomain. The overall RISE score is the mean score of the 52 content areas.

Analyses were conducted using SPSS Statistics for Windows, Version 28.0 (IBM Corp). Validity evidence for the RISE Assessment was collected in 3 phases: Phase 1 evaluated the internal structure of the matrix by examining Cronbach’s alpha and item-to-total correlations. Phase 2 provided evidence of predictive validity of the RISE Assessment by examining mean change across 4 groups of GME trainees: incoming residents, postgraduate year (PGY) 1-4s, incoming fellows, and PGY-5-9s. Since most residency programs range from 3 to 7 years, there are fewer trainees in lengthier residencies, and fellowships also vary in duration, grouping PGY-1-4s and PGY-5-9s was a pragmatic approach. One-way ANOVA with planned contrasts was used to test whether mean proficiency scores in each domain or subdomain increased as level of training increased. Finally, phase 3 examined whether the same level of RISE proficiency was maintained from time 1 to time 2 of the assessments.

These were retrospective analyses of data previously collected for program evaluation purposes and to improve the quality of our educational efforts. This project was undertaken as a quality improvement initiative, and as per the University of Rochester’s Guideline for Determining Human Subject Research, it did not meet the definition of research according to 45CFR46.

A total of 1797 health care professionals completed the RISE Assessment at least once between May 29, 2018, and August 30, 2022. Faculty, administrators, nurses, and other non-GME trainees were excluded. The primary analysis included: incoming residents (N=418), PGY-1-4s (N=667), incoming fellows (N=120), and PGY-5-9s (N=293), for a total of 1498 unique GME trainees.

Table 1 displays the number of items, mean (SD), Cronbach alpha, and item-to-total correlations overall as well as for each domain and subdomain. All subscales were found to be highly reliable, with adequate item-to-total correlations.¹⁷ 

Mean overall, domain, and subdomain RISE scores for each of the 4 GME trainee groups, as well as statistically significant differences and confidence intervals, are shown in Table 2. Comparing scores at each successive training level, a statistically significant higher mean overall RISE score was found for PGY-1-4s compared with incoming residents, and with PGY-5-9s compared with incoming fellows. There was no significant difference in mean overall RISE scores between incoming fellows and PGY-1-4s. Scores were then analyzed by the 4 RISE domains. As before, mean RISE scores for each domain were compared for successive training levels. In each of the domains, statistically significant higher mean RISE scores were seen for PGY-1-4s compared with incoming residents, and for PGY-5-9s compared with incoming fellows. There was no significant difference in mean RISE scores for each domain between incoming fellows and PGY-1-4s.

Scores were further analyzed by the 8 RISE subdomains. Mean RISE scores for each subdomain were compared stepwise for successive training levels (Figure 1a-d). The pattern remained the same. In each of the subdomains, there were statistically significant higher mean RISE scores for PGY-1-4s compared with incoming residents and for PGY-5-9s compared with incoming fellows. No significant difference in subdomain mean RISE scores was seen between incoming fellows and PGY-1-4s.

We also analyzed a cohort of incoming residents who completed the RISE Assessment upon starting their residency and then again at least 2 years later (N=136). There was a statistically significant increase in the mean overall RISE score, mean RISE scores for each of the 4 domains, and mean RISE scores for each of the 8 subdomains comparing pre and post assessments (Table 3).

The RISE Assessment was developed to measure resident and fellow competencies in PSQI. We found that proficiency scores increased with years of postgraduate training. We also found an increase in scores for residents who completed the RISE Assessment upon entering their residency program and then again at least 2 years later.

The RISE Assessment helps fill a gap in the ability to measure and track PSQI competencies in GME. Although administered as a self-assessment in this study, the tool can be used by program directors, clinical faculty, and others to assess individual trainees or programs. The RISE Assessment is aligned with and can support broader PSQI education efforts, such as the ACGME Clinical Learning Environment Review (CLER) Pathways to Excellence.¹⁸  Targeted assessments utilizing subsets of the overall tool, eg, one domain or subdomain, or specific content areas, can make its application more feasible and acceptable. We have successfully applied this approach in our general surgery residency program.

Our study has limitations. It was performed at only one institution with its specific PSQI curriculum, so findings may not be generalizable to all health care environments. Residents and fellows performed self-assessments, which can introduce bias. The comprehensive approach to the full assessment with respect to the number of content areas and the specificity of the proficiency levels within each makes it lengthy. This may have introduced response fatigue that influenced scores, particularly for content areas toward the latter portion of the assessment.

There are several potential next steps for the RISE Assessment. Utilizing it to assess residents and fellows in other academic medical centers and other health care settings can help better elucidate its applicability and value. Other institutions can easily replicate our approach of entering and administering the RISE Assessment via REDCap or similar platforms. Its use can be evaluated for students, attending physicians, nurses, and other health care professionals. Patient experience and health equity have been suggested by experts in these areas at our institution as additional domains to expand the breadth of the assessment and meet a need to evaluate resident learning within these important aspects of health care which are inextricable from PSQI. Additional content areas within the current domains and subdomains can also be created as expectations for PSQI competencies continue to evolve.

This study found higher proficiency scores with increasing PGY levels and with assessments performed by the same residents at the beginning of their training and again at least 2 years later. These findings provide validity evidence for the RISE Assessment.

The authors would like to thank Michael J. Apostolakos, MD, John (Jack) Bramley, MS, Diane M. Hartmann, MD, Anna E. Lambert MS, RN, John Lanphere, MBA, Robert J. Panzer, MD, Sarah E. Peyre, EdD, Brandon W. Qualls, PhD, MPA, and Stephanie VonBacho, EdD, MSEd, MS, RN, for their commitment to patient safety, quality improvement, and interprofessional education efforts which supported the work described in this article.

The online supplementary data contains the RISE Assessment matrix.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.