Translating Patient Safety Research Into Clinical Practice

The Institute of Medicine has identified medication error prevention as a priority area for transforming health care.¹  Medication usage errors are among the most common types of medical errors.^2–8  In the Harvard Medical Practice Study, adverse drug events accounted for 19 percent of injuries in hospitalized patients.^6,9  Medication use in the ambulatory care setting carries similar risks.^10–16  For example, Gandhi and Bates¹³  found that 25 percent of ambulatory patients had experienced adverse drug events. Of these adverse events, 25 to 75 percent were preventable.^2,13,17 

Errors can occur at several points in the medication use process, including drug ordering, dispensing, administration, efficacy, or toxicity monitoring. Strategies to target each of these potential vulnerabilities in the medication process include bar coded medication packaging, unit-dose medication distribution, bedside scanning technology, computerized prescriber order entry, linked laboratory and pharmacy systems, including a pharmacist on physician’s patient rounds, clinical decision support systems, reviewing medication orders, interdisciplinary drug therapy protocols, and information technology providing assistance with dosage calculations, communication and monitoring.^3,18–28 

Several reviews of medication safety interventions have been published.^{13,18,22,26,29–31}  While the effect of most interventions was positive, it is unclear if successful interventions made the transition to mainstream use, once the research project was completed (i.e., the translation of research into practice). This lack of information on the translation of medication safety interventions may be in part due to the absence of clear criteria for evaluating the applicability of medication safety interventions to other populations or settings.

The purpose of this paper is to outline a conceptual model and methodology for evaluating the potential for translation of patient safety interventions into practice, using the RE-AIM framework of Reach, Efficacy/effectiveness, Adoption, Implementation and Maintenance. The RE-AIM approach has been used to evaluate a range of health promotion programs (e.g., physical activity or tobacco cessation programs) in a variety of populations (e.g., healthy subjects and patients with chronic illnesses) and settings (e.g., schools, worksites and medical clinics).³²  We illustrate each of the RE-AIM dimensions with an example from the medication safety literature and end with an example of an ongoing medication safety study that was designed and is being evaluated using the REAIM framework.

The development of effective patient safety interventions clearly is a national priority. There are standard study designs (e.g., randomized controlled trials) and reporting methods (e.g., CONSORT criteria) for assessing an intervention’s efficacy. However, in recent years several investigators have also identified generalizability and translatability as important factors to consider in the evaluation of health care interventions.^33,34 

Glasgow et al.³⁵  designed a framework to expand the evaluation of interventions beyond efficacy to include additional criteria that may better identify the translatability of health promotion interventions. They proposed that the translatability of initiatives is best evaluated by the examination of five dimensions (Table 1). These five dimensions are an intervention’s (1) Reach—the absolute number, proportion and representativeness of those individuals who participated in the intervention, compared to the target population; (2) Efficacy/effectiveness—the success in achieving the study goal, including quality of life outcomes; (3) Adoption—the absolute number, proportion, and representativeness of settings or organizations that implemented the intervention; (4) Implementation—the fidelity to the intervention protocol; and (5) Maintenance—the level of sustained individual behavior and organizational use of the intervention over time.³⁵  These five dimensions comprise the RE-AIM framework. In assessing an intervention for generalizability and translation, it is important to consider the individual and organizational impacts of the intervention. Within the REAIM framework, reach and efficacy assess individual levels of impact, while adoption and implementation assess organizational levels of impact. Maintenance can be assessed at an individual level, as well as an organizational level. Both individual and organizational levels of impact provide valuable independent information. Take, for example, an electronic medical records-based medication safety program that has a large impact in terms of its reach and efficacy at the individual level, but is adopted, implemented and maintained in only those organizations with electronic medical records (which are not currently available in most health care systems). If only the individual dimensions of the framework were used to evaluate this intervention, it would be concluded that the intervention has a large potential for reducing medical errors. In reality, this intervention is likely to have little public health impact because it could not be adopted, implemented or maintained in most existing health care systems.

The inclusion of reach in the RE-AIM framework was influenced by Abrams et al.,³³  who suggested that in addition to efficacy or effectiveness, an intervention program’s reach into a target population also should be considered. They proposed that the impact of an intervention should be calculated as the product of its reach and its efficacy. Consider, for example, two programs designed to reduce medication prescribing errors for hospitalized patients. Program A reduces prescribing errors by 80 percent but reaches only the clinicians who care for patients in the CCU, representing 10 percent of the hospitalized patients (overall impact is 80% × 10% = 8%). Program B reduces prescribing errors by 50 percent and reaches clinicians who care for patients on all floors of the hospital, that is 100 percent of the hospitalized patients (overall impact is 50% x 100% = 50%). While this example assumes a simplistic relationship between reach and efficacy or effectiveness, it illustrates the importance of considering both reach and efficacy in assessing the impact of an intervention.

The reach of an intervention may be characterized by (a) the absolute number of participants, (b) the participation rate and (c) the degree to which the sample is an accurate representation of the target population—sometimes known as the “representativeness.” Within the medication safety literature, the absolute number of participants (i.e., sample size) and participation rate are commonly reported. For example, Lipton et al.³⁶  conducted a randomized controlled trial to examine the impact of clinical pharmacists’ consultations on physicians’ drug prescribing for geriatric patients. This study reported a total of 1,383 patients who met the eligibility criteria for participation in the intervention. Of this total, 512 patients could not be contacted and 156 of those contacted declined to participate. Of the remaining patients, 274 were invited to participate and 236 of these actually participated in the study. Thus, the reach of the study was to 236 patients or 17 percent of the 1,383 eligible patients.

In contrast to the regular reporting of sample size and participation rate, the representativeness of the samples used in medication safety intervention trials is rarely reported. While obtaining information on eligible individuals who decline to participate in studies may be difficult, it is nonetheless important to determine if those who choose to participate are different from those who do not participate. The study’s findings are likely to be more generalizable if the study subjects are comparable to the general population approached to participate.

Within the RE-AIM framework, efficacy or effectiveness is assessed at the level of the individual and is a measure of the intervention’s impact.³⁵  Efficacy is a measure of how an intervention performs under optimal conditions, while effectiveness is a measure of how an intervention performs under “real-world” conditions. Typically, interventions that are effective also are efficacious. The converse is not always true, however, because some efficacious interventions are not effective when they are implemented under more complex and less controlled conditions. From the perspective of generalizability and translation, measures of effectiveness are of greater utility than are measures of efficacy.

Most studies in the medication safety literature have reported the efficacy of interventions. Hunt et al.,²²  for example, conducted a systematic review of 15 studies that examined the efficacy of clinical decision support systems designed to assist with drug dosing. Overall, nine of 15 studies showed improvements in drug dosing with computer-based clinical decision support systems. Similarly, a review of studies addressing the impact of clinical pharmacists in preventing medication errors and adverse events found improvements in outcomes for ambulatory care patients with hypertension, diabetes, heart failure and hyperlipidemia.²⁹ 

When reporting on efficacy or effectiveness, it is necessary to document the possible negative or unintended consequences of the intervention. An example of a medication safety study that also reported on negative outcomes was a computer alert system designed to prevent injury from adverse drug events.³⁷  The study showed that an improvement in medication prescribing was accompanied by some unintended negative consequences, specifically, a reported false-positive alert rate of 47 percent. Considerable staff time was spent resolving the false-positive alerts, which, on average, required 15 minutes of pharmacist time. Such findings suggest the importance of evaluating the potential positive and negative outcomes of an intervention.

Adoption is defined as the number, proportion and representativeness of settings, organizations or staff that use an intervention program. Thus, adoption can be considered an assessment of an intervention’s reach at the organizational level. Similar to reach, the adoption of an intervention may be characterized by (a) the absolute number of settings, (b) the setting participation rate and (c) the representativeness of the settings and intervention agents sample.

With the exception of the absolute number of settings involved, patient safety researchers rarely report on issues of adoption. A notable exception was the study by Bates et al.³  of the effect of computerized physician order entry on the prevention of serious medication errors. In this study, two hospitals were recruited to participate in the trial. Only one of the two hospitals was able to execute the study successfully, as staff changes and administrative changes at the second hospital brought about inadequate data collection. While the intervention was shown to be successful in a single hospital, it is difficult to conclude whether the intervention could be generalized to other settings.

Understanding the adoption of interventions by settings or individuals who will actually deliver the intervention is helpful in projecting the potential impact of an intervention. Adoption also is important because investigations often are conducted in specialty settings that have the highest levels of resources or staff with expertise that is not likely to be replicated outside of a research environment.

In intervention studies, one of the most frequent reasons for failure to find an effect is that the intervention was not delivered consistently in the context of competing demands. Implementation is defined as the extent to which an intervention is delivered as intended, and is typically measured at the organizational level.³⁵  The organizational level can be operationalized as the setting (e.g., clinics, hospitals or heath plans) or staff (e.g., physician, nurse or pharmacist) that deliver the intervention.

Chertow et al.³⁸  provide an example of implementation. They assessed the implementation of a decision support system that was added to an existing computerized order entry system for prescribing drugs for patients with renal insufficiency. They compared appropriate drug prescribing in four consecutive two-month intervals consisting of control (i.e., usual computerized order entry) alternating with intervention (i.e., computerized order entry plus the decision support system). Rates of appropriate prescriptions by dose and frequency were higher in the intervention time periods than in the control periods. Overall, 49 percent of medication orders for patients in the intervention group were inappropriate—usually because the physicians disregarded the advice provided by the system at the time of ordering. The researchers did not investigate the reasons the physicians accepted or rejected the medication advice, but speculated that it might be due to comfort with already-established prescribing practices.

Maintenance, the long-term change associated with an intervention, can be assessed at the individual level or the organizational level. Maintenance is important to assess because many processes decay with time. Research in medication safety has been inconsistent in reporting on maintenance at the individual level. Some intervention studies include a variety of follow-up data points for assessing intervention maintenance. Hanlon et al.,³⁹  for example, conducted a randomized controlled trial of a clinical pharmacist intervention to improve inappropriate prescribing in elderly outpatients with polypharmacy. Measures of outcomes included prescribing appropriateness, health-related quality of life, adverse drug events, medication compliance and knowledge, number of medications, patient satisfaction and physician receptivity. These measures were obtained at the beginning of the clinical trial, at three months and at 12 months. The study found that a clinical pharmacist providing pharmaceutical care for elderly outpatients reduced inappropriate prescribing. The improvement in prescribing with the clinical pharmacist was noted by three months and was sustained at 12 months.

Maintenance also should be assessed at the organizational level by determining the length of time that an intervention is sustained. In the medication safety domain, persistence is seldom reported once a program is adopted in a real-world setting. Information of this type needs to be reported, as a means of gauging the public health impact of medication safety programs that are adopted in real-world settings.

We are currently using the RE-AIM approach to evaluate a pharmacy alert system intervention designed to detect errors in medication prescribing to patients with renal insufficiency. In this project, a pharmacy alert system electronically screens prescriptions for potential errors using guideline-driven decision rules. A medication alert is issued when the system detects a potential error, the prescription label is not produced and the medication cannot be dispensed without an active intervention by a pharmacist. The pharmacist confirms the validity of the alert and then consults a decision support guide that helps to resolve the possible error in collaboration with the prescriber.

In this study, Reach is being assessed through evaluation of the participation rate and the representativeness of the enrolled patients. Our primary Effectiveness measure is the incidence of medication errors, defined as the rate at which potentially harmful medications or doses of medications are dispended to patients with renal disease. Other measures of Effectiveness include prescriber, pharmacist and patient satisfaction; measures of unintended consequences (e.g., false-positive alerts); and potential negative impacts (e.g., repeat patient visits to the pharmacy). Adoption is being measured by the participation rates of the pharmacies and pharmacists asked to take part in the project. Representativeness at the Adoption level is being assessed by a comparison of the characteristics of pharmacies and pharmacists that participate with those that do not. Implementation is being assessed using measures of technical performance (including the validity of alerts and system reliability) and measures of pharmacist performance (including completeness of progress notes and adherence to alert recommendations). Maintenance is being assessed as the sustainability of the intervention’s impact during the study (measured at three, six, nine and 12 months), and for six months following completion of the study, to assess the degree to which the host organization continues the intervention after research funding ends.

Although the RE-AIM framework focuses attention on issues important to research translation, it also has limitations. Given that this framework is relatively new, there are few studies reporting RE-AIM measures. Thus, it is difficult to compare results across studies. Second, it can be difficult or impossible to know with precision the exact “denominator” of patients or settings in the target population, complicating estimates of reach or adoption. Finally, while the RE-AIM framework includes an assessment of the intervention maintenance, the recommendations for the timing and frequency with which interventions should be evaluated during and following completion of the study are not standardized.

In its landmark report, Crossing the Quality Chasm, the Institute of Medicine identified the translation of research into practice as a priority area.⁴⁰  For patient safety research to be more relevant to clinicians and decisionmakers, it is important to promote the conduct of practice-based, patient-outcomes research in applied settings.⁴¹  The RE-AIM framework of Reach, Effectiveness, Adoption, Implementation and Maintenance is one approach that investigators can use to assess a study’s potential for translation from research to clinical practice.

This work was supported by Grant # HS14249, from the Agency for Healthcare Research and Quality, and a grant from the Garfield Foundation.