Cardiopulmonary monitors (CPMs) generate false alarm rates ranging from 85%–99% with few of these alarms actually representing serious clinical events. The overabundance of clinically insignificant alarms in hospitals desensitizes the clinician to true-positive alarms and poses significant safety issues. In this IRB-approved externally funded study, we sought to assess the clinical conditions associated with true and false-positive CPM alarms and attempted to define optimal alarm parameters that would reduce false-positive alarm rates (as they relate to clinically significant events) and thus improve overall CPM performance in critically ill children.
Prior to the study, clinically significant events (CSEs) were defined and validated. Over a seven-month period in 2009, critically ill children underwent evaluation of CSEs while connected to a CPM. Comparative CPM and CSE data were analyzed with an aim to estimate sensitivity, specificity, and positive and negative predictive values for CSEs.
CPM and CSE data were evaluated in 98 critically ill children. Overall, 2,245 high priority alarms were recorded with 68 CSEs noted in 45 observational days. During the course of the study, the team developed a firm understanding of CPM functionality, including the pitfalls associated with aggregation and analysis of CPM alarm data. The inability to capture all levels of CPM alarms represented a significant study challenge. Selective CPM data can be easily queried with standard reporting, however the default settings with this reporting exclude critical information necessary in compiling a coherent study denominator database. Although the association between CPM alarms and CSEs could not be comprehensively evaluated, preliminary analysis reflected poor CPM alarm specificity. This study provided the necessary considerations for the proper design of a future study that improves the positive predictive value of CPM alarms. In addition, this investigation has resulted in improved awareness of CPM alarm parameter settings and associated false-positive alarms. This information has been incorporated into nursing educational programs.
Cardiopulmonary monitors (CPM) are currently designed with flexible alarm parameters to warn providers about patient conditions, events, or devices that deviate from a predetermined “normal” status.1 When an alarm is triggered, the provider is expected to respond to the alarm, identify its cause, and intervene as necessary.2,3 According to the American College of Clinical Engineering (ACCE) Healthcare Technology Foundation, clinical alarms should deliver information that is accurate, intuitive, and provide alerts which are readily interpreted and acted on by clinicians.4 However, the ACCE Healthcare Technology Foundation reported that CPM alarms were not performing as expected because of a complex set of interdependent issues.4
Some alarms may reflect a change in the patient's condition (true-positive) while many others are not clinically significant and/or reflect poorly set monitoring fparameters (potentially causing false-positive/nuisance alarms).3 A false-positive or clinically insignificant alarm is defined as an alarm that occurs in the absence of an intended, valid patient or alarm system trigger.4 The sheer volume of clinically insignificant alarms in the hospital setting is an important safety issue.5 False-positive alarm rates have been reported ranging from 85%–99% with few representing significant clinical events requiring provider intervention.4–6 In one report, the number of alarms in a medical progressive care unit was documented during an 18-day period (patient census of 12). The number of alarms totaled 16,953, or 942 alarms/day with one alarm occurring every 92 seconds.7 Data from one of our critical care units in 2009 were similar over a 30-day period (patient census of 35). A total of 39,000 alarms occurred or 1300 alarms/day and one alarm sounding every 66 seconds.8
The overwhelming number of false-positive alarms has been likened to the Aesop's fable of the boy who cried wolf.9 Alarm fatigue can occur when the large number of monitor alarms overwhelms and desensitizes providers7, causing them to divert attention away from clinically significant events.3 With such fatigue, providers often ignore the sound, lower the volume, extend alarm limits outside of a reasonable range, or disable the alarms.3,10–12
Paradoxically, CPM may contribute to the generation of adverse patient events. Because of the disproportionate number of false-positive alarms, there is a lower likelihood of effectively responding to an alarm if the false-positive alarm rate is high.7,11,12 Despite regulatory and accreditation guidelines regarding CPMs established by The Joint Commission in 2002, CPM-related adverse events including patient death continue to occur.13 Although reporting of sentinel and adverse events is sparse in the literature, the authors have experienced incidents of inattention to alarms with significant adverse patient outcomes.
We conducted an eight-month study on multiple units at our pediatric medical center and found that the mean monitor alarm response time exceeded three minutes in 50% of the cases (range 25–65%).14 These findings led to the assignment of a monitor technician stationed at a central monitoring bank for the purposes of notifying nurses of CPM alarms. These efforts did not result in any detectable improvement in provider alarm response time. In addition, at our institution, although the CPM alarm parameters are to be ordered by a physician or licensed independent prescriber every 24 hours, a recent evaluation documented poor compliance with this policy with less than 50% of our physicians/providers ordering CPM parameters.14
In this study, a team of nurses, biomedical engineers, physicians, and biostatisticians was assembled to develop a project to assess the conditions associated with the generation of CPM alarms including false-positive alarms in critically ill children. In addition, this team set out to define alternative alarm parameters that would improve CPM alarm generation performance. We hypothesized that the sensitivity, specificity, and positive predictive value of CPM alarms could be optimized resulting in a significant reduction in false-positive alarms. The purpose of this article is to describe the study methodology, lessons learned, and implications for future research and practice.
The specific aims of the study were to:
Compare CPM alarms to clinically significant events (CSEs) in the pediatric intensive care unit (PICU) to estimate sensitivity and specificity of alarms based on current procedures.
Improve the performance of the CPM alarm system by using a statsically guided approach for manipulating alarm settings for the optimized triggering of true-positive and true-negative signals for CSEs, and thereby, minimizing the rate of false-positive alarms.
Methods Inclusion Criteria
This externally funded study was approved and deemed exempt by the hospital's Institutional Review Board. The study was conducted in a 24-bed, Level I Pediatric Intensive Care Unit (PICU) with an average daily census of 20 children. All children with severe or potentially life-threatening diseases and those with multisystem as well as postoperative severe conditions were eligible for inclusion in the study. Patients were excluded from the study if they were admitted pending organ donation, were admitted for less than 12 hours, or had an anticipated length of stay of less than 24 hours.
In this study, a team of nurses, biomedical engineers, physicians, and biostatisticians was assembled to developed a project to assess the conditions associated with the generation of CPM alarms, including false-positive alarms in critically ill children.
Clinically Significant Event (CSE)
A focus group of PICU nurses was convened to explore and develop the definition of a CSE. The nurses were asked to describe what types of patient events prompt a monitor alarm, what types of clinical events require them to intervene on the patient's behalf, and to describe the times when their patients may have had a CSE but the nurse was not alerted by a CPM alarm. From this consensus work, a CSE was defined as an event that requires intervention without which the patient's condition would worsen or deteriorate.
CSEs were confirmed by the research data collection nurse and bedside nurse and then recorded. Events or data that were in question or difficult to interpret were reviewed by two coinvestigators and two independent critical care nurses and physicians for analysis and adjudication.
Cardiopulmonary Monitoring Equipment
The bedside CPM devices used for the study were the same devices used for patient care (Philips MP70 devices with individual parameters available for heart rate, cardiac monitoring, pulse oximetry, non-invasive blood pressure measurement, invasive pressure measurements, temperature, and respiratory rate). The bedside devices (MP70) were connected to a networked central station (Philips Patient Information Center - PIC) that saved vital sign results, graphs, and alarm data associated with each monitor on a database server and were automatically exported from the database server to the Philips Research Data Export Tool every four hours and stored indefinitely for all patients in the PICU. A script was used to extract patient information from the database and store it in a lookup table. These data were electronically filed by patient lookup number on the system server until a potential study patient was identified. When a study patient was identified, biomedical engineering extracted the two files associated with that patient (alarm file and vital signs file) and sent them to the study coordinator.
The alarm file was a text file containing a time stamp and a description of the alarm that occurred and the type of alarm (Philips classifies alarms as one-, two-, or three-star alarms). Three-star alarms were defined as a cardiac arrhythmia, apnea, or oxygen desaturation; two-star alarms were defined as vital signs that exceeded high/low parameter settings; and one-star alarms represented equipment alerts. There were two available files for each device (patient): a list of alarms and a minute-by-minute table for all of the vital sign parameters that were measured. The specific data sent from the monitor to the research data export tool were configured at the PIC central station.
The vital signs file was a text file listing the measured vital signs in one-minute intervals and only displayed/recorded vital signs that were being measured by the patient monitor. For example, the vital signs for non-invasive blood pressure were not displayed if that blood pressure connection was turned off on the monitor. The one-minute vital sign recorded was an average of the vital signs measurements over that one-minute period.
There were three routes of data acquisition: direct data recording witnessed by the research data collection nurse; indirect data recording obtained from the bedside nurse when not observed by the data collection nurse during the study period, and extraction through daily electronic medical record review.
Prior to the collection of data, a standardized case surveillance data sheet (i.e., the monitor data collection form) was developed and piloted with the research data collection nurses trained to perform study functions. The case surveil-lance data sheet was then refined to accommodate clinically necessary changes and add validity to the measures being recorded. Data collected were recorded on the form shown in Table 1.
In addition, data from the CPM were collected for each patient for up to 72 hours per patient using Philips Intellivue Trend and Alarm monitor query software to allow full disclosure review of data. CSEs were characterized independently of the CPM alarms. There was no attempt at the bedside to establish whether an alarm was a false- or true-positive alarm. That decision was based solely on whether the alarm was coincident or occurred within several minutes of the CSE. Occurrence, type, and timing of alarms were based on the CPM data retrieval and analysis during the observation period.
Data collectors were trained on how to complete the case surveillance data sheet to promote a standard methodology that minimized variability. The principal investigator met regularly with the data collectors and provided oversight and periodic review of data collection. The PICU staff were provided with an overview of the study purpose and design.
At the beginning of each direct data observation, the research data collection nurse notified Biomedical Engineering of the need to extract CPM data for all eligible patients enrolled in the study that day. Demographic and clinical data were then recorded for each patient. Data collection rounds were performed at least hourly. The direct data observation periods ranged from two hours to seven hours (average five hours) per day over the course of three days per patient. The bedside nurses were informed of the patient's participation in this study and were asked to report CSEs not observed by the data collector during the direct data recording period.
In aim 1, cross tabulations were developed to assess the sensitivity, proportion positive by the gold standard CSE that are CPM positive, and specificity, proportion negative by the gold standard that are CPM negative and set the 95% confidence interval (CI) around each estimate. We defined cut-points for acceptable levels of sensitivity and specificity. In addition to an overall analysis based on all types of CSE, we planned to estimate sensitivity and specificity for selected subtypes of these events. The purpose behind this type of subgroup analysis was to identify whether CPM performance varied greatly by subtype of event.
For aim 2, we planned to use receiver-operator characteristic (ROC) analyses on each CPM clinical parameter being monitored to identify the best cut-point(s) to maximize sensitivity and specificity for CSEs overall and by subtype. ROC analysis was used to evaluate sensitivity versus 1- specificity (false positivity) associated with moving the cut-point for signaling an event warning (alarm) across the full range of values of each monitored parameter. Based on ROC analysis, we planned to choose a single cut-point or set of cut-points that met pre-specified criteria. We defined these selection criteria as 1) that set of cut-points for which the specificity was ≥ 70%, and 2) where the sensitivity was ≥ 90%. We intended to repeat this testing for each parameter defining the set of values that met the defined criteria or designate that no such criteria existed.
Prior to the study, clinically significant events (CSEs) were defined and validated. Over a seven-month period in 2009, critically ill children underwent evaluation of CSEs while connected to a CPM (MP70, Philips Healthcare, Andover, MA). Comparative CPM and CSE data were analyzed with an aim to estimate sensitivity, specificity, and positive and negative predictive values for CSEs.
CPM and CSE data were evaluated in 98 critically ill children. During the observation period, 2,245 alarms were recorded with 68 CSEs noted in 45 observational days. Types and characteristics of CSEs are noted in Table 2.
During the course of the study, the team developed a firm understanding of CPM functionality, including the pitfalls associated with aggregation and analysis of CPM alarm data. One significant challenge included the inability to query all levels of CPM alarm data. The alarm file for each patient only recorded three-star alarms but did not record the one-star and two-star alarms secondary to a setup issue with the Philips central station. Accordingly, the association between CPM alarms and CSEs could not be fully evaluated with the anticipated ROC analyses.
Investigational time stamps were also noted to be problematic in that the time posted on the data collection sheets did not always match the time on the two study files and were in error by up to four minutes. The Philips bedside monitors, the Philips database server, and the hospital time devices (computers and phones) were not problematic as they were all on the same time server.
In addition, there were some patients whose medical record number was not recorded on the bedside monitor. Therefore, when there was an attempt to match these two files, they could not be validated and were, therefore, excluded from study analysis.
CSEs are common in critically ill children.15 In this study of pediatric critical care patients, it was not surprising to discover that respiratory CSEs, including hypoxia and apnea, comprised the majority of the events. We set out to examine the relationship between CPM and CSEs. The largest impact to the study was related to the recording of alarms. Although CPM data can be easily queried, reporting configuration default settings can exclude critical information that is necessary in compiling a coherent denominator database.
This investigation has resulted in improved awareness of CPM alarm parameter settings, associated false-positive alarm rates, and the potential impact on quality care delivery.
During the study, we were unaware that all alarms were not saved into the alarm file because the central station patient information center was defaulted to send only three-star alarms to the research data export tool to limit the file size. Because this issue was not identified by the research team until all study data had been collected, the data stored did not definitively identify all alarms that occurred with each study patient. As a result, our inability to capture all relevant CPM data impeded our ability to rigorously test the relationship between CPM and CSEs. Initial impressions, however, from this investigation are that many, but not all, CSEs can be detected with the CPMs currently in use.
This investigation has resulted in improved awareness of CPM alarm parameter settings, associated false-positive alarm rates, and the potential impact on quality care delivery. In addition, this information has been incorporated into an annual education for all nursing staff regarding bedside monitoring.
Because of the complex and interdependent issues involved in CPM alarms, we believe one of the strengths of our project was the interdisciplinary nature of our study team. The clinical and technical expertise and contributions of our frontline and research nurses, biomedical engineers, physicians, and biostatisticians were critical in expanding our knowledge and understanding of the relationship between CPM alarms and CSEs.
Implications for Future Research and Practice
CPM devices are physiological parameter screening tools that attempt to identify patients whose condition is deteriorating for early preventative intervention. There are well-established criteria for the use of clinical monitoring screening tools.16 We recommend that researchers consider these criteria in designing future studies.
First, the screening outcome should be an important patient-specific health issue. Clearly, CSEs in a critically ill population meet this criterion. However, in conducting CPM studies, it is important to clearly define the clinical events that necessitate prevention. In the absence of such clarity, the study methodology would likely characterize clinical deterioration only in terms of the monitor setting parameters. In the current study for example, one type of CSE was defined as oxygen desaturation. In this case, data also could be recorded to determine whether deterioration is occurring based on the patient's clinical status.
Second, the investigative team should have a clear definition of whom to screen for the study. In this study, most patients were included if admitted to the PICU, despite marked variability in severity of illness and, therefore, the likelihood of developing a CSE.
Third, there should be an acceptable treatment or preventative intervention that alters the outcome should a CSE occur. For example, performing tracheal intubation for a patient who develops apnea would represent such an intervention, whereas it is not clear that calming a crying child who has developed tachycardia represents an intervention of the same importance.
Fourth, there must be a valid and acceptable screening test that will identify persons at risk of a CSE in which an intervention can be applied successfully. A valid monitoring tool must have adequate sensitivity and specificity. In this preliminary analysis, it appears that the CPM, as currently used, has high sensitivity but poor specificity and, therefore, a high false-positive rate.
Despite the analytical challenges, several important findings in our study design were illuminated for future investigation:
improved methodology in conducting the next iteration of this study so that all appropriate monitor alarm categories are accurately and reliably captured to ensure comprehensive data analyses,
appropriate design in defining and measuring CSEs in the PICU,
the relationship between PICU CSEs and CPM data.
About the Authors
Linda B. Talley, MS, BSN, RN, is vice president of nursing systems and neonatal services at Children's National Medical Center in Washington, DC. Email: email@example.com
Jeffrey Hooper, MS is director of the department of biomedical engineering at Children's National Medical Center. Email: firstname.lastname@example.org
Brian Jacobs, MD is vice president and chief medical information officer at Children's National Medical Center. Email: email@example.com
Cathie Guzzetta, PhD, RN, AHC-BC, FAAN, is a nursing research consultant at Children's National Medical Center. Email: CGuzzett@cnmc.org
Robert McCarter, ScD, is director of biostatistics and informatics at the Children's Research Institute, Children's National Medical Center. Email: firstname.lastname@example.org.
Anne Sill, BA, is a research associate at the Children's Research Institute, Children's National Medical Center. Email: email@example.com
Sherry Cain, BSN, RN, is an RNII within the heart and kidney unit at Children's National Medical Center. Email: firstname.lastname@example.org
Sally L. Wilson, BSN, RN, is education, prevention and outreach coordinator at the Trauma Burn Service at Children's National Medical Center. Email: SLWilson@cnmc.org