This report consists of two separate studies on the use of continuous capnography monitoring conducted in an effort to improve patient safety at Virtua Health System. The desire for improved patient safety is motivating continuous monitoring and improved surveillance in clinical areas not traditionally equipped for such monitoring. We explored the use of remote monitoring of capnography, using enterprise middleware, in patients recovering from surgery in a medical-surgical unit. Continuous monitoring traditionally has been used in higher-acuity settings, such as intensive care units. Patients diagnosed or suspected to have obstructive or central sleep apnea may benefit from the increased surveillance afforded by continuous monitoring. Pain management in this cohort of patients, recovering from bariatric, joint replacement, or other major surgery, often involves administration of opioids (e.g., hydromorphone, morphine sulfate), which are known to increase risk of respiratory depression. Continuous monitoring of these patients increases the likelihood of detecting adverse clinical events. Our goal was to implement continuous monitoring in order to identify alarm conditions caused by adverse clinical events requiring intervention (e.g., opioid-induced respiratory depression) and artifacts related to patient movement, suspect measurements, or other medical device–generated alarm signals.

Increasingly, surgical patients present with complex medical conditions and multiple comorbidities that make perioperative care more challenging. Patients often are prescribed postoperative pain medications, mainly opioids, to control pain associated with surgical procedures. These pain medications can produce undesirable and potentially life-threatening adverse effects.

Virtua Health System (VHS) sought to prioritize narcotic safety by implementing noninvasive capnography monitoring in 2013.1 As part of this narcotic safety program, an approach for remotely monitoring and identifying respiratory depression was sought. Respiratory depression associated with use of opioids in the postoperative period is a well-recognized adverse effect.2–4 Close monitoring of these patients is suggested as a means to identify early deterioration.

Capnography is defined as the “noninvasive measurement of the partial pressure of carbon dioxide in exhaled breath,” and through its use, apnea can be detected almost instantaneously.5 Recent studies identified capnography as providing a more sensitive and early predictor of opioid-induced respiratory depression (OIRD), particularly in patients with obstructive sleep apnea (OSA) or central sleep apnea (CSA), and capnography has been recommended as a best practice for monitoring these adverse effects.6–11 Many postsurgical patients, who are at risk for OIRD, are cared for in medical-surgical units (MSUs), where capability for close monitoring is not as readily available as in higher-acuity settings, such as intensive care units (ICUs).

When setting up increased levels of surveillance in MSUs, care must be taken to differentiate actionable from nonactionable alarm signals, so as to minimize false alarms and the associated potential for alarm fatigue. Alarm signal annunciation and the triggering of false alarm signals is deemed a major patient safety concern, with the number of nonactionable alarm signals yielding falsepositive alarm rates considerably exceeding actionable alarm signal quantities. Estimates have indicated that these false-positive, nonactionable, clinically insignificant alarms account for 85% to 99% of all alarm signals.12 Some approaches to false alarm signal reduction involve basing alarm signaling thresholds on derived parameters, such as the Integrated Pulmonary Index, which is a combinatorial index based on four parameters. 11 Although we do not dispute the value of such combinatorial measures, the objective of the current work was to assess the relative benefit of sustained and combinatorial alarms using clinical measures, including end-tidal carbon dioxide (etCO2), spontaneous respiratory rate (or respiratory frequency [fR]), pulse rate (PR; measured using peripheral capillary oxygen saturation [SpO2] cuff), and arterial oxygen saturation (measured using SpO2 cuff), as they provide clearer clinical meaning compared with indices.

Techniques for mitigating nonactionable alarms also are recognized as being of supreme importance to reduce clinician workload and improve patient safety.13 Hospitals are concerned about alarm fatigue because “it interferes with patient safety, and it exposes patients … to grave harm.”14 It also has been asserted that “a majority of alarm-related adverse events result in brain injury or death, carrying a median claim of nearly $500,000.”15,16 

The American Society of Anesthesiologists in 2009 stated that “end-tidal carbon dioxide monitoring is more likely to detect hypercapnia/hypercarbia and respiratory depression than are clinical signs.”17 The Anesthesia Patient Safety Foundation recommended that “monitoring the ventilation of patients receiving narcotics with capnography … is the most reliable detector of hypoventilation” and that concerns regarding the potentially high number of false-positive alarms in the postoperative patient may be ameliorated by developing “algorithms blending pulse oximetry and capnography to yield greater benefit with fewer false-positive events.”18 

In 2009, the Emergency Nurses Association identified certain “conclusions and recommendations about the use of capnography for procedural sedation and analgesia in adults and children in the emergency department,” recommending “etCO2 is a more sensitive indicator of respiratory depression than [arterial oxygen saturation] or clinician assessment.”17 

In the process of implementing remote capnography monitoring, VHS engaged in both an initial pilot and a follow-on clinical study. The initial pilot focused on remote monitoring of capnography alarm signals issued through middleware to telemetry technicians within the three hospitals comprising the VHS enterprise. At the end of the initial pilot, remote capnography monitoring was discontinued to analyze data and make recommendations as to its utility and value. Following this period of analysis, a subsequent clinical trial was conducted in which recommendations resulting from the assessment of the initial pilot were implemented. A summary of the results of both the initial pilot and follow-on clinical study is provided, along with conclusions and lessons learned.

Initial Pilot Study

Noninvasive capnography monitoring was done using Capnostream 20 Bedside Monitors (Covidien, Needham, MA). Creation and remote communication of both middleware-generated and medical device–generated alarm signals were accomplished using an enterprise middleware rules engine (Bernoulli One Analytics Software; Bernoulli Enterprise, Inc., Milford, CT). Upon initial rollout of capnography monitoring, both monitor- and middleware-generated alarm signals were communicated remotely to telemetry monitoring units, where telemetry technicians (“tele-techs”) could assist in the management and monitoring of these patients and relay events to nursing units when alarm signals were received. Data were communicated over the hospital information technology network using TCP/IP protocols. In-room alarm signals (both audible and visual) from the monitors were not impeded or changed in any way and were allowed to annunciate per normal bedside monitor operation. Middleware-generated alarm signals were created based on the raw measurements obtained from the monitors. Alarm signals were created based on end-user–defined thresholds, which, when breached, were communicated as visual and audible alarms to a dashboard view presented to the tele-techs.

As alarm signals were created and communicated, tele-techs would monitor the dashboard view, which identified the status of all patients on capnography. The tele-techs would communicate the alarm event to the nursing units associated with the patient for whom the alarm event was issued, as well as print a copy of the alarm event for retention within the telemetry bunker (a room dedicated for telemetry monitoring within each of the three hospitals).

The monitor communicates device-issued discrete alarm signals in the form of binary “on/off” messages: an alarm either is or is not issued. A summary of the available measurements and device-issued alarm signals is provided in Table 1 in the data supplement (available online at http://aami-bit.org). These alarm signals provide no information relative to severity of specific values. Rather, they serve to indicate that an alarm of predefined threshold, based on monitor-set threshold values, was breached. In all cases, monitors were configured to the same on-board threshold settings, and these values remained unchanged throughout the course of both the initial pilot and the follow-on clinical study. In contrast, middleware-generated alarm signals were created to communicate threshold breaches that contained both the threshold and the particular value causing the threshold breach.

Table 1.

Characteristics of the study patient population.Abbreviation used: PCA, patient-controlled analgesia.

Characteristics of the study patient population.Abbreviation used: PCA, patient-controlled analgesia.
Characteristics of the study patient population.Abbreviation used: PCA, patient-controlled analgesia.

Data were captured from the monitors using Bernoulli Serial-to-Ethernet Bridges (Figure 1). The data collected through the bridges were communicated wirelessly to Bernoulli analytics software, which was installed in the enterprise data center. There, the data were stored and processed in real time and alarm signal messages were communicated and displayed within a telemetry bunker–based dashboard display on a dedicated computer monitor. The analytics software also received admission-discharge-transfer transactions from the enterprise electronic health record system to facilitate patient association to the monitors.

Figure 1.

Capnostream 20p monitors communicated through Bernoulli IDM 3400 serial-to-ethernet bridges. Bridges would poll the monitor at the rate of of 10 times per minute and communicate data wirelessly to the Bernoulli Analytics Software located within the enterprise data center. Rules were maintained within the Bernoulli data storage and rules library and recalled for application against the received data. The serial-to-ethernet bridges were outfitted with barcodes to facilitate a patient-to-device association workflow using a Gryphon barcode reader. Also shown are the data collection and alarm signal communication from the monitors to the dashboard displayed within the telemetry bunkers. Abbreviations used: ADT, admission discharge transfer; etCO2, end-tidal carbon dioxide; SpO2, peripheral capillary oxygen saturation.

Figure 1.

Capnostream 20p monitors communicated through Bernoulli IDM 3400 serial-to-ethernet bridges. Bridges would poll the monitor at the rate of of 10 times per minute and communicate data wirelessly to the Bernoulli Analytics Software located within the enterprise data center. Rules were maintained within the Bernoulli data storage and rules library and recalled for application against the received data. The serial-to-ethernet bridges were outfitted with barcodes to facilitate a patient-to-device association workflow using a Gryphon barcode reader. Also shown are the data collection and alarm signal communication from the monitors to the dashboard displayed within the telemetry bunkers. Abbreviations used: ADT, admission discharge transfer; etCO2, end-tidal carbon dioxide; SpO2, peripheral capillary oxygen saturation.

The workflow for data collection and display were as depicted in Figure 1. Data received by the bridges were communicated wirelessly to the analytics software, where the data were stored and processed against predefined alarm signal thresholds. Following evaluation against stored rules, threshold breaches then were communicated to a web-based dashboard within the tele-tech bunkers.

Upon arrival in postanesthesia care units (PACUs) from surgery, patients were attached to the monitors via pulse oximetry cuff and nasal cannula. The monitor assemblies with bridges attached were mounted on roll stands and wheeled to the patient bedsides in PACUs. There, the nurse would barcode the patient's wrist bracelet, barcode the label on the bridge, verify the coupling on the barcode liquid-crystal display (LCD) screen, and validate the coupling of the bridge to the patient by pressing a button on the barcode scanner. All data collected from the monitor were then transmitted to the analytics software until the patient was discharged from capnography monitoring. The workflow for associating the patient with the monitor is shown in Figure 1 in the online supplement.

A dedicated Datalogic Gryphon I GM4100-HC wireless barcode scanner was used for the patient association process. This barcode scanner has an LCD screen that allows for the display of patient identifiers and the bridge serial number. A workflow was created whereby the identity of the patient, the assigned name of the bridge, and a confirmation request were displayed on the Gryphon LCD screen with which the nurse could interact. The purpose of this coupling was to assign patient name and identifiers to the data collected from the monitor so that all alarm signals and data would be issued unambiguously and displayed through the tele-tech dashboard. The dashboard view is shown in Figure 2 in the online supplement. This view is created by the analytics software in the form of a web page that is displayed within the telemetry bunkers for continuous monitoring purposes. The green rectangle indicates an active patient for which no alarm signals are being issued and contains information pertaining to the patient, including name and identifiers.

Figure 2.

Combined respiratory rate (fR) and end-tidal carbon dioxide (etCO2) measurements for patient 14 in the initial pilot. Apnea events can be seen between 15 and 20 minutes, briefly at 38 minutes, briefly between 70 and 72 minutes, and for the extended period between 245 and 250 minutes, when both etCO2 and fR dropped to 0.

Figure 2.

Combined respiratory rate (fR) and end-tidal carbon dioxide (etCO2) measurements for patient 14 in the initial pilot. Apnea events can be seen between 15 and 20 minutes, briefly at 38 minutes, briefly between 70 and 72 minutes, and for the extended period between 245 and 250 minutes, when both etCO2 and fR dropped to 0.

The clinical user to which the patient is assigned can also be optionally displayed. Clicking on the green rectangle causes a vital display box to be made visible. This vital display box depicts the current value of measurements together with any issued alarms. Note that the findings in this display can be tailored to the purposes of the clinical end user. In this case, those parameters measured from the monitor are shown, and included etCO2, fR, SpO2, and PR. Measurements were initiated in PACUs. Both etCO2 and fR measurements were made using FilterLine etCO2 Sampling Lines (Medtronic, Minneapolis, MN) designed for use with Microstream-enabled capnography monitors, and measurements of PR and SpO2 were made using Nellcor finger-based pulse oximetry sensors (Medtronic).

The rules and thresholds detailed in the next section were applied to the data received from the monitors and visually depicted in the telemetry dashboard view. Threshold breaches that corresponded to clinically significant cautionary or low-priority levels (i.e., breaches that corresponded to a cautionary level of concern) were shown in yellow within the dashboard (Figure 3 in the online supplement). The specific text associated with the middleware-generated alarm signal event is customizable and is contained within the rule maintained by the analytics software.

Figure 3.

Example of self-correcting threshold breaches: individual, nonadjacent measurements exceeding a specified end-tidal carbon dioxide (etCO2) threshold. These occurrences may have been due to noise, bad measurements, or a combination of both.

Figure 3.

Example of self-correcting threshold breaches: individual, nonadjacent measurements exceeding a specified end-tidal carbon dioxide (etCO2) threshold. These occurrences may have been due to noise, bad measurements, or a combination of both.

Clicking within the vital display box triggered the flowsheet view (Figure 4 in the online supplement). The data may be viewed graphically or in tabular format. Data were collected and displayed for the duration of the patient encounter.

Figure 4.

Example of sustained threshold breach: Measurements do not self-correct to the normal range. This behavior may be indicative of an actionable event or may result from a systematic problem, such as suspect measurements.

Figure 4.

Example of sustained threshold breach: Measurements do not self-correct to the normal range. This behavior may be indicative of an actionable event or may result from a systematic problem, such as suspect measurements.

Data that exceeded urgent (i.e., high-priority) levels based on the predefined rules associated with a specific measurement appear as a red rectangle in Figure 5 in the online supplement. Of note, the color scheme adopted (green = normal; yellow = warning or low priority; red = urgent or high priority) was entirely customizable. The selection of the threshold levels, color schemes, and audible notifications were defined by the clinical team prior to deployment and initiation of the study. Note that in the case of the alarms displayed in this figure, a second notification is shown: that of an alarm threshold breach. This threshold breach was defined using the analytics software as an indicator when a value exceeded a specific threshold. Thus, this threshold breach could be displayed, a custom alarm signal threshold breach associated with a rule could be displayed, or both could be displayed simultaneously. Again, clicking on the vital display box causes the flowsheet to be displayed (Figure 6 in the online supplement). This figure displays the occurrence of a low etCO2 threshold breach occurring at a measurement level of 15 mmHg.

Figure 5.

Multiple measurements plotted against time. Combinatorial alarm signals are issued only when the threshold criteria for all parameters are met simultaneously. The combinatorial alarm signal can have the effect of reducing the overall alarm reporting frequency due to false alarms because the potential for individual measures all meeting their threshold criteria simultaneously is hypothesized to be more likely associated with a true-positive event than false alarm signals.

Figure 5.

Multiple measurements plotted against time. Combinatorial alarm signals are issued only when the threshold criteria for all parameters are met simultaneously. The combinatorial alarm signal can have the effect of reducing the overall alarm reporting frequency due to false alarms because the potential for individual measures all meeting their threshold criteria simultaneously is hypothesized to be more likely associated with a true-positive event than false alarm signals.

The alarms as defined using the rules engine provide for both a color metaphor indicating urgency (e.g., red color) and a numeric value, both of which are customizable. For the purpose of this study, green levels were identified with an alarm level of 0, indicating no alarm condition. Low-priority levels were associated with an alarm level of 1, indicating cautionary or moderate urgency. Finally, high-priority levels were associated with an alarm level of 2, indicating immediate urgency.

Technical alarm signals also were captured and communicated visually. For example, during the course of patient care, nursing might need to pause the monitor (e.g., when patient is eating). In such instances, a notification would be displayed on the dashboard indicating that the patient was away from the monitor. Clinical staff elected for such notifications to be displayed in purple (Figure 7 in the online supplement). This was done to remove any confusion and to prevent any nonactionable alarm signals associated with sensors removed from patients being sent to the telemetry units.

Rules, Alarms, and Alarm Settings

Table 2 in the online supplement summarizes the monitor alarm condition limit settings on key clinical parameters. These parameters remained unchanged from the initial pilot through the follow-on clinical study. Note that while the alarm condition limit settings are most often referred to as high and low priority, the nomenclature used here (i.e., “urgent” and “caution”) was that preferred by the clinical staff.

Table 2.

Distribution of measurements exceeding monitor alarm settings. All technical alarms were removed. Threshold breaches were dominated by respiratory rate and low end-tidal carbon dioxide (etCO2). Abbreviation used: SpO2, peripheral capillary oxygen saturation.

Distribution of measurements exceeding monitor alarm settings. All technical alarms were removed. Threshold breaches were dominated by respiratory rate and low end-tidal carbon dioxide (etCO2). Abbreviation used: SpO2, peripheral capillary oxygen saturation.
Distribution of measurements exceeding monitor alarm settings. All technical alarms were removed. Threshold breaches were dominated by respiratory rate and low end-tidal carbon dioxide (etCO2). Abbreviation used: SpO2, peripheral capillary oxygen saturation.

Data were collected and processed at the rate of 10 sets of measurements per minute. The dashboard-displayed alarm signal events presented breaches of the clinically relevant thresholds as defined in Table 2 in the online supplement. However, although cross-over was recognized between technical and clinical alarm signals (i.e., alarm signals intended for direct intervention by the clinical end user [nurse or respiratory therapist]), technical alarm signals also could cause and trigger clinical alarm signals. Also, in certain cases (e.g., flow blockage, disconnects), the clinical end user must be made aware of these so as to intervene and correct.

The initial rollout involved an evaluation period of 2 weeks, after which the methods, workflow, and impressions of clinical and technical staff were analyzed. A summary of the alarm and data findings are provided in Table 3 in the online supplement. Not all patients who were placed on capnography monitoring remained for the duration of their stays in MSUs, and in certain cases, patient data were disqualified for technical or clinical reasons (e.g., suspect measurements, technical glitches, patients removed from capnography for clinical reasons). Patients ranged widely in terms of duration on capnography monitoring, from as little as an hour to several days. The summary results shown in Table 3 in the online supplement are for patients whose data were not disqualified based on the criteria of suspect measurements, technical glitches, or premature removal from capnography monitoring. To illustrate the character of the findings, time-based plots of the data associated with one patient (patient 14) are shown in Figure 2 (fR and etCO2) and Figures 8 and 9 in the online supplement (PR and SpO2). The accompanying thresholds are displayed as overlays on these plots to add context to the raw measurements.

Table 3.

Middleware-generated alarm signals based on sustained alarm delays, parametrized against five different alarm delay levels. A sharp reduction in quantities of middleware-generated alarm signals is achieved when increasing sustained delay from 42 to 48 seconds. Note that even with sustained alarm delay upwards of 60 seconds, the number of alarm threshold breaches remains considerably high (in excess of 1,000 total). Abbreviations used: etCO2, end-tidal carbon dioxide; SpO2, peripheral capillary oxygen saturation.

Middleware-generated alarm signals based on sustained alarm delays, parametrized against five different alarm delay levels. A sharp reduction in quantities of middleware-generated alarm signals is achieved when increasing sustained delay from 42 to 48 seconds. Note that even with sustained alarm delay upwards of 60 seconds, the number of alarm threshold breaches remains considerably high (in excess of 1,000 total). Abbreviations used: etCO2, end-tidal carbon dioxide; SpO2, peripheral capillary oxygen saturation.
Middleware-generated alarm signals based on sustained alarm delays, parametrized against five different alarm delay levels. A sharp reduction in quantities of middleware-generated alarm signals is achieved when increasing sustained delay from 42 to 48 seconds. Note that even with sustained alarm delay upwards of 60 seconds, the number of alarm threshold breaches remains considerably high (in excess of 1,000 total). Abbreviations used: etCO2, end-tidal carbon dioxide; SpO2, peripheral capillary oxygen saturation.

The expectation at the outset of the initial pilot was that the alarm signals issued on each patient would be manageable and informative. This assumption was quickly dismissed in practice, as is evident from the summary shown in Table 3 in the online supplement in which, in some cases, as many as 427 alarms per hour were issued on a patient (patient 12) corresponding to threshold breaches (principally) in low respiratory rate and low etCO2. An overall average across all patients of 182 alarms per hour was determined from the raw counts of the data. By far, low etCO2 and both low and high respiratory rate alarm signals dominated. At the end of this evaluation period, it was determined that the quantities of alarm signals being issued to the telemetry units were so high as to make the process of telemetry monitoring of these patients untenable using the method of communicating measurement threshold breaches. Hence, a decision was made to investigate ways in which to reduce alarm signals and provide only actionable notifications to appropriate clinical staff.

A decision was made to investigate ways in which to reduce alarm signals and provide only actionable notifications to appropriate clinical staff.

A key point to be communicated in regard to monitoring patients receiving intravenous or oral opioid administration is that a substantial amount of artifact results from patient movement. The nasal cannula and SpO2 sensors are a prime source of the technical (and ultimately clinical) alarm signals. For example, movement of the nasal cannula can result in the issuing of false readings, particularly if adjusted by the patient so that only one nostril is recorded. Patients report these cannula to be particularly uncomfortable and often were required to wear them for many hours, leading to patients (or families) adjusting the tubing. Thus, alarm signals associated with artifact occurred. The sounding of alarms at the point of care was intended to awaken or otherwise notify the patient and/or family of situations that might be dangerous to the patient (e.g., cessation of breathing), which can occur in cases OSA or CSA. However, as was observed, some patients who were asleep or sedated would not respond to these audible in-room alarm signals.

Postpilot Data Investigation

A poststudy investigation of the data was performed to gain insight into alarm signal source and, ultimately, to assist in aiding reduction. The most immediate observation was that the vast majority of the alarm signals resulted from threshold breaches of two or more consecutive measurements. These consecutive alarm breaches, also termed sustained alarm signals, could be used to filter out certain types of artifact that were spurious and nonrepeating. Of note, a single sustained alarm might consist of multiple instances of measurements breaching a predefined threshold. Thus, a single sustained alarm would be issued when multiple measurements breached the individual thresholds of a single parameter. In the case of instantaneous alarm signals, these were counted as individual alarms when they were distinctly issued, nonconsecutive events.

The decision to intervene when non–self-correcting measurements appear is a clinical one, and the duration of the trend at or beyond the threshold also is a question of clinical judgment.

The clinical team hypothesized that individual, self-correcting measurements (i.e., those that breached a threshold, then returned to normal range) should not be communicated. Rather, only those instances where measurements continuously trended below/above a specified threshold for a predefined period of time should be communicated. This sustained alarm signal communication is frequently used by physiologic monitors (and the subject monitor) to reduce the likelihood of noise being communicated. Figures 3 and 4 illustrate the concept of self-correcting threshold breaches of etCO2 versus a sustained etCO2 threshold breach, respectively. In the case of self-correcting threshold breaches, individual measurements may exceed an identified parameter threshold, but the following measurement will correct to the normal, within-threshold value. In the case of sustained threshold breaches, a set of adjacent measurements will breach a threshold and “trend” at or below the threshold for a particular period of time. They may eventually self-correct or may not self-correct. While the individual threshold breaches could be due to artifact, they also could be the harbinger of a true event. The decision to intervene when non–self-correcting measurements appear is a clinical one, and the duration of the trend at or beyond the threshold also is a question of clinical judgment. The exact cause of such threshold breaches is not known unless an actual observation of the patient takes place during or surrounding the occurrence of the event to validate and verify cause. In the case of individual measurements exceeding a threshold, the cause could be bad measurements, movement of the patient, or issues with the bedside monitor. In the case of sustained measurements exceeding a threshold, the cause could, again, be bad measurements, movement of the patient, a true patient event, or issues with the bedside monitor.

To validate the hypothesis that sustained alarm signal generation would considerably reduce the overall number of alarm signals issued, the data of the initial phase pilot were retrospectively evaluated against sustained delays of 30 seconds. The results are shown in Table 4 in the online supplement. As hypothesized, the overall number of events per hour dropped to less than one-third those reported in Table 3 in the online supplement, with a maximum number of 122 alarm signals per hour (patient 14).

Table 4.

Combinatorial middleware-generated alarm signals based on respiratory rate (fR), peripheral capillary oxygen saturation (SpO2), and end-tidal carbon dioxide (etCO2). Combinational alarm signals (generated by calculating hypopneic hypoventilation both with and without hypoxia) were weighted primarily by low fR and low etCO2. Combinatorial alarm signals also were parametrized against sustained delays of 6, 12, 18, and 24 seconds. *fR ≤ 6 breaths/minute, SpO2 ≤ 85% etCO2 ≤ 15 mmHg. † fR ≤ 6 breaths/minute, etCO2 ≤ 15 mmHg.

Combinatorial middleware-generated alarm signals based on respiratory rate (fR), peripheral capillary oxygen saturation (SpO2), and end-tidal carbon dioxide (etCO2). Combinational alarm signals (generated by calculating hypopneic hypoventilation both with and without hypoxia) were weighted primarily by low fR and low etCO2. Combinatorial alarm signals also were parametrized against sustained delays of 6, 12, 18, and 24 seconds. *fR ≤ 6 breaths/minute, SpO2 ≤ 85% etCO2 ≤ 15 mmHg. † fR ≤ 6 breaths/minute, etCO2 ≤ 15 mmHg.
Combinatorial middleware-generated alarm signals based on respiratory rate (fR), peripheral capillary oxygen saturation (SpO2), and end-tidal carbon dioxide (etCO2). Combinational alarm signals (generated by calculating hypopneic hypoventilation both with and without hypoxia) were weighted primarily by low fR and low etCO2. Combinatorial alarm signals also were parametrized against sustained delays of 6, 12, 18, and 24 seconds. *fR ≤ 6 breaths/minute, SpO2 ≤ 85% etCO2 ≤ 15 mmHg. † fR ≤ 6 breaths/minute, etCO2 ≤ 15 mmHg.

However, the nagging clinical question surrounding this mathematical reduction in alarm signals was how many measurements should be sustained before reporting an event as clinically actionable to a care provider? A clinical discussion and survey of the literature resulted in a decision to consider 30 seconds as the sustained alarm threshold. That is, if measurements in any individual parameter were sustained at or below/above the threshold value for 30 seconds or longer, then an alarm signal should be issued on an individual parameter.

In addition, however, alarm signals should only be issued when the data are known to be valid. That is, if a known technical alarm condition (e.g., nasal cannula off patient, pulse oximetry cuff off patient, calibration error) occurs, these technical alarm conditions should be taken into account via communication to the clinical engineering staff and any data collected during a period in which such suspect measurements were obtained should not be used to calculate whether a sustained clinical alarm signal should be issued on a patient. Thus, several questions were identified:

  • Is the selected duration of the sustained alarm delay sufficient to reduce alarm signal traffic while not concomitantly introducing a patient safety concern?

  • Which alarm signals should be communicated to care providers?

  • How long should alarm signals be communicated before escalating?

  • Can other measures or combinations of data provide an early indicator of patient respiratory compromise?

A discussion regarding how best to validate these questions led to the conclusion that answering them in one investigation may not be possible. However, it may be possible to further quantify the findings to provide greater insight into the management of this cohort of patients, which can then lead to answers. Thus, a follow-on study was formulated. This new pilot study would be focused on attempting to quantify the alarm signals associated with the selected sustained delays. To gain feedback on causality and validity of the alarm signals, dedicated nursing research staff would receive the alarm signals via phone and would proceed to each patient to observe and validate or otherwise identify the root cause of the alarm signal condition.

Follow-on Clinical Study

Recognizing the prevalence of artifact and the large quantity of alarm signals that can result, a decision was taken to perform a follow-on study with human participants. A steering committee of clinical and technical staff was brought together to study and evaluate methods to reduce alarm signals and to communicate only those that were deemed truly actionable. To reduce alarm signals, a series of rules was developed requiring that multiple conditions be met before middleware-generated alarm signals were transmitted to clinical staff. These rules were reviewed and tested on simulated data. The rules combined characteristics of the measured parameters based on a study of the data collected from the initial rollout and from the literature. The follow-on study received approval from the VHS Institutional Review Board (reference identifier G15020), and informed consent was obtained from all participants.

Patient Eligibility

This clinical trial was designed to evaluate the use of alarm signals generated using sustained and combinatorial alarm rule conditions over a period of 4 weeks at one hospital within the health system (Virtua Memorial). Patients were enrolled in the study based on existing diagnoses of OSA or meeting the STOP-BANG criteria for OSA (Table 5 in the online supplement). A total of 31 patients were recruited during this period, four of whom were disqualified due to irregularities in data collection or clinical issues. In addition, one patient was discharged from the PACU and one patient transferred from the PACU to the ICU. Thus, a total of 25 patients were placed on capnography monitoring in the PACU. A summary of the patient population (17 women and 8 men, mean age 60 years) is provided in Table 1. Most patients received intravenous hydromorphone or morphine sulfate for pain management. For this follow-on study, the dashboard display was replaced with a direct communication to research nursing using Voice-over-Internet-Protocol (VoIP) phones (Cisco).

Table 5.

Hierarchy of device-issued alarm signals, sorted from highest to lowest quantities

Hierarchy of device-issued alarm signals, sorted from highest to lowest quantities
Hierarchy of device-issued alarm signals, sorted from highest to lowest quantities

Data Collection Workflow

Data collection and processing workflow were unchanged from the initial pilot. However, communication workflow was changed (shown in Figure 10 in the online supplement). The workflow was similar to that of the initial pilot study, except that the processed alarms were communicated in text from the analytics software to the VoIP phones. When an alarm signal was transmitted to the phone, patient name, identifier, location, and cause of the alarm signal were displayed on the phone LCD screen. After receiving the alarm signals, research nurses visited patients. Patients requiring intervention were referred to the floor nursing staff, or in the case of a need for rapid response, the standard intervention process was followed.

Sustained and Combinatorial Rules and Alarm Signals Calculation

The clinical team considered the use of sustained (or persistent) and combinatorial (or multicriteria) alarms to identify actionable events. In the case of sustained alarms, criteria for reporting necessitated that a condition be maintained for a predetermined period of time. Clinical team members hypothesized prior to the initiation of the clinical study that a sustained alarm signal delay of 30 seconds was a key measure of whether an alarm originating from monitors was actionable. Therefore, an alarm condition would not be signaled to research nursing staff unless it persisted for at least 30 seconds.

Combinatorial alarms are defined as those for which multiple criteria must be met simultaneously before an alarm condition is signaled. For example, a given parameter or set of parameters must meet specific conditions simultaneously for the combinatorial alarm signal to be issued. Examples include no-breath events combined with low pulse oxygen saturation events.

Figure 5 illustrates how combinatorial alarms can reduce alarm signal quantity. Plots of individual parameters (quantity is arbitrary and is taken as the number N) are plotted against time. Each of these parameters is displayed with threshold overlays on each respective plot. A signal value exceeding a specified threshold constitutes a threshold breach. Individually, the quantity of threshold breaches could translate into alarm signal notifications per each parameter, with the total quantity of issued alarm signals being the sum of all threshold breaches for each parameter. Thus, a total of 16 threshold breaches are counted, corresponding to a total of 16 alarm signal events. The simultaneous occurrence of these threshold breaches can form the basis for a combinatorial alarm signal associated with all N parameters. Depending on the specific parameters, the simultaneous occurrence of these threshold breaches could constitute a clinically significant event. In this simplified illustration, six (shaded circles) of the overall 16 (shaded plus clear circles) threshold breaches occur simultaneously, thereby reducing the total quantity of issued alarm signal events from 16 to six.

All alarm signals (i.e., sustained individual alarms, alarm signals calculated based on combinatorial criteria) were communicated to research nursing staff. As might be expected, the total number of sustained alarm signals outweighed combinatorial alarm signals by a wide margin. The results of both types of alarm signals are reported below.

As patients were selected to participate in the study on the basis of passing the STOPBANG criteria, it was anticipated that some patients would experience a respiratory-related adverse event. The literature suggests that most incidents of hypoxic events are preceded by respiratory depression19; thus, a decision was made to include several specific combinatorial calculations in the assessment to evaluate how our data compared with identified combinatorial alarm conditions and alarm conditions reported in the literature. These included:

  • 1. Calculated apnea (30 seconds) alarm signal based on measured etCO2 and fR.

  • 2. Calculated no-breath and no-pulse combinatorial alarm signal.

  • A modified form of hypopneic hypoventilation (mHHH) alarm signal, both with and without hypoxia, defined here as the combinatorial assessment of etCO2 less than 15 mmHg, fR fewer than 6 breaths/minute, and SpO2 less than 85%.

  • A modified form of bradypneic hypoventilation (mBHH) alarm signal, both with and without hypoxia, defined here as the combinatorial calculation of etCO2 greater than 65 mmHg, fR fewer than 6 breaths/minute, and SpO2 less than 85%.

  • 3. Sustained alarms on etCO2, pulse rate, SpO2, and fR, whereby alarms would only be issued in the event of a non–self-correcting threshold breach of 30 seconds' minimum duration.

Of note, bradypneic and hypopneic hypoventilation are sometimes referred to as type 1 and type 2 hypoventilation, respectively, and, as the sensitivity of etCO2 has been documented to precede the onset of a hypoxic event, these are clinically significant findings that may require intervention.11,19 

In all cases, technical alarm signals were filtered or otherwise removed from the clinical alarm signal reporting. However, a report of all alarm signals issued from the monitors is provided in the following section.

Findings

Continuously measured parameters included arterial oxygen saturation measurement via finger-based SpO2, etCO2, PR, and fR. Key metrics included sustained levels of hypoxia (SpO2 ≤85%), hypocarbia (etCO2 ≤15 mmHg) and hypercarbia (etCO2 ≥65 mmHg), respiratory rate (fR ≤6 breaths/minute; fR ≥24 breaths/minute), and bradycardia (pulse rate ≤40 bpm) and tachycardia (pulse rate ≥150 bpm).

Data collected from the monitors showed that low etCO2 thresholds were breached in 5% of measurements, while fewer than 1% of measurements exceeded the high etCO2 threshold of 65 mmHg. Approximately 6% of measurements fell below the low fR threshold of 6 breaths/minute, and 15% fell below 8 breaths/minute. Fewer than 1% of measurements fell below SpO2 threshold of 85%. Of all PR measurements taken, fewer than 1% fell below the bradycardia threshold of 40 bpm. No patients experienced high PR alarm limit breaches.

The correlation coefficient between machine-issued low etCO2 alarm signal counts and low fR alarm signal counts was determined to be 0.95. That is, low etCO2 correlated highly with low fR, which might be expected in a patient population predisposed to OSA. No significant correlation was found between low SpO2 and low etCO2, including monitor-issued no-breath alarms (i.e., fR = 0), and it is hypothesized that because these patients were monitored so closely and SpO2 drops relatively slowly compared with etCO2, interventions occurred before hypoxia was experienced by most patients within the study population.

A total of 193,177 data points were measured per parameter. Average fR was 15 ± 6 breaths or respirations per minute, average etCO2 was 36 ± 9 mmHg, average pulse rate was 72 ± 15 bpm, and average SpO2 97 ± 3%. Key findings of middleware-generated alarm signals are summarized in Tables 2 through 4. Low respiratory rate presented the largest source of alarm signals generated, followed by low etCO2. This tended to make sense given that the patient selection criteria weighted patients heavily toward diagnosis of OSA (i.e., most of the recruited patients had OSA).

A parametric was run on the data to assess the effect of sustained alarm delay (Table 3). Measurements were collected every 6 seconds. Parametrically, as sustained delay was varied from 18 to 60 seconds, the number of sustained alarm signals decreased. Hence, as the data collection interval was a set of measurements every 6 seconds, this meant that data collection occurred at 0, 6, 12, 18, 24, 30, 36, 42, 48, 54, and 60 seconds (corresponding to 0 seconds in the next minute). The drop in number of sustained alarm signals achieved between 42 and 48 seconds of sustained delay was significant (by one order of magnitude).

Sustained alarm signals for single parameters remained high, even through 42 seconds of delay. The question arose as to whether patient safety is put at risk by increasing the delay time up to 1 minute. During this clinical trial, clinical staff were most comfortable with 30-second sustained delays and were not desirous of increasing beyond that threshold. At 30 seconds of sustained delay, however, a large quantity of alarm signals occurred that research nursing reported to be mostly nonactionable.

The combinatorial rules produced alarm signal quantities that were at least an order of magnitude and several factors fewer than the sustained alarm signals on individual parameters (Table 4). More significantly, the combinatorial alarm signals also were parametrized as sustained delays from 6 through 24 seconds. A 6-second combinatorial alarm implies two adjacent measurements that meet the threshold criteria specified by the combinatorial alarm rule. Of note, the quantities of combinatorial alarm signals are significantly lower than the single-parameter sustained alarm signals. Also, the total number of alarm signal quantities decreased much more quickly with sustained delay compared with the single-parameter sustained alarm signals, as the criterion that multiple parameters must simultaneously meet the specified thresholds is much more stringent.

Patients were experiencing extended periods of single-parameter threshold breaches that were continuous or repetitive in nature. In most cases, these single-parameter alarm signals did not signify clinically meaningful events requiring intervention, as verified by research nursing staff.

The results yielded the following observations:

  • Sustained middleware-generated alarm signals (i.e., persisting for 30 seconds or longer) remained quite high in this cohort, even when the duration of the persistent delay was increased from 18 through 42 seconds, with a steep reduction in alarm signals achieved at 48 seconds of sustained delay (Table 3). A significant reduction in middleware-generated alarm signals occurred when the sustained delay was increased to 48 seconds. Patients were experiencing extended periods of single-parameter threshold breaches that were continuous or repetitive in nature. In most cases, these single-parameter alarm signals did not signify clinically meaningful events requiring intervention, as verified by research nursing staff.

  • Low mean respiratory rate correlated with both low and high etCO2 measurements (Figure 11 in the online supplement). The average respiratory rate determined in this cohort corroborated similar findings.20 

  • A total of 5% of measurements fell below the 15 mmHg etCO2 threshold, while fewer than 1% of measurements exceeded the hypercarbia threshold of 65 mmHg (Figure 12 in the online supplement), indicating that most of the thresholds breaches on etCO2 were related to hypocarbia. This made sense to the researchers, as most patients were diagnosed with OSA. Further, low respiratory rate was the source of most respiratory alarm signals (Figure 13 in the online supplement).

  • Pulse oximetry and pulse rate alarm signals occurred far less often than either respiratory rate or etCO2 alarms (Figures 14 and 15, respectively, in the online supplement).

  • No patients were found to have met the mBHH combinatorial alarm threshold, though a number of patients met the mHHH threshold criteria (Table 4). Strong correlation was observed between middleware-calculated mHHH alarm signals and machine-issued low etCO2 alarm signals (mHHH and machine-issued low etCO2: 0.71; mHHH and machine-issued low fR: 0.68; machine-issued low fR and machineissued low etCO2: 0.95).

  • Combinatorial alarm signals associated with these events for 6- and 12-second sustained delay are 106 and 59, respectively (Table 4). Of note, when SpO2 was removed from the combinatorial calculations, the number of alarm signals was large (>4,500) until sustained delay of 18 seconds was used, at which point the quantity of alarm signals decreased to 209. In both of these combinatorial rule calculation cases, the quantity of alarm signals was much more manageable and provided a hint as to how to improve alarm management. However, by using hypopneic hypoventilation combinatorial alarm signals, more than a 98% reduction over 30-second sustained middleware-generated respiratory fR and etCO2 alarm signals was achieved.

  • Finally, among seven patients who were identified as requiring some form of intervention, four had true respiratory distress, with one patient requiring intervention (i.e., administration of naloxone hydrochloride to reverse the effect of the opioid and placement on noninvasive ventilator support). These patients were discovered as a result of the sustained alarms, and combinatorial alarms were triggered for these patients as well.

Table 6 in the online supplement summarizes the alarm signals issued by the monitors. A review of overall quantities of machine-issued alarms revealed the following:

  • There were a total of 8,181 machine-issued no-breath (7,065) and low-fR (1,116) alarm signals compared with 5,832 middleware-generated low-fR and no-breath alarm signals. The comparative middleware-issued alarm signals were based on a 30-second sustained delay. Thus, to a degree, differences in reports can be attributed to differences in calculations and algorithms contained within the monitor versus the rules created within the analytics software. By decreasing the sustained delay to 18 seconds, the result is 6,739 middleware-generated low-fR alarm signals.

  • Machine-issued high-fR alarm signals totaled 2,173 compared with 2,698 middleware-issued alarm signals (based on 30-second sustained delay).

  • Machine-issued low etCO2 alarm signals totaled 3,065 compared 4,635 middlewareissued alarm signals (based on 30-second sustained delay). Again, by decreasing sustained delay to 18 seconds, the result is 5,361 middleware-generated alarm signals.

  • The quantities of SpO2 disconnects (8,527) and low battery (8,301) were significantly high, thereby suggesting the need for workflow improvements in terms of compliance and preparation of monitoring devices before use.

  • CO2 flow disconnect alarm signals (3,051) corresponded to the number of low etCO2 alarm signal reports.

  • Although the number of middleware-issued etCO2 alarm signals exceeded that of machine-issued alarm signals, the alarm signals based on combinatorial rules (Table 4) suggested that the combinatorial middleware-generated alarm signals can be reduced greatly compared with either single-parameter sustained or machine-issued alarm signals.

The distribution of all machine-issued alarm signals, considering both clinical and technical alarms, also is of interest. Table 5 summarizes the machine-issued alarm signals issued cumulatively across the patient population. A significant number of nasal cannula and pulse oximetry sensor disconnects were issued during the study. These are clinically actionable. The large quantities resulted from the fact that the alarm was communicated each time the monitor was polled until the condition was corrected. The quantities, together with the type of technical alarm signals issued, motivate the need for a workflow that integrates clinical engineering into the alarm reporting infrastructure, so that technical alarms (e.g., low battery notifications) can be communicated to staff to ensure that technical intervention takes place in a timely manner. The existence of calibration errors also suggests the need for a systematic quality control equipment turnaround process by identifying the type and number of monitors that fail to meet specifications before use on the next patient.

Conclusion

Two successive investigative studies were conducted to identify the utility and practical implementation of remotely communicated sustained and combinatorial capnography alarm signals, in order to reduce alarm signal load and improve patient surveillance. The results suggested that combinatorial alarm signals based on multiparameter assessment reduced overall load better than individual-parameter sustained alarm signals and appeared to be more effective at identifying at-risk patients. Refinement of these combinatorial alarm signals requires further investigation to validate whether they can serve as a safe and effective means of detecting respiratory depression among patients at risk for opioid-induced respiratory depression. It is encouraging that none of the monitored patients required rapid response team activation for events that were undetected by either the sustained or combinatorial alarms. However, the authors acknowledge that the study population was limited and that further investigation in a larger population is merited.

The results suggested that combinatorial alarm signals based on multiparameter assessment reduced overall load better than individual-parameter sustained alarm signals and appeared to be more effective at identifying at-risk patients.

The number of patients needed to draw valid conclusions is a matter of frequent debate. Pilot studies of comparable size have been performed, thereby establishing some precedent as to the validity of results concerning this patient population.6,21,22 Further, the current population was highly targeted toward patients who were either diagnosed to have OSA or who may have been susceptible to respiratory depression (as assessed using the STOP-BANG criteria). This places the population bias more on the likelihood of expecting, versus not expecting, apneic events. Considering that the objective of this pilot study was to inform and refine assumptions, these results should help inform future rollout efforts.

In-room alarm settings and audible alarms were not changed on any monitor. Somewhat anecdotally, in a subset of patients, the in-room capnography alarms did not have the effect of arousing them from a state of respiratory depression. Hence, the remotely communicated alarms to the nursing phones had the added effect of a safety net.

Finally, to mitigate the effects of repetitive trended alarm signal fatigue, as was experienced, the authors recommend using annunciated alarms based on combinatorial rules, particularly with this patient cohort, and to implement a protocol whereby response and escalation occurred promptly upon receiving combinatorial alarm signals. If etCO2 monitoring is to be used, then the authors recommend thoroughly training clinical and biomedical staff, patients, and families on its application, use, and limitations.

Acknowledgments

To Virtua Health System for supporting the study and to the following Virtua employees for their assistance: Anne Miller, RN, research nurse; Jennylin Schott, RN, research nurse; Tom Decker, systems integration architect, medical device integration; and Kyle Parker, systems integrator, medical device integration.

Conflict of Interest

Beaton, Venella, Williams, and Zaleski are employed by Bernoulli Enterprise, Inc., which was the middleware vendor for data collection and alarm processing used in the study.

Financial Disclosure

Research nursing time associated with the study was funded by Bernoulli Enterprise, Inc. The authors did not receive compensation for research participation, and the research was not grant supported.

References

References
1.
Baron
L
,
Decker
T
,
Supe
D
,
et al
.
AAMI Foundation Safety Innovations: Virtua Health: Implementing Capnography in Low Acuity Settings
. .
2.
Lenz
G
,
Heipertz
W
,
Epple
E.
Capnometry for Continuous Postoperative Monitoring of Nonintubated, Spontaneously Breathing Patients
.
J Clin Monit
.
1991
;
7
(
3
):
245
8
.
3.
Stoelting
RK
,
Weinger
MB.
Special Editorial: Dangers of Postoperative Opioids: Is There a Cure?
Available at: www.apsf.org/newsletters/html/2009/summer/01_opiods.htm. Accessed March 27, 2017
.
4.
Weinger
MB
,
Lee
LA.
No Patient Shall Be Harmed By Opioid-Induced Respiratory Depression
.
Available at: www.apsf.org/newsletters/html/2011/fall/01_opioid.htm. Accessed March 21, 2017
.
5.
Krauss
B
,
Hess
DR.
Capnography for Procedural Sedation and Analgesia in the Emergency Department
.
Ann Emerg Med
.
2007
;
50
(
2
):
172
81
.
6.
Soto
RG
,
Fu
ES
,
Vila
H
Jr
,
Miguel
RV.
Capnography Accurately Detects Apnea During Monitored Anesthesia Care
.
Anesth Analg
.
2004
;
99
(
2
):
379
82
.
7.
Stocki
D
,
Matot
I
,
Einav
S
,
et al
.
A Randomized Controlled Trial of the Efficacy and Respiratory Effects of Patient-Controlled Intravenous Remifentanil Analgesia and Patient-Controlled Epidural Analgesia in Laboring Women
.
Anesth Analg
.
2014
;
118
(
3
):
589
97
.
8.
Overdyk
FJ
,
Carter
R
,
Maddox
RR
,
et al
.
Continuous Oximetry/Capnometry Monitoring Reveals Frequent Desaturation and Bradypnea During Patient-Controlled Analgesia
.
Anesth Analg
.
2007
;
105
(
2
):
412
8
.
9.
Wong
M
,
Mabuyi
A
,
Gonzalez
B.
First National Survey of Patient-Controlled Analgesia Practices
. .
10.
Oridion
.
Monitoring End Tidal CO2 in Post Surgical Patients: Continuous Breath Monitoring Increases Patient Safety
. .
11.
Garah
J
,
Adiv
OE
,
Rosen
I
,
Shaoul
R.
The value of Integrated Pulmonary Index (IPI) monitoring during endoscopies in children
.
J Clin Monit Comput
.
2015
;
29
(
6
):
773
8
.
12.
Association for the Advancement of Medical Instrumentation
.
Clinical Alarms 2011 Summit: Convened by AAMI, FDA, TJC, ACCE, and ECRI Institute
. .
13.
AAMI Foundation
.
Clinical Alarm Management Compendium
.
Available at: www.aami.org/alarmcompendium. Accessed March 26, 2017
.
14.
MacDonald
I.
Hospitals rank alarm fatigue as top patient safety concern
. .
15.
Bennett
CL
,
McNeer
RR.
PT-SAFE: A Software Tool for Development and Annunciation of Medical Audible Alarms
.
Anesth Analg
.
2012
;
114
(
3
):
576
83
.
16.
Olympio
MA.
APSF Workshop Recommends New Standards
.
Available at: www.apsf.org/newsletters/html/2004/winter/01workshop.htm. Accessed March 29, 2017
.
17.
Oridion
.
Standards Mandating Capnography Monitoring
. .
18.
Kodali
BS.
Capnography Outside the Operating Rooms
.
Anesthesiology
.
2013
;
118
(
1
):
192
201
.
19.
Deitch
K
,
Rowden
A
,
Damiron
K
,
et al
.
Unrecognized Hypoxia and Respiratory Depression in Emergency Department Patients Sedated For Psychomotor Agitation: Pilot Study
.
West J Emerg Med
.
2014
;
15
(
4
):
430
7
.
20.
McGrath
SP
,
Pyke
J
,
Taenzer
AH.
Assessment of continuous acoustic respiratory rate monitoring as an addition to a pulse oximetry-based patient surveillance system
.
J Clin Monit Comput
.
2016
May
3
.
[Epub ahead of print]
.
21.
Kyriacou
PA
,
Jones
DP
,
Langford
RM
,
Petros
AJ.
A Pilot Study of Neonatal and Pediatric Esophageal Pulse Oximetry
.
Anesth Analg
.
2008
;
107
(
3
):
905
8
.
22.
Restrepo
RD
,
Otaibi
AA
,
Wittnebel
L
,
et al
.
Correlation between ABG Parameters and the Integrated Pulmonary Index in a Medical-Surgical ICU in Saudi Arabia
.
Society for Technology in Anesthesia Annual Meeting
,
Phoenix, AZ
,
2013
.

About the Authors

Dana Supe, MD, MBA, DABSM, is program director for clinical patient safety at Virtua Health in Mount Holly, NJ, and medical director of operations at Virtua Memorial Hospital of Burlington County. Email: dsupe@virtua.org

Leah Baron, MD, is chief of the department of anesthesiology at Virtua Memorial Hospital in Mount Holly, NJ, and a member of the Burlington Anesthesia Associates/Mednax. Email: lbaron@virtua.org

Tom Decker is a systems integration architect at Virtua Health in Marlton, NJ. Email: tdecker@virtua.org

Kyle Parker is a systems integrator at Virtua Health in Marlton, NJ. Email: kparker@virtua.org

Jeanne Venella, DNP, MS, RN, CEN, CPEN, is chief nursing officer at Bernoulli Enterprise, Inc., in Milford, CT. Email: jvenella@bernoullihealth.com

Sarah Williams, RT, is senior product manager at Bernoulli Enterprise, Inc., in Milford, CT. Email: swilliams@bernoullihealth.com

Kari Beaton, RN, is director of enterprise solutions at Bernoulli Enterprise, Inc., in Milford, CT. Email: kbeaton@bernoullihealth.com

John Zaleski, PhD, CAP, CPHIMS, is chief analytics officer at Bernoulli Enterprise, Inc., in Milford, CT. Email: jzaleski@bernoullihealth.com

Supplementary data