We present a black-box messaging test approach employed to achieve a level of rigor which improves—if not assures, given no optionality—correct data exchange. In particular, we verify that physiological information derived and communicated via messaging from a source medical device (e.g., an infusion pump) or healthcare information system, to another medical device (e.g., a patient monitor) or healthcare information system, which consumes or makes use of the data, is syntactically and semantically correct. In other words, the structure of information exchanged within the healthcare system is compliant to a defined specification(s) and the information meaning conveyed and interpreted by the consumer is exactly the same and as intended by the source. Our approach for developing a test system to validate messages is based on constraining identified and recognized specifications. The test system validation performed uses codified assertions derived from the specifications and constraints placed upon those specifications. To first show conformance—which subsequently enables interoperability—these assertions, which are atomic requirements traceable by clause to the base specifications, are employed by our medical device test tools to rigorously enforce standards to facilitate safe and effective plug-and-play information exchange.
At the U.S. Department of Commerce's (DOC) National Institute of Standards and Technology (NIST), researchers are collaborating with medical device experts to facilitate the development and adoption of standards for medical device communications throughout the healthcare enterprise as well as integrating it into the electronic health record (EHR). We have developed test tools1 and a modeling application, including a corresponding electronic representation of an international standard's information model2, which provides several important capabilities leading toward device interoperability.3
Conformance testing is a key step leading to, although not guaranteeing, interoperability.4 Sparked by involvement over the past several years of working with medical device domain experts and vendors who participate in standards development organizations (SDOs) and use established standards such as Health Level 75 (HL7) and ISO/IEEE 11073 Health informatics—Point-of-care medical device communication6 and Personal health device communication,7 an approach used to identify testable assertions derived from such standards and constrained by important use cases is presented.
Medical devices need to communicate with tens, if not hundreds, of other devices of varying makes, models, and modalities, a reality that has large market and substantial healthcare implications. Acute point-of-care settings such as a hospital's intensive care unit, a patient's bedside, or personal telehealth location require each class of medical device to use the same terminology and data organization to seamlessly and reliably communicate physiological data. Healthcare communication standards that address plug-and-play medical device interoper-ability are critical. While providing the groundwork to enable device communication, standards are developed in an open-ended manner (and for good reason). It is our contention, through experience in software testing, that only until standards and defined specifications are constrained (ultimately removing all optionality to create profiles) can the desired “guarantee” of syntactic and semantic correctness be achieved.
Conformance test methodologies are being employed by NIST via software test tools to help get closer to that guarantee. These tools are publicly available and being used by the medical device industry to ensure that critical devices correctly implement the medical device standards. A consortium of medical device vendors using these test methodologies to successfully meet a level of compliance to standards sufficient to achieve truly efficient interoperability is the Integrating the Health-care Enterprise-Patient Care Device (IHE-PCD) domain.8 Correct implementation of standards leads to effective exchange of critical physiologic data derived from the patient at the device and exchanged throughout the health-care enterprise. As more and more devices are able to achieve plug-and-play capabilities, clinicians are empowered to focus more on the patient and less on the devices. The ability to reliably and effectively integrate data from a broad range of point-of-care devices will ultimately lead to a reduction in medical errors and the associated loss of life.
Medical Device Communication Standard
The ISO/IEEE 11073 Health Informatics—Point of Care and Personal Health Medical Device Communication (x73) standards define a set of information objects and functions needed for medical device communication. Such a family of standards was developed to address the critical need of enabling medical devices to share physiologic data between devices and computerized healthcare information systems. Two primary parts of these standards used in our approach pertain to the domain information model (DIM)9,10 and nomenclature.11 The DIM provides the objects and object relationships necessary to abstractly define a device. It defines the overall set of information objects as well as the attributes, methods, and access functions which are abstractions of real-world entities in the domain of medical devices and device communication. Nomenclature defines terminology and codes used across classes of medical devices.
IHE-PCD Integration Profiles, Technical Frameworks, and Integration Statements
IHE-PCD participant vendors define use cases in which at least one actor is a regulated patient care device. IHE Integration Profiles are defined and provide the necessary detail to enable demonstration, through implementation (i.e., specific implementations of established standards to achieve integration goals), of important use cases. The IHE-PCD Integration Profiles, defined in IHE-PCD Technical Framework documents,12 organize and leverage the integration capabilities that can be achieved by coordinated implementation of communication standards, such as HL7 and x73. They provide precise definitions of how standards are constrained and may be implemented to meet specific clinical needs.13
Based on these specifications which constrain the reference standards, the IHE conducts cyclical interoperability testing events; NIST test tools are used in the IHE-PCD domain to evaluate conformance to the specified Integration Profiles and executed test cases. If successful, industry participants publish IHE Integration Statements, which indicate their system's conformance and can be useful for medical device procurers during their evaluation.
Currently within the IHE-PCD, participants are working on several Integration Profiles14 including Device Enterprise Communication (DEC) with options to Patient Identity Binding (PIB) and Subscribe to Patient Data (SPD), which provides a subscription/data filtering mechanism; Alarm Communication Management (ACM); Point-of-care Infusion Verification (PIV) addressing infusion safety issues such as “five rights of medication safety;”15 Implantable Device Cardiac Observation (IDCO); and Rosetta Terminology Mapping (RTM), which provides a mapping between proprietary device semantics to the x73 nomenclature and associated co-constraints (e.g., associated reference identifier, terminology code, unit(s) of measurement, lead sites where measurements may be taken, and enumerations).
The Need for Conformance Test Tools
Conformance and interoperability testing of medical device data communication is essential leading to long-term value propositions which include:
Integrity of data—automatic population of all information systems—reducing medical errors
Automating systems to capture clinical data into EHRs, thus saving time for clinicians
Access to patient data across devices and systems so custom communication interfaces can be eliminated, thus allowing for best of breed and even plug-and-play devices
Improving agility of enterprises to meet varied patient loads
Improving life-cycle cost of ownership
To address real-world semantic interoperability, the transfer of data must be (in many cases) near real-time data from a gateway to an electronic medical record (EMR) system in a rich, accurate, and consistent manner. To first show conformance which subsequently enables such interoperability, test tools that rigorously enforce defined specifications to facilitate safe and effective plug-and play interoperability are necessary.
Constraining Specifications to Derive Testable Assertions
Our approach for developing a test system to validate messages is based on constraining identified specifications. The validation is defined by assertions derived from the specifications and constraints placed upon the specifications. The premise at getting to any level of rigor is that specifications are complete (as possible) and constrain open-ended assertions. The more well-formed, formal, and complete the specifications, the greater level of rigor can be achieved by the test system.
Figure 1 shows the specifications used by our test tooling to address message validation in the IHE-PCD domain environment. Messages being exchanged contain physiologic observations. The messages (i.e., defined using HL7 Version 2) are tested against the specifications which define the standards used, any domain specific specifications, terminology and nomenclature employed, and any specific values or value sets being conveyed as identified in test cases.
It is unrealistic to assume all standards and specifications are correct or mature to a level of “complete.” However as specifications are implemented and a collaborative, iterative, feedback process occurs, so too can the rigor-level and coverage provided by the test tools via updates, enhancements, and issue resolution. Should we consider different enterprise-level testing outside of IHE, other specifications as made available by the domain could be integrated in a similar manner into the test tooling.
Based on the specifications and any constraints identified in those specifications, messages are validated by the test system which employs various test components. For example, an HL7 message derived from an infusion pump (or generated from the pump system or gateway) is evaluated against the HL7 standard for its syntax and semantics, the x73 standard for terminology, terminology co-constraints, and information model (i.e., the device object hierarchy), and the test case for any specific values or attributes.
Specification Ingredients Employed
The recipe for correctly effecting validation of messages in our approach calls for specification ingredients as shown in Figure 2. Given the IHE-PCD domain and integration goals, these specifications include the HL7 Version 2 standard for message definition and value sets, the x73 standard for medical device nomenclature, the IHE-PCD Technical Framework documents for message transaction definition, and the IHE-PCD test cases for specific value definition.
These specifications define and lead to what we call “testable assertions,” which are atomic test requirements traceable to the aforementioned specifications. Identified test assertions are codified into “context validation” files. Context validation files are defined in XML and provide the precise assertions that the test system uses as input to a validation engine, which performs the validation service (and in the future, other services such as message generation). Each testable assertion references the specific clause in the base specification or ingredient of our recipe. Test reports are generated by the test tool identifying the specific error within the message along with a reference to the clause from which the assertion is based.
HL7 Standard, Value Sets, and IHE Technical Framework Assertions
Validation of the device information carried within the HL7 messages occurs at both the syntactic and (low-level) semantic levels. Messages are validated against defined value sets and what we refer to as “failure types.” The test tool uses validation context files codified in XML to perform message validation checks against the HL7 V2 standard, value set tables, and any further constraints defined by IHE-PCD with the Technical Framework documents (e.g., “local” value sets not defined in HL7) for message transactions. Validation of failure types include:
Version (e.g., the HL7 version and IHE-PCD Technical Framework Integration Profile)
Message Structure ID (e.g., the HL7 message type [MSH.9 element] defined in the profile shall match what's in the message)
Message Structure (e.g., the message shall have a valid HL7 message structure, including correct usage, correct cardinality, and correct element name)
Usage (e.g., HL7 ‘R’ elements should be present; ‘X’ elements should not be present in the message)
Cardinality (e.g., elements shall be present at least the minimum times and at most the maximum times specified in the conformance profile)
Length (e.g., the value of the element shall have a length equal or less than the value specified in the profile)
Datatype (e.g., for the HL7 data types ‘NM,’ ‘DT,’ ‘DTM,’ ‘SI,’ and ‘TM’—the value of the element shall match the regular expression defined in the standard)
Data (e.g., the value of the element shall match a constant specified in the profile, a value set specified in a table, or a value or a regular expression specified in the message validation context [derived from a test case])
Table Not Found (e.g., an error when a referenced table can't be found in the table files HL7 or local defined set of allowable tables)
The above attributes defined in HL7 are often referred to as HL7 Conformance Profiles. They are produced using third-party software and define the constraints desired when implementing HL7 messages. HL7 Conformance Profiles may be used as input into the test tools and become testable assertions enforced by the validation engine.
Common Medical Device Information Model and Nomenclature Assertions
In considering and developing our test approach one of the overarching goals is to achieve semantic interoperability—communicate medical device data using a single unified nomenclature and semantic model that can be rigorously defined and enforced to facilitate safe and effective plug-and play interoperability.
This is where the aforementioned x73 domain information model and nomenclature are an essential ingredient. Today, nearly all vendors have an internal, and often proprietary, representation of device and corresponding device generated information. Vendors can correctly and consistently map information that has been generated, either by the same or another device make or model or system, by applying a common model and nomenclature based on recognized standards. Further, from a black-box testing perspective in which medical device observations are exchanged via messaging, rigorous validation can be applied using those very same standards which are constrained via profiles by communicating entities. Profiles may include device profiles as defined in x73 (x73-103xx16 series of device specializations for point-of-care health devices, such as an infusion pump or ventilator or x73-104yy17,18,19,20,21,22,23,24,25 series of device specializations for personal health devices, such as a weight scale or pulse oximeter) or Integration Profiles as defined by the IHE-PCD domain.
One of the profiles, Rosetta Terminology Mapping, identifies the nomenclature and provides a containment hierarchy to abstractly represent medical devices as defined in the x73 standard. This Integration Profile provides the testable assertions of device information carried within the observation segments (i.e., HL7 Version 2 OBX segments). These constraints or test assertions lead to test validation context files as depicted in Figure 2 and provide traceability to the x73 standard's nomenclature and information model.
IHE-PCD Transaction and Test Case Defined Assertions
IHE-PCD domain defines the technical framework documents and test cases in which vendors are evaluated against. The framework documents define and constrain (at the HL7 usage level) transactions (i.e., HL7 messages). IHE-PCD defined test cases identify specific values required in vendor implementations and demonstrated during the test event(s). The corresponding validation context information contained in the test cases is codified in XML as testable assertions.
Advancing the Approach
The presented test approach of validating static messages by constraining specifications is foundational. However, there is much work to be done to achieve greater levels of rigor. Test tool enhancements are forthcoming to advance functionality from a static message checker over what we refer to as in an “instance test environment,” which essentially evaluates message against the specification(s) from which the message is based (e.g., conformance testing an HL7 V2 message) to an “isolated system test environment.” Ultimately we strive to provide a test infrastructure providing a “peer-to-peer environment.”26
Isolated system type testing involves real scenarios in which transactions exchanged as well as behavior exhibited by the system under test (SUT) are evaluated. Typically this involves a meaningful scenario in which transaction exchange occurs between the SUT and test system, thus isolating the SUT. Protocol conformance and functional behavior, including features and operation, are evaluated by the test system according to identified specifications. For example, each step within a scenario may involve one or more messages transmitted to/from the SUT to/from the test system. The test system views the SUT as a black box, evaluating transactions and behavior (i.e., expected syntax and semantic content).
Peer-to-peer system testing involves multiple (two or more) SUTs interacting, with the test system involved as a proxy. In addition to the functionality of isolated system testing, peer-to-peer includes the complete application environment to achieve interoperability testing. Peer-to-peer test environment may include interacting with many services including a database, network communication, other hardware, applications or systems as appropriate.
Another software application27,28,29 we developed at NIST allows users to define medical device profiles in strict accordance to the x73 standard. The resultant XML file provides abstract representations of real devices defined using x73 nomenclature and with an x73 DIM containment hierarchy. Using the application's interface a user can define and constrain the device abstract representation to a particular class of device and furthermore to the specific make and model.
We are considering approaches to integrate this device representation with the message validation test tools. Such integration would enable validation of specific device classes for each IHE-PCD use case that is appropriate for that device class. Conformance testing device classes, makes, and models are important as devices exhibit variant behavior, even when applied to the same test case (within a use case, Integration Profile, or scenario).
In related efforts, NIST has developed validation tooling being used in several other domains (including the Health and Human Services' National Health Information Network, the IHE IT Infrastructure domain30 Cross Enterprise Document Sharing [XDS],31 Patient Identifier Cross Referencing [PIX],32 and Patient Demographics Query [PDQ]33).
Developing our initial set of test tools has been enhanced through our involvement with industry consortium. As active participants in IHE, standards development organizations and other consortium, NIST researchers have gained invaluable insight into the needs and issues of medical device vendors, clinicians, clinical engineers, and, in general, the healthcare community. We continue to focus our attention on open consensus forums and processes based on open consensus standards. We are actively monitoring other related work34,35 and efforts using related medical device standards, focused on critical issues such as patient safety and device risk analysis. We believe our approach offers benefits to most of these efforts, if not all. As we continue to build upon and enhance the test tooling, the likely hood of interoperability increases. It is our hope that, “As we build it, they will come.”
About the Authors
John J. Garguilo and Sandra I. Martinez are computer scientists with the National Institute of Standards and Technology (NIST). At the time this paper was written, Maria Cherkaoui was a NIST guest research computer scientist from ISIMA (Institut Supérieur d'Informatique, de Modélisation et de leurs Applications in France. E-mails: John.email@example.com, firstname.lastname@example.org, and email@example.com