Clinical laboratories are obligated to implement Logical Observation Identifier Names and Codes (LOINC), an informatics standard used to uniquely identify laboratory tests. The historical progress of laboratories in achieving this goal is unknown.
To evaluate the implementation of LOINC by clinical laboratories with attention to LOINC's adoption, diversity, and correctness over time.
We aggregated data from 130 facilities within the Veterans Health Administration (VA), an early adopter of LOINC, during a 20-year period (1999–2018). To assess the adoption of LOINC, we calculated the annual proportion of tests and results without a LOINC. To assess the diversity of LOINC, we counted the yearly number of distinct LOINCs in active use. To assess the correctness of LOINC over time, we compared the assigned LOINCs to a manually reviewed gold standard for each year.
We reviewed a total of 586 000 tests and 9.162 billion results. LOINC adoption, measured as a proportion of both tests and results, improved over time (P < .001). In the final year reviewed, 85% (172 142 of 202 125) of laboratory tests and 99% (547 229 066 of 551 205 087) of results had LOINCs. The number of distinct LOINCs in active use from 1999 to 2018 increased 2.78-fold from 4502 to 12 503 (P < .001). Correctness generally improved but varied considerably by test and across time.
The adoption of LOINC has improved during the past 2 decades. More diverse LOINCs were associated with increased adoption and were a challenge to keep up-to-date. The correctness of LOINCs has improved but remains an issue that likely necessitates supplemental review for most applications.
In the era of the electronic medical record, clinical laboratories are required to generate standardized data.1,2 Standardized laboratory data ease patient care transfers, enable epidemiologic research at a national scale, and facilitate the monitoring of infectious disease outbreaks.3–5 Laboratories previously reported test results on paper, and this imposed limits on the sharing of information.6 The advent of automated analyzers and computerized databases made it possible to exchange laboratory results within and among health care systems.6 This phenomenon uncovered a lack of standardization among laboratories.7–9 Clinical laboratories, which commonly shared the same manufacturer's reagents and equipment, devised their own unique names for laboratory tests.10 A standardized vocabulary was needed to identify equivalent tests and encourage interoperability among clinical laboratories.11–13
Logical Observation Identifiers Names and Codes (LOINC) is a standardized vocabulary that has become the de facto standard to uniquely identify a laboratory test.14 For example, the test for serum or plasma creatinine, performed millions of times a day in laboratories around the world, is defined in LOINC as 2160-0. The standard has been in continuous development since 1994 by the Regenstrief Institute15 and was derived from existing nomenclatures provided by the European Clinical Laboratory Data Exchange Standard.16 LOINC aims to create unambiguous identifiers in the form of unique codes for each laboratory test.16
In the United States, legislation prompted clinical laboratories to adopt LOINC.17 The Veterans Healthcare Administration (VA) began to adopt LOINC in 1999, and in 2001 the Under Secretary for Health mandated the adoption of LOINC throughout the VA. The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 also promoted widespread adoption of LOINC through incentivized reimbursement for nongovernment health care systems.18
Although the theoretical utility of LOINC is widely recognized, clinical laboratories appear to have struggled with its practical implementation.19 During the last 20 years, research groups have devised numerous methodologies to simplify the nontrivial task of assigning a LOINC to each laboratory test.20–24 As an example, a lengthy 259-page book was authored by the director of LOINC as a “step-by-step” guide to LOINC implementation.25 Recent literature continues to chronicle the ongoing efforts of authors to implement LOINC.26–36
Presently, important details of LOINC's implementation remain unanswered:
How has the adoption of LOINC progressed over time?
Has the diversity of LOINC, in terms of the number of codes in active use, changed over time?
When a LOINC is assigned to a test, is it correct?
We provide longitudinal answers to these questions from the viewpoint of the VA, the largest integrated health care system in the United States. As an early adopter of LOINC, the VA's progress may provide insight into the current state of LOINC's implementation and serve to guide the implementation of future standardized terminologies.
MATERIALS AND METHODS
Study Setting and Design
Study data originate from the VA's Corporate Data Warehouse (CDW), a relational database that aggregates laboratory results from all facilities across the health care system.37–39 Facilities provide care to patients living in the Philippines and all 50 United States including Alaska and Hawaii. The data in our study include all inpatient and outpatient laboratory results obtained between October 1999 and November 2018. Nearly all facilities contributed data for the full duration of the study.
P values were obtained by using 2-tailed tests.
LOINC Adoption Over Time
LOINC adoption refers to the assignment of a LOINC to a laboratory test. For instance, before LOINC, no test had a LOINC and the adoption of LOINC was 0% of tests, but after a test had a LOINC assigned, adoption increased. Results are assigned the LOINC of their parent test when made available for clinician review. In this way, changes to LOINC assignments are not retrospectively applied to results. Adoption is reported as both the proportion of tests and the proportion of results. Relatively few tests (eg, sodium, hematocrit) produce most results, and so these metrics provide different perspectives on the data.
Each of the 130 facilities within the health care system maintains a separate menu of laboratory tests. Individual facilities maintain the LOINCs associated with the tests they offer. Maintenance of LOINC generally occurs when a test is added or modified. Assignment of a LOINC to a test at one facility does not affect the LOINC assignment of that same test at any other facility.
We considered LOINC adoption to have occurred if a test had a LOINC assigned at any point of the year, even if the assignment occurred mid-year or was removed by year's end.
We required at least 1 reportable result per year for each test. This approach excluded tests from the analysis if they were outdated but still present on a facility's test menu.
LOINC Diversity Over Time
LOINC diversity refers to the number of different LOINCs in active use. To calculate this metric, we counted the number of distinct LOINCs with at least 1 reportable result in each calendar year.
Correctness of LOINCs Over Time
A correctly assigned LOINC accurately describes the test to which it is assigned. To establish the correctness of a test's LOINC assignment, we established a gold standard for comparison (Figure 1, A and B). The gold standard included tests that met the study definition independently of their LOINC assignment. This allowed us to calculate the true-negative rate, in addition to true- and false-positive rates, which could be calculated by review of only the LOINC assignments.
Ten representative laboratory tests were chosen to establish a manually reviewed gold standard (Supplemental Table 1 [column 1] found in the supplemental digital content file containing 18 figures and 2 tables at www.archivesofpathology.org in the April 2020 table of contents). Before review, we established a working definition for each of the 10 tests to improve interannotator agreement upon manual review. Our test definitions were generally permissive. For example, we included together all pH tests in blood, serum, and plasma, in contrast to LOINC, which offers more nuanced codes that distinguish between pH in blood and pH in blood adjusted for a patient's temperature (LOINCs 11558-4 versus 49701-6). We adopted permissive criteria because test descriptions are generally not specific enough to distinguish between clinically similar codes, and we sought to avoid overinterpretation of the available data.
For each of the selected tests, we established a list of synonymous LOINCs. For example, we added both LOINCs mentioned previously (11558-4 and 49701-6) to our test definition of pH in blood, serum, or plasma (BSP). (See Supplemental Table 1, column 2.)
Two annotators reviewed the laboratory metadata (eg, laboratory test name, specimen, result values, result datatype [eg, numeric, ordinal]) from the CDW for all laboratory tests to identify the 10 tests selected for review. This process followed previously published methods.27,40 Disagreements were adjudicated by consensus. We report the interannotator agreement using Cohen κ.
We compared the assigned LOINC to the gold standard for each of the 10 selected tests. If the gold standard agreed with the assigned LOINC the comparison was considered a true positive. (We omitted reporting the true-negative category because it accounted for a very large proportion of the total data.) Disagreement between the assigned LOINC and gold standard represented either a false positive or false negative. A false positive, for example, occurred when a test for pH in urine had a LOINC for pH in blood. In contrast, a false negative occurred when a test labeled as pH in BSP had a LOINC for pH of urine. False negatives also occurred when a test did not have a LOINC assigned. (See Supplemental Table 2 for examples.)
LOINC Adoption Over Time
We reviewed 9.162 billion results from 586 134 tests contributed by 130 facilities from 1999 to 2018. LOINC adoption improved as measured by both tests (1999: 57 800 of 88 914 [65.01%]; 2018: 172 142 of 202 125 [85.17%]) and results (1999: 55 082 674 of 65 698 155 [83.84%]; 2018: 547 229 066 of 551 205 087 [99.29%]) (Figure 2). These trends were both highly statistically significant (P < .001). Throughout the review period, LOINC adoption by result consistently exceeded LOINC adoption by test.
We investigated the difference in LOINC adoption by result and test. In 2018, approximately 13 779 of 29 983 tests (46.0%) without a LOINC had only 1 reported result (Figure 3, A). Results from these tests comprised a negligible percentage of the total results without a LOINC reported in that year (Figure 3, B). Variation in the adoption of LOINC varied by site (Supplemental Figure 3).
LOINC Diversity Over Time
The number of distinct LOINCs in active use from 1999 to 2018 increased 2.78-fold from 4502 to 12 503 (P < .001) (Figure 4, A). Many new LOINCs were added in 2005 and 2016 (Figure 4, B). Upon further investigation of these 2 years, we found the addition of new codes corresponded to the adoption of a more up-to-date version of LOINC (Figure 5).
Correctness of LOINCs Over Time
Ten representative tests were chosen to evaluate the correctness of LOINC assignments: calcium in BSP, choriogonadotropin in BSP, codeine in urine, erythropoietin in BSP, gentamicin in BSP, Giardia in stool, lipase in BSP, pH in BSP, rapid plasma reagin (RPR) in BSP, and theophylline in BSP. Upon review, we found these tests represented a reasonable cross section of tests performed within our health care system. The selected tests included high volume (ie, calcium), low volume (ie, Giardia), high complexity (ie, codeine), and low complexity (ie, pH) tests. For each of the 10 tests, we established a manually curated gold standard to which we compared the assigned LOINCs.
LOINCs used in the comparison were identified and are shown associated with their corresponding test in Supplemental Table 1. Lipase in BSP had only 1 synonymous LOINC (ie, 3040-3), while choriogonadotropin in BSP had 31. The average test had 10 synonymous LOINCs.
To establish the gold standard, we reviewed all tests within our health care records (n = 586 134), identifying a total of 7008 relevant tests. Cohen κ for intrarater agreement to establish the gold standard was 0.999. In comparing the gold standard to the assigned LOINCs, we found true positives (4478 of 7008 [64%]), false negatives (2098 of 7008 [30%]), and false positives (375 of 7008 [5%]). Tests with ambiguous information were excluded from the analysis (eg, “RPR/VDRL” [Reactive Plasma Reagin/Venereal Disease Research Laboratory]: 2 distinct syphilis tests combined into a single name; 57 of 7008 [1%]). Mismatches between the assigned LOINC and the gold standard are listed in Supplemental Table 2.
To investigate the variability of correct LOINC assignments among tests, we graphed the false-negative, false-positive, and true-positive rates for each of the 10 tests for a period of 20 years (Figure 6, A and B, and Figure 7, A and B; Supplemental Figures 1 and 2). Overall, LOINC assignments tended to improve over time. For any given test, however, the correctness of assigned LOINCs varied without a clear pattern. For example, codeine demonstrated an increase in the false-negative rate in the last 10 years of the review. Tests for choriogonadotropin and RPR had upwards of 20% false-negative rates in recent years (eg, choriogonadotropin in 2013 had 10 367 of 53 674 [19.3%]). Calcium achieved a true-positive rate of 98.9% (10 694 946 of 10 813 868) in 2018, the last year reviewed. Owing to the high volume of approximately 10.8 million reported results for calcium, more than 100 000 results (1.1% of the total) did not have a correct LOINC. (Supplemental Figure 2 contains additional graphs showing the absolute, rather than percentage, change in result counts.)
In a review of 9.162 billion laboratory results produced by 130 facilities during a 20-year period, we found the adoption of LOINC has improved. In 2018, the last year reviewed, 85% (172 142 of 202 125) of laboratory tests and 99% (547 229 066 of 551 205 087) of reported results had a LOINC assigned. Like adoption, the diversity of LOINCs also increased over time. In contrast, the accuracy of LOINCs varied considerably across time and among tests.
LOINC Adoption Over Time
Multiple factors could explain the increase in LOINC adoption. Administrative factors include the 2001 mandate by the under secretary of health for the VA to adopt LOINC,41 the assignment of personnel from the VA's Office of Information Technology to improve LOINCs, and increased demand for LOINC mapping by secondary users of laboratory data. Technical factors, such as the increased availability of applicable LOINCs and experience acquired with LOINC by laboratory personnel, may have also improved adoption.
Although adoption in 2018 reached 99% of yearly results, the remaining 1% without a LOINC represented a sizable 4 million results. While much of the literature on LOINC has historically reported a percentage,* an absolute number may be more appropriate given the size of our dataset. In addition, a sizable proportion of these 4 million results without a LOINC originated from moderate volume tests (11–1000 results per test per year per facility) (Figure 3). These tests may add considerable value if mapped to LOINC, since the clinical importance of a test is not necessarily related to the volume in which it is performed. For instance, a human immunodeficiency virus (HIV) diagnostic test is generally performed in moderate volume relative to other tests, but a diagnosis of HIV carries lifelong implications for a patient's medical care.
LOINC Diversity Over Time
The Regenstrief Institute, the curators of LOINC, released multiple updates to LOINC over the timeframe of our study. These newer versions of LOINC contained additional codes, which were adopted and led to increased code diversity. We specifically noted 2 large increases in the diversity of LOINCs following the adoption of newer versions of LOINC in 2005 and 2016 (Figure 4, bottom, and Figure 5).
While the increased availability of codes may have increased the adoption of LOINC, the added diversity likely complicates the retrieval of laboratory data. A search for choriogonadotropin in BSP conducted at the beginning of our study would use 13 LOINCs, while the same search at the end of our study would require 31 LOINCs. The difference of 18 codes represents the additional LOINCs added during our study. Similar complications arise from the appearance of methodless and specimenless codes, LOINCs that do not specify a method or specimen, respectively.43
Two additional issues related to new versions of LOINC involve the deprecation of codes over time and the timely adoption of new LOINC versions. Historically, the VA has avoided modification of existing LOINCs, including LOINCs deprecated by Regenstrief. The persistence of deprecated LOINCs in the VA has become an additional challenge for end users. The VA did not adopt newer versions of LOINC from 2005 to 2015 (Figure 5).
Correctness of LOINCs Over Time
We found a general improvement in the correctness of LOINCs over time across the 10 tests we reviewed (Figures 6 and 7), which corresponds to the increased adoption of LOINC across the health system (Figure 2 versus Figures 6 and 7). The correctness and adoption of LOINC trended in a similar fashion because a common reason for an incorrect LOINC was the lack of adoption (ie, the test did not have a LOINC) (Supplemental Table 2). Over time the correctness of LOINC improved when viewed across all tests, but the correctness of LOINC varied considerably at the level of an individual test (Figure 6, A and B, and Figure 7, A and B; Supplemental Figure 1). At the test level, correctness is heavily influenced by an individual facility's correct assignment of LOINC. If a facility performing a test in high volume omits the LOINC for a given test, many results are affected, and the correctness of LOINC for that test declines.
Most (2098 of 2473; 85%) observed errors in the LOINCs consisted of incorrectly assigned or missing codes (false negatives). Three possible reasons for a false negative include the absence of an available LOINC, an omission by the site in assigning a LOINC, or the assignment of an incorrect LOINC. Tests with missing LOINCs represent the most common occurrence (Supplemental Table 2). The VA used an outdated version of LOINC for much of the time under review, which may have contributed to the number of missing LOINCs (Figure 5).
A minority of errors (375 of 2473; 15%) were classified as false positives. These tests have a LOINC from Supplemental Table 1 (column 2), but the gold standard disagrees with this assignment. They appear to be random errors.
As LOINC currently exists at the VA, a variable degree of supplementary mapping is required before secondary use of LOINC can be carried out. The degree of supplementary mapping required cannot be known a priori for a given test. We have observed up to a 15% error rate, a rate which would be deemed unacceptable for most applications. The requirement that the error rate must be minimized before data can be used increases the complexity of research projects.
To improve the correctness of LOINC, multiple stakeholders must collaborate. Reference laboratories, which perform tests on the behalf of other laboratories, were early adopters of LOINC.43,47 Test manufacturers, who have an especially large influence on the correct assignment of LOINCs, have more recently begun to provide LOINCs for the tests they produce. The United States Food and Drug Administration (FDA) has not mandated manufacturers to provide LOINCs, instead they have collaborated with industry experts to improve the pairing of tests with LOINC.48,49
Laboratories have the ultimate responsibility for correct LOINC assignments, yet they depend heavily on reference laboratories and test manufacturers to fulfill this duty. Laboratories should request LOINCs from the manufacturers of the tests they perform. The FDA has discouraged manufacturers from publicizing LOINCs for “uncleared or unapproved indications,” but manufacturers may provide these codes when requested by a laboratory. Although perhaps warranted, unintended consequences of the FDA's position include the need for laboratories to request LOINCs from test manufacturers through a private communication, in contrast to visiting a public Web site. This reduces the transparency between a manufacturer's test and its LOINC.
A possible solution to this impasse would be the maintenance of a map between LOINC and a manufacturer's test by a neutral third party (eg, Regenstrief, IVD [In Vitro Diagnostic] Industry Connectivity Consortium, or College of American Pathologists [CAP]). CAP, with its knowledge of a laboratory's test manufacturers, could help quality control LOINC assignments.
Our study has several limitations. The scope of our study includes several aspects related to LOINC's implementation, such as the adoption, diversity, and correctness of LOINCs. A proficient implementation is needed to make LOINC useful, but alone, is insufficient. Additional studies on the usability of LOINC are needed to assess the benefit LOINC provides to its users. Secondly, the challenges of the VA's implementation of LOINC, although geographically and chronologically diverse, may not be representative of other health care systems.
We investigated the implementation of LOINC with a focus on its adoption, diversity, and correctness. Adoption has improved over time. Presently more than 99% of results reported by laboratories within our health care system have a LOINC. Increased adoption likely resulted from a variety of factors including more diverse codes introduced in updates to LOINC. While useful to promote adoption, staying up-to-date with the most recent version of LOINC presents an additional challenge. The correctness of LOINCs varied considerably by test and across time. Secondary data users should exercise caution in data extraction by LOINC alone. Investigations into the usability of LOINC, particularly with respect to its perceived benefits (eg, interoperability, secondary data analysis), are needed.
We would like to acknowledge the help of Scott Gigante, PhD candidate (Computational Biology & Bioinformatics, Yale University), for manuscript review.
Supplemental digital content is available for this article at archivesofpathology.org in the April 2020 table of contents.
The authors have no relevant financial interest in the products or companies described in this article.
The research presented in this paper is original work conceived by the authors. The viewpoints presented in this paper reflect the viewpoints of this research group and do not reflect opinions of the United States Government.