State of the Art in Trueness and Interlaboratory Harmonization for 10 Analytes in General Clinical Chemistry

A long-standing objective of clinical laboratory medicine has been to produce harmonized test results among laboratories and methods.1 Harmonized results are those that agree within clinically acceptable limits among different methods and laboratories. This goal has received renewed emphasis with the development of clinical guidelines for interpretation of laboratory tests that require standardized test results regardless of methodology used for the measurement procedures. Recent recommendations from the European External Quality Assessment–Organizers Working Groups A and B,2,3 the European Union directive for in vitro diagnostic medical devices,4 the International Organization for Standardization,5 and the Clinical and Laboratory Standards Institute6 have recognized the objective of harmonized results and the importance of using commutable reference materials for method calibration traceability and for proficiency testing (PT) programs. A PT specimen is commutable when the mathematical relationships between the results from different measurement procedures are equivalent for the PT material and for native clinical samples.7 

Matrix interferences are generally present in the materials used by PT providers at the present time. Matrix interferences are caused by alterations to the serum composition during the processing steps in material manufacturing. As a result, a PT survey specimen may have a serum matrix with artificial differences not present in native clinical samples. Consequently, a PT specimen may not be commutable among all the methods used for routine analysis. Literature reports have documented that processed PT materials are frequently noncommutable and the occurrence of noncommutability has been unpredictable for any particular material-method combination.8 Direct comparison of results between methods is not possible using noncommutable materials.

We report here results for 10 routine analytes from a PT survey of approximately 6000 laboratories using a wide range of routine instrument-method combinations conducted by the College of American Pathologists (CAP) in October 2003. This survey included 1 specimen that was a specially prepared fresh frozen serum (FFS) pool intended to be commutable among all methods and thus able to evaluate the state of the art in harmonization of results and their traceability to reference measurement procedures.

Specimen C-02 was prepared by Aalto Scientific (Carlesbad, Calif) to CAP specifications as previously described9 using a modification of the Clinical and Laboratory Standards Institute C37-A guideline.10 Briefly, off-the-clot serum was collected from 670 donors; individual units were frozen and stored up to 2 months at −70°C; units were thawed, recentrifuged, pooled, aliquotted, filtered to 0.22 μm, and refrozen at −50°C within 27 hours of thawing. The FFS vials were stored 6 months at −40°C and bulk shipped with frozen carbon dioxide to the survey packager.

The general chemistry survey specimens were prepared to CAP specifications by Bio-Rad Laboratories (Irving, Calif). They were frozen serum prepared from defibrinated human plasma. Briefly, frozen plasma collected at donor centers was converted to serum in large batches that were dialyzed to remove anticoagulants. Various analyte concentrates, including nonhuman components, were added to the base serum protein material to prepare 2 master pools containing minimum and maximum desired quantities of each analyte. The intermediate concentration survey specimens were prepared by admixture of the 2 master pools to achieve the range of values needed to challenge methods at different concentrations. The general chemistry survey specimens included specifications for 58 analytes of which results for 10 are reported here.

For shipment to participants, specimen C-02 (FFS) was included as a regular specimen in the set of C Survey frozen vials. The survey vials were packaged frozen in Styrofoam cartons containing a frozen pack intended to allow thawing but to maintain cool conditions during transit.

Participants were instructed not to refreeze the specimens, to store them at 2 to 8°C, to mix the vials by inversion 4 to 5 times, and to perform assays within 10 days of receipt. Specimen C-02 was not identified as a different preparation from other survey specimens and was handled according to usual practices by participants. Participants performed the assays once and also provided information on the instruments and methods used.

Target values were assigned to survey specimens by the reference measurement procedures (RMPs) listed in Table 1. Specimens were shipped to the reference laboratories on solid carbon dioxide and stored at −70°C until measured.

The survey participant results were screened for errors prior to statistical analysis. First, the histograms of the data were visually inspected and results from participants who completed the reporting form incorrectly were removed. The data were then subjected to a 2-pass, 3-SD test for outliers. Laboratory data greater than 3 SD from their peer group mean on the first and second passes were eliminated. After outlier exclusion, participant results from peer groups with fewer than 10 laboratories were not included in the data evaluation.

Biases between survey results and RMP values were calculated for each participating laboratory. A 1-sample 2-sided t test was applied to test the significance of absolute bias between the target value set by the reference measurement procedure and each peer group mean value. To test the bias differences between C-02 (FFS) and conventional PT materials with similar concentration values, a paired t test was performed within each peer group to compare the percent bias versus the RMP value for C-02 (FFS) to that of conventional PT specimens. The magnitude of percent bias differences between C-02 (FFS) and conventional PT materials was also evaluated against the allowable bias based on biologic variability criteria described by Fraser.11 ,Table 2 shows the allowable bias criteria used. The within individual and among individuals biologic variability were taken from the database developed by the Spanish Society of Clinical Chemistry and Molecular Pathology12 that is updated and published on the Westgard QC Web site.13 Analysis of variance using a linear mixed model was applied to examine the fixed effects of method and instrument on participant bias versus RMP for specimen C-02 (FFS). All data analyses were performed using SAS for Windows version 8.2 software (SAS Inc, Cary, NC).

For evaluation of commutability of conventionally prepared PT samples with the fresh frozen sample, only analyte concentrations relatively near to those of the C-02 (FFS) concentrations were used. This precaution avoided the inappropriate influence of differences in variance that may occur when comparing results at different analyte concentrations. For chloride, C-02 (104.0 mEq/L) was compared with C-01 (104.9 mEq/L); for glucose, C-02 (98.5 mg/dL [5.47 mmol/L]) was compared with C-04 (99.9 mg/dL [5.55 mmol/L]); for iron, C-02 (65.4 μg/dL [11.7 μmol/L]) was compared with C-03 (72.8 μg/dL [13.0 μmol/L]) and C-04 (108.5 μg/dL [19.4 μmol/L]); for magnesium, C-02 (1.59 mEq/L [0.794 mmol/L]) was compared with C-03 (1.26 mEq/L [0.629 mmol/L]) and C-04 (1.77 mEq/L [0.884 mmol/L]); for potassium, C-02 (4.38 mEq/L) was compared with C-01 (4.91 mEq/L) and C-04 (3.49 mEq/L); for sodium, C-02 (140.7 mEq/L) was compared with C-01 (141.3 mEq/L); for urea nitrogen, C-02 (12.2 mg/dL [4.36 mmol urea/L]) was compared with C-03 (14.8 mg/dL [5.28 mmol urea/L]); and for uric acid, C-02 (5.38 mg/dL [320 μmol/L]) was compared with C-03 (4.52 mg/ dL [269 μmol/L]) and C-04 (5.55 mg/dL [330 μmol/L]). Bilirubin and phosphate were not included in the commutability evaluation because the concentrations in C-02 (FFS) were not adequately close to those in other specimens for which both participant and RMP results were available.

Figures 1 through 10 present the bias and distribution of participant peer group values versus the RMP values for specimen C-02 (FFS). The x-axis shows the peer groups as defined by the CAP Survey and the y-axis shows the bias for a peer group as the difference between the peer group mean value and the reference measurement result. Different plot symbols indicate different chemical methods used by the respective peer group as defined by the CAP survey. The error bars represent ±1.96 times the SD for the distribution of participant results in a peer group. Table 3 gives the percent of peer groups with biases versus the RMP that were statistically significant (P < .001) by a Student t test. Although a substantial portion of peer groups had significant biases, the statistical criteria were stringent because many peer groups had a large number of participants that resulted in small standard errors for the mean biases. Some of the statistically significant, but small, biases may not be clinically meaningful. Table 3 also includes evaluation of bias using criteria based on biologic variability that attempts to consider the clinical application of test results to establish optimal, desirable, and minimum performance criteria.11 

For most analytes, the observed biases were associated with instrument manufacturer rather than with the chemical methodology. Table 4 presents an analysis of variance of the effects of method and instrument on participant bias versus RMP for specimen C-02 (FFS). The P values for instrument effects were clearly more significant than for reagent effects for chloride, glucose, iron, and phosphate and were similar for bilirubin, magnesium, and urea. Note that for sodium and potassium the method effects were more significant than instrument effects, but 98% of direct ion selective electrode results were from a single instrument type (Vitros, Rochester, NY) that actually reflected instrument effects. Uric acid could not be evaluated for method effects because all instruments used the same method based on uricase.

Three conventionally prepared survey specimens, C-01, C-03, and C-04, were also assayed by RMPs, which allowed determination of the influence of matrix bias on the overall method bias observed for these materials. Matrix bias was determined for each peer group as the difference between the observed bias for the conventional specimens versus the RMP result and the trueness bias from the C-02 (FFS) specimen for each analyte. The C-02 (FFS) specimen was assumed to have results that were commutable with those for native sera and thus to have no matrix bias. Figure 11 shows, for each analyte, the percent of peer group mean biases that were statistically different from the trueness bias based on specimen C-02 (FFS). Also shown are the percent of peer group mean biases whose difference from the trueness bias based on specimen C-02 (FFS) exceeded the allowable bias for an optimal method based on biologic variability criteria.

Trueness for a measurement is the agreement between replicate measurements of a sample and the numeric value assigned to that sample by an RMP. Trueness is typically expressed as mean systematic bias because the replication reduces random imprecision or variability to a low level. The related term accuracy is the agreement between an individual measurement on a sample and a value assigned by an RMP. Accuracy for an individual patient's sample includes contributions from systematic bias, random variability, and specimen-specific method nonspecificity.

Trueness is an attribute that can be evaluated from a large PT survey only when the PT material is commutable with native patients' samples among all the methods used by participants and the RMP. The C-02 specimen was prepared by a protocol that produced a nonadulterated FFS pool that was expected to be commutable with native patients' samples among the RMPs and the routine methods reported in the PT survey. Other serum pools prepared by the same protocol have been reported to be commutable with native patients' samples for cholesterol, high-density lipoprotein–cholesterol, triglycerides, and creatinine.10,14,15 

The statistical criterion based on a t test for a difference in bias estimated for routine methods versus an RMP for specimen C-02 (FFS) is influenced by the number of observations and can give a misleading conclusion for peer groups with large numbers of participants. Consequently, a second criterion was used to evaluate the bias in the context of its clinical impact. For each analyte, the allowable bias was determined based on biologic variability suggested by Fraser11 and endorsed by a conference on strategies to set global quality specifications in laboratory medicine16 (Table 2). The biologic variability–based criteria relate method performance to the requirements for appropriate clinical use of the results. Although the criteria for optimal, desirable, and minimum performance are somewhat arbitrary, the desirable goals were based on the generally accepted premise that a measurement error should not cause more than approximately 12% error in the ability to categorize an individual result as belonging to a nondiseased population for which a distribution-based reference interval has been established.11 Using the biologic variability approach, biases in excess of the criteria are considered clinically significant and would compromise interpretation of results for diagnostic or monitoring decisions.

Figures 1 through 10 and Table 3 summarize the state of the art for performance of routine methods to measure these 10 analytes. Glucose, iron, potassium, and uric acid methods exhibited the best performance, with all peer groups meeting the minimum, and more than 87.5% of peer groups meeting the desirable, bias goals based on biologic variability. In addition, the limits that included 95% of results for all the peer groups for these 4 analytes were not excessive.

Sodium and chloride methods had the poorest overall performance because even the best performing peer groups had results that were distributed over a 3 to 4 mEq/L range. From Table 2, the minimum bias requirement based on biologic variability for sodium or chloride is 0.7 mEq/L at the concentrations of specimen C-02 (FFS). Current technology is challenged to provide trueness and imprecision adequate for the biologic variability of these analytes. Using arbitrary criteria for mean bias of ±2 mEq/L versus the RMP values, 24 (77%) of 31 peer groups for sodium and 25 (83%) of 30 peer groups for chloride met this limit. However, 95% of individual laboratory results varied from −3 to +8 mEq/L for sodium and −4.5 to +6 mEq/L for chloride.

Magnesium failed to meet statistical or minimum biologic variability–based criteria for more than 50% of peer groups. Bilirubin, phosphate, and urea had several peer groups with fairly large biases that were clearly different from the other peer groups and different from the RMP values.

A possible contributor to systematic bias was nonspecificity of an affected method for one or more components in the fresh frozen pooled serum. The FFS was pooled from 670 donor serum units. Consequently, any unit that may have contributed an interfering substance would have represented 0.15% of the pool, making it unlikely that a substance would remain at a concentration expected to cause an interference. Previous reports have supported that creation of pooled sera from a large number of healthy donors minimizes the nonspecificity impact of any potentially interfering substances.10,17 

In general, the notable biases in Figures 1 through 10 were not associated with a particular methodology but rather with an instrument manufacturer. For many analytes, the plot symbols indicating different reaction chemistries or method designs were quite intermixed even among different instrument platforms from a single instrument manufacturer. Table 4 suggests that for bilirubin, chloride, glucose, iron, magnesium, and phosphate the biases versus an RMP were more associated with the instrument manufacturer than the type of measurement methodology. In addition, the significant method effects for potassium and sodium were an anomaly and actually represented instrument effects because a single manufacturer dominated the numbers of participants reporting a direct ion selective electrode methodology. Uric acid had only instrument effects because all instruments used the same enzymatic methodology. The preceding observations suggest that calibration traceability was more influential than chemical methodology in determining the agreement of results among peer groups and with an RMP value. This conclusion is consistent with an earlier report from Ross et al.18 

Overall, 95% of all participant results for these 10 analytes were generated using instrument systems from 7 manufacturers. Some instrument systems can use reagents and calibrators from third party suppliers but these are a small percent of results reported by participants. Consequently, harmonization of results among measurement systems, and among laboratories, can be effectively achieved by standardization programs that provide manufacturers with practical tools and procedures to make calibration of routine methods metrologically traceable to higher order reference systems.5,6 

Standardized calibration is a critical component of trueness but does not address nonspecificity issues that may be present in some methods. It is possible for a method to be calibrated to compensate for “average” nonspecificity bias from normally occurring substances in clinical specimens, but individual patients with various pathologic conditions could have abnormal amounts of such substances or other metabolites or drugs that cause nonspecific interferences. Thus, attempting to compensate for average nonspecificity does not adequately address method nonspecificity when it is a contributor to variability in results among individuals.

The bias versus an RMP observed for peer groups using conventionally prepared survey materials was frequently statistically different than the bias observed for the C-02 (FFS) specimen. The C-02 (FFS) specimen was pooled native serum and was assumed to be commutable with native patients' samples. As discussed previously, a statistical criterion based on a t test for a difference in bias between the C-02 (FFS) and conventional PT specimens is sensitive to the number of observations in a peer group and may be excessively stringent for large peer groups. A second criterion based on the allowable bias using biologic variability criteria was used to evaluate the equivalence of results for the conventionally prepared PT specimens with the results for the C-02 (FFS) specimen to allow the evaluation to be related to the clinical application of the analyte. The optimal, rather than desirable or minimum, method performance criterion based on biologic variability was used because the optimal bias criterion represents variability in method calibration traceability that will not compromise clinical interpretation of a measurement result. Consequently, if numeric biases from conventional PT specimens were equivalent to those from the C-02 (FFS) specimen within the optimal criterion, the PT material would be considered commutable with native patients' samples within an uncertainty consistent with its use to assess trueness of a method.

Examination of Figure 11 suggests that commutability of conventional PT specimens is not adequate to evaluate the trueness of a routine method. Similarly, the peer group means from conventional PT results cannot be used to harmonize results across testing platforms. In this regard, it should be noted that proficiency testing is typically used to measure a laboratory's proficiency at performing a test and not the trueness of the test method itself or its performance relative to other methods. Proficiency testing does provide a good measure of the precision within a peer group. Results from this study suggest that traditional PT materials are not suitable for field-based postmarketing assessments of a method's trueness.

We appreciate the effort of Sharon Burr, MT(ASCP), MBA, of the CAP who coordinated the preparation of the FFS specimen, its inclusion in the survey, and the reference measurement procedure value assignment. We appreciate the assistance of Masao Umemoto, PhD, to coordinate reference measurements at the HECTEF Standard Reference Center Foundation, Kanagawa, Japan.