Verification of Performance Specifications of a Molecular Test: Cystic Fibrosis Carrier Testing Using the Luminex Liquid Bead Array

Before undertaking assay verification of a qualitative IVD, it is important to review the practice guidelines for testing7,8 as well as general guidelines for performance verification of a qualitative IVD.1–6 In reviewing the literature, one should pay particular attention to the items enumerated in Table 1 that are applicable to the test to be verified. These items cover the preanalytic, analytic, and postanalytic phases of a test as covered in standard checklists by the Clinical Laboratory Improvement Amendments of 1988 and the College of American Pathologists. One should know the test's intended use, such as target mutation detection, allele identification, or others. The indication or indications for use should be clear, whether diagnostic, predictive of treatment efficacy, and/or prognostic. The laboratory must understand the test principle and its limitations, the specific sample requirements, the quality controls used, and the equipment required and its physical specifications. The type of results or readout (technical interpretation) and clinical interpretation must be defined. The analytic validation of the test conducted by the manufacturer should be examined closely for the identification of test performance characteristics that need verification. Along with all these items, a laboratory is required to determine the target population and the specimen type or types to be processed based on the test indication. Once this review is completed, the test protocol, working documents, controls, and verification samples can be prepared and tested.

To illustrate the verification process, we describe an example of a laboratory test verification for a qualitative test for high throughput mutation analysis of the cystic fibrosis transmembrane conductance regulator (CFTR) gene for cystic fibrosis (CF) carrier testing. The assay uses a Luminex liquid bead array (Luminex Molecular Diagnostics, Inc, Toronto, Ontario, Canada) to identify mutations after multiplex polymerase chain reaction amplification and allele-specific primer extension. The assay is US Food and Drug Administration approved for use as an IVD for carrier testing in adults of reproductive age, for confirmatory diagnostic testing in newborns and children, and as an aid in newborn screening using peripheral blood. The assay, performed on human blood specimens, simultaneously screens for the 23 CFTR gene mutations recommended by the American College of Medical Genetics and American College of Obstetricians and Gynecologists in 2004 and for 4 variants (polymorphisms) plus 16 additional mutations. This provides approximate mutation detection rates of 90.5%, 73.8%, 67.5%, 48.9%, and 94.0% in North American Caucasians, Hispanic Americans, African Americans, Asian Americans, and the Ashkenazi Jewish population, respectively.9 The manufacturer claims excellent performance characteristics.10 The reported interassay precision from different days, laboratories, lots of reagent, and equipment (thermocyclers and Luminex) is greater than 99.9%. The reported reproducibility from different sites, reagent lots, and operators is also greater than 99.9%. Accuracy was tested by the manufacturer by comparing genotypes obtained using this assay with the results from the bidirectional DNA sequencing of wild-type and mutant and variant alleles in clinical samples, genomic replicates (purified DNA samples from the Coriell Institute for Medical Research Biorepository, Camden, New Jersey), and synthetic controls. The concordance rate is reported to be 100%. The manufacturer's recommendation for genomic DNA (gDNA) input is 25 ng. It is stated that the assay requires purified gDNA with an A_260/280nm ratio of 1.7 to 2.0 (gDNA from blood-EDTA or citrate using available methodologies provides sufficient purity). There are no further studies on interfering substances by the manufacturer. The shelf life of the kit is 1 year at −25°C to −15°C storage and the kit integrity is reported to be stable with up to 3 freeze-thaw cycles. To exclude insufficient and degraded DNA or samples generating suboptimal results, genotyping calls are made by the data analysis software within the acceptance criteria. Each given allele of a sample must demonstrate a signal intensity higher than background as compared with the no target, polymerase chain reaction–negative, control. This is referred to as mean fluorescence intensity (MFI). The MFI should be at least 10 times the no target MFI for the allele, at least 300 MFI units, and at least 5% of the highest signal obtained for the sample. The genotype is determined based on allelic ratios (ratio of net MFI of an allele to sum of net MFIs for all alleles at a locus); for an allele to be considered positive, the allelic ratio must be at least 0.3, and for a wild type the allelic ratio must be at least 0.85. These thresholds have been empirically determined by the manufacturer.

The standard laboratory protocols, from sample collection, transport, processing, and storage to generation of reports, should be written clearly. The controls to be used and the parameters that need to be verified are dictated by the principle of the assay and its limitations. It is essential that properly trained technologists perform the verification, ideally in a laboratory with an appropriate floor plan to avoid carryover polymerase chain reaction product contamination. The design of the verification is dependent on the intended use of the test in a particular clinical laboratory.

Along with the protocol or standard assay procedure, the worksheets, reagent logs, instrument maintenance logs, and the document control process are typically prepared before creation of the detailed verification plan. If the assay is a molecular genetic test, such as in this example, the test requisition form, the patient information sheet (if the laboratory chooses to provide this), and, if required, consent processes need to be developed. The physician's information and educational materials are also compiled. The requisition form typically includes the clinical status of the patient (affected or unaffected), indication for testing (carrier or diagnostic testing), the pertinent family history, and ethnic background in addition to demographics like sex and age (or date of birth). The patient information sheet, if provided by the laboratory, may contain information including the nature of the disease, the purpose of testing, and the possible interpretations of the test results. In addition, it may convey the recurrence risks and implications for family members and a pregnancy, the turnaround time of the test, the possible outcomes of the test, together with information about the available support groups and materials for additional information. Subsequently, the specific performance characteristics of the assay are assessed to ensure that the verification design will satisfactorily address each pertinent characteristic. It is also necessary to determine and procure the appropriate positive and negative controls to be used, if they are not provided by the manufacturer or distributor. Reference materials can be obtained from a commercial source such as Coriell. Patient samples (n  =  20 in this example) that may be split samples or samples exchanged with other laboratories can be used for assay verification.4 

Given that this assay is a qualitative genotyping test intended, in this example, to be used specifically for carrier testing in adults of reproductive age in a particular clinical laboratory, relevant performance characteristics are accuracy, precision (reproducibility in a qualitative assay), analytic sensitivity (including limit of detection), and analytic specificity (including interfering substances). Other characteristics that are addressed in quantitative tests, such as linearity and limit of quantification as part of analytical sensitivity, are not applicable in this particular test example.2,6 

Each of the relevant terms is briefly defined here so that the plan of verification for this particular assay is clear. Accuracy is the “closeness of the agreement between the result of a measurement and a true value of the measurand.”9 This is measured by the presence or absence of the analyte (such as mutation, variant, or targeted nucleic acid sequence) and can be assessed by testing reference materials or comparing results of a reference method or results of split samples from a clinically valid test method. Precision is the “closeness of agreement between independent test results obtained under stipulated conditions.”11 It encompasses repeatability (agreement between test results for an analyte under the same conditions such as intrarun assay) and reproducibility (agreement between independent test results for the same analyte under different conditions such as interrun, interoperator, or interlaboratory assay). Analytic sensitivity is the ability of an assay to detect a given analyte or the limit of detection (lowest concentration of analyte with an informative result that is distinguishable from background). For molecular genetic testing, analytic sensitivity is considered as “the proportion of biological samples that have a positive test result or known mutation and that are correctly classified as positive.”12 Analytic specificity is the ability of the assay to detect only the target analyte, or, in molecular genetic testing, the ability of the assay to distinguish the target sequence, mutation, or variant from the other sequences or the rest of the genome in the specimen.13 

At this stage the assay work flow, personnel requirements, and turnaround time can be determined. It is an opportunity to test run the laboratory information system to answer the demands of the particular assay before putting the test on line. Scheduling of proper controls to be used for each run can be finalized. The quality assurance measures can also be refined.

For this particular assay, the verification studies were performed using 25 homogeneous, stable, certified gDNA reference positive control samples (23 samples from the Mut CF-2 Panel offered by Coriell composed of the 23 CFTR mutations recommended by the American College of Medical Genetics and American College of Obstetricians and Gynecologists and 2 positive control samples provided by the manufacturer). In addition, 20 split residual patient samples were sent to an outside reference laboratory. The split samples were gDNA from whole blood collected in EDTA tubes of patients who consented to CF carrier testing in the hospital. After the automated extraction of gDNA on acceptable samples, the concentration and quality of the gDNA samples were determined by spectrophotometry. The manufacturer's recommended 25 ng of each of the samples (having A_{260 /280} ratio of > 1.7) was used for the assay runs. Catalog and lot numbers for all reagents used in each of the runs were incorporated with the worksheet for each run. To further demonstrate the accuracy of the kit, the 20 split samples that were sent to an outside reference laboratory were run using the Luminex assay. The concordance of the results from the reference laboratory, which used a comparable and clinically acceptable method for detection of the CFTR mutations, was analyzed.

Following the procedure outlined in the kit, the verification of specified performance characteristics was performed in 3 phases. Worksheets and raw data for each run were kept on record in the order of their corresponding run dates. In the first phase, the Mut CF-2 panel (23 samples), the 2 positive controls from the manufacturer, and the 5 split samples were run on 2 consecutive days by the same operator using the same equipment to measure the accuracy and repeatability (Table 2). To further verify repeatability in the second phase, the most common CFTR mutation in the Mut CF-2 panel (deltaF508), both homozygous and compound heterozygous, was assayed in triplicate within the same run by the same operator in the same equipment using the same conditions that were used for the intrarun assay (Table 3). The same samples were used in the interrun assay by the same operator (Table 4). Parallel testing with the same samples was also done using reagent batches of different lots.

By additional mixing studies one could measure the limit of detection of the assay, but because this assay is not intended to be quantitative, verification of the range of DNA concentrations that would be used for this assay in the laboratory was appropriate. For an IVD assay, determination of analytic sensitivity is not required as part of the verification.

Determination of analytic specificity is not required as part of the verification for an IVD assay.

However, analytic specificity or cross reactivity of the kit may be verified in the clinical laboratory. Positive samples with mutations that are located adjacent to each other, specifically deltaF508/deltaI507 and G551D/R553X, were analyzed by the same operator under identical conditions in an assay performed on the same day along with the positive controls, the no template control, and the negative control. In addition to testing individual samples of 25 ng of DNA each, the samples were also mixed together at 2 different concentrations (25 ng and 12.5 ng of each paired sample in the mixture). Such an experiment was useful because the results pointed to the need to use 25 ng of DNA sample for specific mutations to be identified properly during data analysis. Furthermore, this experiment demonstrated that allelic variations are flagged appropriately in this assay and that the review of allelic ratios confirms the expected allelic copies present in the adjacent mutations. Although this was not a comprehensive test of the effect of different DNA concentrations on the ability of the assay to detect each of the possible mutations, this indirectly verified the range of DNA concentrations for which this assay can be used and at the same time investigated the possibility of cross-reactivity for those mutations that would be most prone to this interference.

This parameter is not applicable to this qualitative assay.

This parameter is not applicable to this qualitative assay.

After gathering the verification data, a written summary of the results of the verification of performance was prepared. Summary tables were prepared on analytic sensitivity, analytic specificity, accuracy, precision, and reproducibility (interrun, intrarun, split sample, and concordance assays) to highlight the positive and negative agreements (Tables 5 and 6). The acceptability of the manufacturer's claims for the intended use of the assay was stated and the laboratory's specific quality control and quality assurance measures were defined. This included information on criteria for considering when to call an assay run acceptable and procedures to address test failures. It also included a description of the appropriate controls for this assay, that is, inclusion of a homozygous and heterozygous deltaF508 control, a no mutation control, and a no template control (blank) in each run, and a scheduled rotation of the other 22 positive controls in the MUT-2 CF panel (Coriell) with regular assay runs such that each positive control is run regularly.

Finally, template laboratory reports were generated before test implementation. Report templates can be used for each of the possible results, for example, no mutation, heterozygous, homozygous, benign polymorphisms, indeterminate, or no test performed. Essential components of each report include the proper identifiers, demographics, name of the referring physician, and all pertinent data from the requisition form that may guide the final interpretation of the results. Residual risk should be reported, if related data are available. Recommendations for genetic counseling and/or definitive diagnostic testing and guidance on next steps in light of the results can follow the section of results interpretation. Information on the method, the components of the mutation panel, detection rates of the test in different ethnicity, and the test limitations are to be covered in a separate section of the report.

While the verification data were being generated, details of the implementation were finalized. Integration of the new test with the workflow and laboratory information system should not be overlooked and were considered early in the process. All equipment to be used was appropriately installed, inspected, and maintained. Procedures for instrument qualification, operation qualification, and performance qualification also need to be in place. The written technical standard operating procedure covered the test indication, intended use, test principle, specimen handling and storage, reagents and controls, equipment, the stepwise assay procedure, results interpretation and report generation, and references. Quality control and quality assurance measures including participation in proficiency testing and archiving of records, reports, and tested specimens were also finalized. The technical standard operating procedure and the verification report were reviewed and signed by the laboratory director. The billing mechanism and budgetary allocations were also finalized before putting the test on line. Depending on the location of the laboratory, application for inspection or approval by the state regulatory body may be necessary. It is only when the proper permit is issued that the laboratory can actually operate and/or offer the test. Finally, it should be emphasized that verification is an ongoing process of continuous improvement and documentation.