Context.—US Food and Drug Administration (FDA)–approved diagnostic tests based on molecular genetic technologies are becoming available for an increasing number of microbial pathogens. Advances in technology and lower costs have moved molecular diagnostic tests formerly performed for research purposes only into much wider use in clinical microbiology laboratories.
Objective.—To provide an example of laboratory studies performed to verify the performance of an FDA-approved assay for the detection of Clostridium difficile cytotoxin B compared with the manufacturer's performance standards.
Design.—We describe the process and protocols used by a laboratory for verification of an FDA-approved assay, assess data from the verification studies, and implement the assay after verification.
Results.—Performance data from the verification studies conducted by the laboratory were consistent with the manufacturer's performance standards and the assay was implemented into the laboratory's test menu.
Conclusion.—Verification studies are required for FDA-approved diagnostic assays prior to use in patient care. Laboratories should develop a standardized approach to verification studies that can be adapted and applied to different types of assays. We describe the verification of an FDA-approved real-time polymerase chain reaction assay for the detection of a toxin gene in a bacterial pathogen.
Molecular tests for detection of pathogens have significantly impacted many areas of medical microbiology. Nucleic acid based tests allow for rapid, specific, and highly sensitive detection of pathogens from a wide range of specimens. Quantitative molecular assays have enabled monitoring of disease progression or treatment response. In addition, DNA sequencing can now be used for pathogen identification, genotyping, and resistance testing.
As with all high-complexity testing, specific requirements and considerations must be addressed by the laboratory prior to implementation of molecular assays. Laboratories must be able to provide personnel trained in high-complexity testing, appropriate strategies for minimizing cross contamination within the laboratory, and continuous quality control and quality assurance protocols appropriate for molecular diagnostic testing.1
Detailed validation studies are required prior to clinical use for laboratory developed (non–Food and Drug Administration [FDA]-approved) molecular diagnostic assays. In contrast, for FDA-approved tests, clinical laboratories are required to verify that the performance characteristics of the test performed within their laboratory are comparable to the performance characteristics determined by the manufacturer. The verification process consists of at least 3 phases: planning, verification, and implementation.2 For an FDA-approved test, the laboratory verifies that the accuracy, precision, reportable range, and reference range are comparable to those of the manufacturer. For qualitative tests, the reportable range is often simply positive or negative; thus, specific verification studies are not required for reportable range. Likewise, analytical sensitivity and analytical specificity are determined by the manufacturer and do not need to be independently verified by the laboratory. Detailed definitions and discussions of individual performance characteristics have been published by the Clinical and Laboratory Standards Institute, American Society for Microbiology, and other organizations.3–5
An increasing number of FDA-approved molecular tests are available for infectious disease diagnosis. The use of automation and the availability of numerous FDA-approved platforms for detection of several different pathogens have resulted in the introduction of molecular diagnostic testing into a growing number of laboratories. This article describes the verification of an FDA-approved molecular diagnostic test for the detection of Clostridium difficile toxin B in fecal specimens, the Becton, Dickinson and Company (BD) GeneOhm Cdiff assay (BD, Franklin Lakes, New Jersey).
FAMILIARIZATION AND PLANNING
It is important for the laboratory director to become familiar with current practice and diagnostic issues addressed by the assay, including the intended use of the test, indications for testing, interpretive guidelines, and test principles and limitations. In addition, practical issues, such as sample and quality control requirements, necessary equipment, physical specifications, and assay complexity need to be considered. Initial evaluation of the test must ensure that it is compatible with available laboratory resources, as well as the patient care requirements of the population served. The laboratory should discuss potential implementation of the test with clinicians and other stakeholders to ensure a smooth transition to use in patient care testing.
For verification of the BD GeneOhm Cdiff assay, a review of the literature for diagnosis of C difficile–associated disease revealed that C difficile is the leading cause of antibiotic-associated diarrhea and a major nosocomial pathogen.6 Pathogenicity is attributed mainly to C difficile toxin B. Early diagnosis is essential for patient management and infection control.
Laboratory diagnosis of C difficile–associated disease has been based either on stool toxin or organism detection assays. Detection of toxin in stool can be performed by enzyme immunoassay, cytotoxicity assay, or molecular detection of the C difficile toxin B gene (tcdB). Organism detection is accomplished by glutamate dehydrogenase antigen detection or isolation by anaerobic bacterial culture, with demonstration of toxin production by C difficile isolates. Limitations of nonmolecular diagnostic assays include lack of specificity for toxin-producing strains (glutamate dehydrogenase antigen detection), suboptimal sensitivity even if repeat testing algorithms are used (enzyme immunoassay), and turnaround times of 72 hours or greater (cytotoxicity assay and toxigenic culture).
Currently, the only single test with both rapid turnaround time and favorable performance characteristics is toxin gene detection by polymerase chain reaction (PCR). One of the FDA-approved molecular tests is the BD GeneOhm Cdiff assay, a rapid diagnostic real-time PCR assay for the qualitative detection of the tcdB gene in liquid or soft stool specimens.7 The assay includes an internal control and is based on molecular beacon chemistry. It is performed on the Cepheid SmartCycler (Cepheid, Sunnyvale, California) and has been found to offer acceptable clinical sensitivity and specificity compared with toxin detection or culture.8,9 The complexity of this test is kept relatively low by use of a crude specimen lysis reaction, rather than requiring extraction of purified nucleic acid. Also, the complete master mix is provided in the kit. The internal control provided in the kit monitors for PCR inhibitors in fecal specimens.
VERIFICATION OF PERFORMANCE CHARACTERISTICS
Prior to initiating verification studies, it is useful to design a verification outline that specifies detailed information concerning experimental design, number of samples to be tested, strategy for resolution of discrepant results, and acceptance criteria. If modifications are made to this outline during the course of verification experiments, these changes should be explained in the final verification summary or the relevant verification components.
Testing performed during the verification process provides important information and experience for implementation of the test. Experience during validation provides an opportunity to optimize workflow, determine test scheduling and turnaround time, identify pitfalls (like steps with increased potential for contamination), and train key personnel. The number and types of controls (eg, kit controls, cultured organism controls, previously tested patient specimens) are determined; proficiency testing and other quality control and quality assurance activities are defined. During the validation, methods for reporting patient results must be established, including report content and format. Instrument interfaces and the compatibility of the laboratory information system with the test and report format need to be established and validated.
Because the BD GeneOhm Cdiff assay is an FDA-approved test, only accuracy (including discrepant analysis and unresolved/internal control failure analysis), precision, and reportable range need to be verified if performed according to the package insert. The reference range can also be addressed as detailed later. Interference and specimen stability data can be used from the manufacturer's validation as long as accuracy is comparable. If the laboratory wishes to extend the specimen stability beyond the manufacturer's guidelines, additional validation is required. Furthermore, because this is a qualitative test, characteristics such as linearity and limit of quantification do not apply. For this verification study, only liquid and soft stool specimens were accepted as had been validated by the manufacturer and approved by the FDA.
Accuracy
Accuracy may be determined by comparing the experimental methods to well-established reference or gold standard methods. When this is not possible (eg, no gold standard exists), then spiking studies may be used in which positive control material (preferably whole organism) is spiked into a suitable matrix. To prevent bias in the interpretation of test results, the personnel performing tests should be blinded to test results obtained by the reference method. The use of generic specimen identifiers is commonly used for blinded studies.
The test procedure provided in the BD GeneOhm Cdiff assay kit10 was adapted to develop a site-specific standard operating procedure (SOP) for the laboratory verification. Established laboratory protocols were used for the reference methods. Liquid or soft stool samples that had been sent to the laboratory for C difficile toxin testing by the reference methods were aliquoted; 1 aliquot was frozen for subsequent testing in the BD GeneOhm Cdiff verification study. Aliquots of stool specimens were thawed at room temperature and tested in batches according to the verification SOP.
The number of specimens used for verification is determined by the laboratory director, using guidance set forth by relevant regulatory agencies (eg, Center for Medicare & Medicaid Services, College of American Pathologists, Clinical and Laboratory Standards Institute). In general, accuracy following discrepant analysis was required to be greater than 90%, although the laboratory director may use discretion in accepting results below 90%.
For this verification, detection of C difficile toxin by the BD GeneOhm assay was compared with 2 nonmolecular tests performed in the laboratory: cytotoxicity assay and toxigenic culture (cycloserine cefoxitin fructose agar with horse blood). A total of 128 stool specimens were tested blindly with the BD GeneOhm Cdiff assay and cytotoxicity assays. In addition, 59 different stool samples were analyzed with the BD GeneOhm Cdiff assay and toxigenic culture. The production of cytotoxin by culture isolates with characteristic morphology was determined with the cytotoxicity assay.
Concordance of results obtained by the test and reference methods was analyzed. Results obtained on specimens tested by BD GeneOhm Cdiff assay and cytotoxicity assay and by BD GeneOhm Cdiff assay and toxigenic culture are shown in Tables 1 and 2. The BD GeneOhm Cdiff assay detected 81.8% and 85.7% of true positives and 92.6% and 95.6% true negatives using the cytotoxicity assay and the toxigenic culture as reference methods, respectively.
Discrepancy Analysis
To resolve discordant results, a second PCR, targeting another distinct and conserved sequence of the C difficile tcdB gene, was performed as previously described.11 Specimens were considered positive if at least 2 different test methods yielded positive results. Using the cytotoxicity assay as the reference standard, 6 of the 128 specimens tested false negative and 7 false positive by the BD GeneOhm Cdiff assay. After resolution, 2 specimens were falsely positive and 3 falsely negative with the BD GeneOhm Cdiff assay (Table 3); 4 specimens were falsely positive and 4 falsely negative with the cytotoxicity assay.
Using toxigenic culture as the gold standard, 2 specimens were falsely positive and 2 specimens were falsely negative. After resolution testing, 1 specimen was falsely positive and 1 specimen was falsely negative with each the BD GeneOhm Cdiff assay (Table 4) and the toxigenic culture assay.
After resolution of discrepant results, the BD GeneOhm Cdiff assay detected 93.9% and 92.9% true positives and 96.8% and 97.8% true negatives using the cytotoxicity assay and the toxigenic culture as reference methods, respectively. These results are in agreement with the performance characteristics determined by the manufacturer and with results previously reported for the use of this assay.8,9,11
Unresolved Rate/Internal Control Failure
Controlling for PCR inhibitors is important for the interpretation of negative test results, especially when testing specimen types with higher rates of inhibition (such as stool, whole blood, and respiratory sources). Specimens are considered unresolved if the internal control and analyte reactions fail to amplify. Final results are reported as unresolved if repeat testing confirms the amplification failure seen on initial testing.
In a multicenter prospective study conducted by the manufacturer in collaboration with 4 clinical sites,10 internal control failure occurred in 4.6% (39 of 852) of fresh specimens initially and in 2% (17 of 852) of fresh specimens after repeat testing. Using frozen stool specimens, the internal control failure rate was 0.4% (1 of 256) initially and remained 0.4% after repeat testing. In this verification study, the internal control failure rate was 2.1% (4 of 188) initially and 0% after repeat testing. This is consistent with the previously reported unresolved rates.8,9
Precision
Precision is equivalent to reproducibility for qualitative and semiquantitative tests and may be verified in a variety of ways. Examples are set forth by various regulatory agencies.12 Reproducibility was assessed in conjunction with specimen stability as described later in “Analytical Specificity” and “Specimen Stability.”
Analytical Sensitivity
Because this is an FDA-approved test, analytical sensitivity was not verified. For the verification summary and assay interpretation, data from the manufacturer's validation study were used. For the BD GeneOhm Cdiff assay, the analytical sensitivity was determined by the manufacturer by comparison with quantitative culture of tcdB gene–positive C difficile strains and by dilution of purified C difficile genomic DNA. The limit of detection was 10 copies of tcdB gene or 4 colony forming units per reaction. Clinical sensitivity data were taken from validation studies performed by the manufacturer.
Analytical Specificity
Results of a validation study by the manufacturer for analytical specificity were cited in the verification summary and used for interpretation of test results. For the BD GeneOhm Cdiff assay, the analytical specificity was determined by the manufacturer by testing for cross-reactivity with tcdB gene–negative C difficile strains and other Clostridium species. Other closely related species, as well as common commensal and pathogenic intestinal tract organisms, were also tested for cross-reactivity in the assay. Tests were negative in all cases.10
Interference
For this FDA-approved test, the validation data provided by the manufacturer in the package insert were used in the verification summary and for interpretation of patient results. The manufacturer tested 26 biological and chemical substances for possible interference with the BD GeneOhm Cdiff assay.10 The only interfering substances were blood and mucus, resulting in unresolved results (internal control failures).
Specimen Stability
Specimen stability had been determined by the manufacturer. Fresh stool samples collected in sterile, leak-proof containers and kept between 2°C and 25°C during transport are acceptable for testing. Specimens can be stored for up to 5 days at 2°C to 8°C or for 48 hours at 15°C to 25°C before testing.
Although 1 freeze-thaw cycle is acceptable according to the package insert, the manufacturer does not specify the stability limit of frozen specimens. Therefore, extended stability of frozen stool specimens was validated as an addendum to the BD GeneOhm Cdiff assay verification study. Ten previously positive samples, with a range of different threshold cycles, and 10 negative stool samples were aliquoted and frozen. One set (day 0) was immediately thawed, extracted, and tested with the BD GeneOhm Cdiff assay. One aliquot from each sample was thawed and extracted on days 3, 7, 14, and 28. Extracted specimens were frozen at −20°C and batch tested at the end of the 28-day period. Representative results for 1 positive sample are shown in Table 5. To test stability during prolonged periods at −20°C and the effect of 1 freeze-thaw cycle, samples that had been tested at the beginning of the validation study were thawed and retested after 81 to 201 days (Table 6).
These studies provided evidence that prolonged storage of stool specimens at −20°C does not significantly affect results obtained with the BD GeneOhm Cdiff assay. Even specimens with low tcdB concentrations (threshold cycle [Ct] or quantification cycle [Cq] >35 on initial testing) were positive on repeat testing, though some variability in Ct was observed. This minor signal variability could be due to sampling differences, normal run-to-run variation, or other factors. The result of the frozen storage validation is that stool specimens are accepted for testing with the BD GeneOhm Cdiff assay when frozen for less than 28 days and 1 freeze-thaw cycle is acceptable.
Reportable Range
The reportable range of the BD GeneOhm Cdiff assay is provided by the manufacturer as either detected, not detected, or unresolved.
Reference Range
Ideally, the reference range of an assay is determined by testing a panel of “normal” donor samples to determine the number of positives in a healthy population. Because the BD GeneOhm Cdiff assay is not FDA-approved for testing formed stool specimens, this type of reference range determination was not possible. Therefore, the laboratory director determined the acceptable reference range for the assay, taking into account the use of the test and the prevalence of the organism in the general population. Although toxin-producing C difficile may be detected in a small portion of asymptomatic individuals, relevant literature demonstrates a high clinical accuracy for C difficile PCR assay results when testing is restricted to patients with signs and symptoms of C difficile–associated disease.13 For this assay, the laboratory director decided that detection of C difficile toxin gene tcdB in liquid or soft stool specimens could be clinically relevant and is considered abnormal. The reference range is, therefore, reported as not detected. The BD GeneOhm Cdiff assay is not used to screen asymptomatic individuals.
Verification Summary
After completion of the study, a written summary of the verification testing was prepared. It contains background information on C difficile–related disease, indications for testing, and a table of the substances tested for interference by the manufacturer. The summary document also contains detailed information on all testing pertaining to the verification. This includes results obtained with the 2 reference methods compared with the test method, date of testing, and run numbers. Specimens are individually listed and identified using a generic verification study number.
The summary contains a comparison of performance characteristics for testing performed in the laboratory with those provided by the manufacturer, as shown in Tables 1 through 4. Sensitivity, specificity, and positive and negative predictive values calculated using standard statistical methods were included. There were no deviations from the outlined experiments to report in the validation summary. The validation components and summary were reviewed and approved by the laboratory director before final implementation of the assay for patient testing.
IMPLEMENTATION OF THE TEST
After verification, the SOP was developed for testing patient specimens based on the manufacturer's package insert and experience gained during the verification study. Standard work practices for molecular testing were incorporated into the SOP. Performance monitors and maintenance schedules were established for equipment. The protocol ensured compliance with the Clinical Laboratory Improvement Amendments of 1988 and other relevant regulations. Relevant components include the following:
Specimen Acceptability
This assay has been validated by the manufacturer for the use only with liquid and soft stool samples, that is, stool samples that assumed the shape of their container. A policy for prescreening clinical specimens, with rejection of formed stool, was implemented. The use of stool specimens that had been frozen for up to 28 days was acceptable, as validated prior to implementation.
Testing Personnel
The BD GeneOhm Cdiff assay is a high-complexity test. Therefore, a medical technologist training level is necessary for personnel performing this testing.14 The content and frequency of competency training and monitoring for laboratory personnel performing the test was determined and the process for documentation established.
Quality Assurance
Testing and documentation of new lots and shipments of test kits were established according to standard laboratory practices. Detailed protocols for equipment maintenance, calibration, inspection, and performance monitoring were provided by the final SOP. To satisfy proficiency testing requirements, sources of proficiency testing material must be selected. The laboratory subscribed to the commercially available proficiency challenges from the College of American Pathologists (www.cap.org. Accessed March 26, 2011).
Controls were selected to provide a realistic assessment of all steps of the assay. For the BD GeneOhm Cdiff assay, a processing control (cultured and aliquoted C difficile American Type Culture Collection 9689; final concentration approximately 107 colony forming units per milliliter) was included in the first run of each new lot or shipment, in addition to the positive and negative controls provided with the kit. Protocols to minimize the risk of contamination include cleaning of all countertop surfaces with 10% bleach followed by 70% ethanol, soaking racks in 10% bleach followed by wiping with 70% ethanol, and wiping of pipettes with 10% bleach and 70% ethanol before and after each run. Swipe tests for detection of exogenous nucleic acid contamination are performed twice per month or whenever contamination is suspected. Sampling is performed from instrument surfaces, countertops, and racks used in the specimen preparation area and frequently touched surfaces, such as handles and keyboards. A specialized cleaning procedure and follow-up testing was developed in case swipe testing indicates laboratory contamination.
Test Interpretation and Reporting
Algorithms for interpretation of test results, including criteria for repeat and unresolved specimens, were developed. The reporting format was determined and includes, in addition to analytical results, details on the test method and limit of detection, as well as any comments needed for specific specimens or test results. Specific notification value policies were established and integrated into the laboratory's system for notification phone calls.
BD GeneOhm Assay reagents for this verification study were provided by the manufacturer (Becton, Dickinson and Company, Franklin Lakes, New Jersey).