We implemented multiple nucleic acid amplification test platforms because of the limited availability of test kits for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) during the early stages of the pandemic. Interpretation of results generated by different platforms and prioritization for testing algorithms required cross-comparison.
To compare the analytical sensitivity of 3 commercial SARS-CoV-2 molecular assays, selected samples were studied in parallel with Cobas SARS-CoV-2 test, NxTAG CoV Extended Panel, and ID NOW COVID-19 assays.
A total of 8043 SARS-CoV-2 tests performed from March 22 to April 19, 2020, were included in this study. For all 1794 positive specimens detected by the cobas SARS-CoV-2 assay, the cycle threshold (Ct) values were manually tracked and plotted to demonstrate the distribution of sample viral levels. Additionally, 50 and 63 low-positive specimens (Ct values >32) as well as 50 and 61 consecutive positive specimens by the cobas assay were tested with NxTAG and ID NOW, respectively, to estimate their relative sensitivities.
The Ct values of cobas SARS-CoV-2–positive samples were evenly distributed throughout ranges of 13.32 to 39.50 (mean, 25.06) and 13.60 to 42.49 (mean, 26.45) for ORF1 and E gene targets, respectively. NxTAG reliably detected only specimens with E gene Ct values lower than 33, and is estimated to detect 89.4% of positive specimens detected by cobas assay. ID NOW had performance variation independent of Ct value and is estimated to detect 83.5% of cobas positives.
Clinical specimens exhibit a wide range of viral burden, with a significant portion at low levels. Analytical sensitivity of testing platforms is critical for reliable detection of SARS-CoV-2 and uniform care to patients.
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an enveloped RNA virus that causes a spectrum of health outcomes ranging from asymptomatic infection to severe respiratory illness that may lead to hospitalization and death, with the disease state referred to as coronavirus disease 2019 (COVID-19). Early diagnosis is extremely valuable for patient self-isolation and management in hospital settings to limit the spread of the virus. SARS-CoV-2 was first reported in Wuhan, the capital city of the central province of Hubei, China, in December 2019; it disseminated rapidly and globally, and was declared a public health emergency of international concern by the World Health Organization1 on January 30, 2020. Its designation was subsequently upgraded to pandemic status by the World Health Organization2 on March 11, 2020, with a global total of approximately 120 000 reported infected individuals and 4300 deaths in a total of 114 countries and a daily incidence of newly reported cases between 4000 and 6000. Despite the magnitude of the SARS-CoV-2 pandemic and the rapid rate of spread, no commercial diagnostic test was available at that time in the United States, where the number of reported cases was relatively limited (696 total).2
On February 29, 2020, the US Food and Drug Administration (FDA) granted Emergency Use Authorization (EUA) status for the diagnostic assays developed by the Centers for Disease Control and Prevention (CDC) and the New York Wadsworth Center, New York State Department of Public Health. Roche Molecular Systems, Inc, received EUA for a reverse transcription polymerase chain reaction (PCR) test, cobas SARS-CoV-2—the first commercially available SARS-CoV-2 diagnostic test—on March 12, 2020. Despite having received EUA, the cobas SARS-CoV-2 test kits and all commercial test kits that followed it were subject to allocation because of greater demand than availability of reagents. Testing was available at select reference laboratories, but often with considerable turnaround times for final results. By the end of March, more than a dozen diagnostic vendors had received EUA from the FDA for SARS-CoV-2 diagnostic assays through a fast-tracked FDA policy. In order to provide SARS-CoV-2 testing to meet the needs of our academic health system, we needed to simultaneously use multiple test platforms, like many other institutions. There was limited opportunity to compare their performance and relative sensitivities beyond the information provided by the vendor. Here we compare the analytical sensitivities of 3 commercial SARS-CoV-2 diagnostic platforms that have been used in our laboratory during the surge of the SARS-CoV-2 pandemic in the United States.
MATERIALS AND METHODS
Cobas SARS-CoV-2 Test
The cobas SARS-CoV-2 Test (Roche Molecular Systems, Inc, Pleasanton, California) was performed using a cobas 6800 analyzer (Roche), which is fully automated for nucleic acid extraction, PCR setup, amplification, and detection. This test uses 400 μL of viral transport medium from nasopharyngeal and oropharyngeal swab specimens and 2 targets, the ORF1/a gene (a nonstructural region specific to SARS-CoV-2) and the envelope (E) gene (a pan-Sarbecovirus target) for virus detection. All tests were carried out with strict adherence to the manufacturer's package insert.3 For the purpose of this study, the ORF1/a gene is designated as cobas target 1 and the E gene cobas target 2. Additionally, specimens with only target 2 cycle threshold (Ct) values, reported clinically as presumptive positive, were considered positive specimens for the purpose of this study. All positive specimens were de-identified and assigned a sample number based on when they were resulted. Cobas target 1 and target 2 Ct values of these positive samples were plotted subsequently.
NxTAG CoV Extended Panel
The NxTAG CoV Extended Panel (Luminex Molecular Diagnostics, Inc, Toronto, Ontario, Canada) was performed using the MagPix system (Luminex), which uses color-coded magnetic beads to detect the presence of designated molecular targets by analysis of images of a magnetized chamber captured with an internal charge-coupled device camera. In this assay, nucleic acid is extracted externally using the EasyMag extraction system (bioMerieux, Inc, Durham, North Carolina). Prior to extraction, 200 μL volume from each sample was spiked with 10 μL of bacteriophage MS2, included in the kit as an internal control to assess sample-related issues such as failed extraction or PCR inhibition. A total of 110 μL of eluted nucleic acid was obtained and 35 μL was added to lyophilized beads containing all necessary PCR reagents and tagged beads with complementary sequence for each target gene. Three targets were used for SARS-CoV-2 detection: ORF1ab, nucleocapsid (N) gene, and E gene. Reverse transcription PCR products were then analyzed on the MagPix instrument using Luminex xPonent software (Luminex), and the output was subsequently analyzed by SYNCT software (Luminex) for final results. Briefly, final results were determined based on the normalized fluorescent values, multidimension detection, from MagPix instrument readout, and thresholds of 35 (ORF1ab), 95 (N), and 25 (E) were used for the indicated targets. When any of the targets had multidimension detection value that passed the threshold, the specimen was classified as positive.4 Between March 30 and April 5, NxTAG was used to test specimens from the same specimen pool as the cobas assay. From April 6 on, NxTAG was used mainly for retest specimens that had invalid results from the cobas assay or for specimens that did not have sufficient volume for the cobas assay.
Of the 50 low-positive specimens that were analyzed in parallel with NxTAG CoV Extended Panel, 41 were taken directly from the original specimens, and 9 were diluted specimens in either universal transport medium or cobas PCR Media as described in previous dilution experiments. These specimens were never frozen and were all tested on the other 2 platforms within 24 hours of the initial cobas SARS-CoV-2 testing.
ID NOW COVID-19 Test
The ID NOW COVID-19 (Abbott Molecular Inc, Des Plaines, Illinois) test, performed on the ID NOW instrument (Abbott), is a point-of-care assay that uses isothermal nucleic acid amplification for qualitative detection of SARS-CoV-2 viral RNA. It detects a unique region of the RdRp gene segment, and includes an internal control. Fluorescently labeled molecular beacons are used to specifically identify each of the amplified RNA targets.
For all our tests performed before April 20, 2020, 200 μL volume of viral transport was used according to the manufacturer's instructions.5 Starting on April 20, 2020, because of an update in the manufacturer's instructions and FDA EUA, patient testing was performed by directly dipping the dry swabs from patients into sample receivers.
All 117 specimens retested by ID NOW were nasopharyngeal or oropharyngeal swabs stored in universal transport medium, and 7 were samples diluted in universal transport medium from positive specimens (the 4 specimens diluted in cobas PCR Media with invalid results were not included in this portion of the analysis; see Table 1 for details). Of these samples, 200 μL was analyzed by ID NOW COVID-19 on the corresponding instrument. These specimens were never frozen and were all tested on the other 2 platforms within 24 hours of initial cobas SARS-CoV-2 testing.
Sample Collection and Processing
All samples included in this study were specimens sent to the Thomas Jefferson University Hospital Clinical Laboratory (Philadelphia, Pennsylvania) for SARS-CoV-2 testing. To obtain the number of all tests performed between March 22, 2020, and April 19, 2020, Epic medical record and Epic Beaker laboratory information system software (Epic Systems, Verona, Wisconsin) were used to extract all SARS-CoV-2 tests resulted during that period of time. This list was then further filtered based on methodology, and the numbers of positive and negative samples for all 4 methods were counted.
Data Analysis and Graphical Software
To obtain Ct values for cobas targets 1 and 2, the original batch reports from the cobas 6800 instrument were used. We manually recorded Ct values for all 1794 positive specimens called by the cobas SARS-CoV-2 Test. All downstream analyses and data visualization were performed in R Studio with dplyr and ggplot2 packages (R Studio, Boston, Massachusetts). All dot plots were made with additional parameter giving a position jitter value of 0.2 to avoid overplotting.
From the initiation of SARS-CoV-2 testing on March 22 to April 19, when the specimen for the ID NOW COVID-19 test was switched to a dry nasopharyngeal swab, 8043 specimens were tested at the Thomas Jefferson University Hospital by 3 commercial platforms: the cobas SARS-CoV-2 test (March 25–present), the NxTAG CoV Extended Panel (March 30–present), and the ID NOW COVID-19 test (April 6–present). The distribution of specimens tested per calendar day is shown in Figure 1, A, with the total volume tested by each platform indicated in the figure legend.
The average positivity rates were calculated. The cobas SARS-CoV-2 test was 30.6% (1794 of 5867) positive, NxTAG CoV Extended Panel 25.7% (356 of 1384), and ID NOW COVID-19 15.3% (121 of 792). An in-depth visualization of positivity rate detected by these assays was plotted against calendar day for direct comparison (Figure 1, B). Interestingly, because specimens were randomly assigned to either NxTAG CoV Extended Panel or cobas SARS-CoV-2 test between March 31 and April 5, the consistent lower positive rate reported by NxTAG CoV Extended Panel raised the question of lower assay sensitivity (Figure 1, B). Note that a significant proportion of specimens tested by ID NOW were preadmission screening specimens for surgical patients that required quick turnaround time, and therefore no conclusions about assay sensitivity of ID NOW were drawn from this particular analysis.
Overview of Roche cobas SARS-CoV-2 Test Performance Characteristics
To understand the distribution of viral content in patient specimens, Ct values of cobas assay targets 1 and 2 for all specimens tested positive between March 25 and April 19 were recorded. As shown in Figure 2, A, there is a broad range of Ct values for cobas targets 1 and 2, consistent with a wide distribution of viral content. Because the samples are displayed sequentially based on the reporting date and time on the x-axis, it is evident that there was no significant change in the variability of viral burdens in this period. As shown in Figure 2, A, samples with low viral content frequently had high Ct values for cobas target 2 and undetectable values for cobas target 1.
Figure 2, B, indicates that cobas target 2 was more sensitive, as evident from the significantly lower number of nondetected Ct values (6 versus 86 in comparison with target 1) and a wider range of detected Ct values than target 1 (14–40 versus 14–35). In addition, although Ct values of cobas targets 1 and 2 were concordant when the viral nucleic acid content was high (Figure 2, C), the mean of target 2 Ct values was higher than that of target 1 (26.45 versus 25.07). Specifically, it is evident from Figure 2, B, that values higher than 35 were rarely detected for cobas target 1, as shown by the significantly smaller number of specimens past the black line. For this reason, target 2 Ct values were chosen for estimation of viral burden in subsequent analyses.
All positive specimens analyzed by the cobas assay were further separated into 6 bins based on the Ct value for target 2, as shown in the first column of Table 2. The percentages of specimens in each bin are shown in column 3.
Dilution Experiments to Test Assay Sensitivity of 3 Platforms
To gain further insights into the performance of these 3 platforms, experiments were designed to specifically evaluate specimens near the limit of detection for each test.
The cobas SARS-CoV-2–positive result has Ct values for both targets, which presents the opportunity to evaluate the relative viral content of specimens. Because the overall Ct values ranged from ∼15 to ∼40 in the cobas SARS-CoV-2 test, specimens with Ct values higher than 30 were considered to have relatively low viral levels. As shown in Table 1, parallel analysis of serial dilutions of 4 clinical specimens with cobas SARS-CoV-2 Ct values greater than 30 using the 3 assays demonstrated the superior sensitivity of the cobas test. Because the dilution experiment revealed that cobas SARS-CoV-2 was the most sensitive among the assay platforms tested, additional direct comparisons of these tests were performed.
NxTAG CoV Extended Panel Versus cobas SARS-CoV-2–Positive Samples
To directly compare the sensitivity of the NxTAG CoV and cobas SARS-CoV-2 assay platforms, 2 test panels were created, one composed of 50 consecutive specimens positive by the cobas test and a second composed of 50 low-positive specimens based on the cobas target 2 Ct value. Forty-five of the 50 consecutive cobas positive specimens and 23 of the 50 cobas low-positive specimens were positive by the NxTAG CoV test. The Ct values for positive and negative specimens called by NxTAG CoV Extended Panel were plotted in 3 box plots to understand the comparative sensitivity of this assay to cobas SARS-CoV-2 (Figure 3). For both 50 low-positive (Figure 3, A) and 50 consecutive positive (Figure 3, B) specimens, those called negative by NxTAG CoV test generally had lower Ct values for both targets. Likewise, after we plotted all 100 specimens in the same graph (Figure 3, C), no clear Ct value cutoff could be established to predict whether a specimen would be detected by NxTAG CoV Extended Panel.
As an attempt to extrapolate what percentage of cobas SARS2-CoV–positive specimens could be detected by NxTAG, these 100 specimens were categorized into 6 bins based on their original cobas target 2 Ct values, as shown in column 4 of Table 2. All specimens in the cobas target 2 Ct less than 33 bin were positive with NxTAG CoV test. With increasing Ct value for target 2 as a surrogate for viral content, the ability of the NxTAG CoV test to detect low levels of virus decreased incrementally in serial bins (Table 2, columns 5–7). By multiplying the percentage of detectable positive by NxTAG CoV and the percentage of total specimens falling into each bin (columns 2 and 3 in Table 2), we calculated that NxTAG CoV could potentially detect 89.39% (100% × 80.6% + 80% × 4.75% + 64% × 4.3% + 36% × 3.58% + 22% × 2.74% + 8% × 4.3%, as shown in column 8 of Table 2) of all cobas positive specimens (Table 2, columns 3 and 7).
ID NOW COVID-19 Test Versus cobas SARS-CoV-2–Positive Samples
To directly compare the sensitivity of the ID NOW COVID-19 and cobas SARS-CoV-2 tests, a parallel set of experiments was performed with 61 consecutive positive specimens and 63 low-positive specimens by the cobas assay. After we separated both 63 low-positive and 61 consecutive positive specimens based on their results by ID NOW, Ct values were plotted in Figure 4, A and B, respectively. Among the 124 specimens tested, 45 were negative by the ID NOW COVID-19 test. These samples had a broad range of cobas target 2 Ct values (Figure 4, C). Assigning these specimens to the previously described 6 Ct bins revealed a broad distribution of samples in each category shown in columns 9 through 12 in Table 2, unlike the stepwise association of Ct value and frequency that was observed with the NxTAG CoV test. By multiplying the percentage of detectable positive by ID NOW and the percentage of total specimens falling into each bin as we did for the NxTAG CoV test, we estimated that ID NOW could potentially detect 83.55% of all positives detectable by cobas SARS-CoV-2 (column 13, Table 2).
ID NOW COVID-19 Dry Swab Versus cobas SARS-CoV-2 Test
Recently, Abbott issued an amendment in the ID NOW COVID-19 EUA protocol stating that the previously approved nasopharyngeal/oropharyngeal swab stored in universal transport medium was no longer acceptable, and that only dry swabs could be used as of April 20, 2020. To test the concordance of results with the ID NOW COVID-19 using dry swabs with the cobas SARS-CoV-2 assay, 52 paired patient specimens collected from April 23 to April 26 were tested in parallel. The cobas SARS-CoV-2 test detected 6 specimens containing COVID-19 nucleic acid, but only 4 were positive in the ID NOW COVID-19 assay. The remaining 46 specimens were concordantly negative by both tests. Although the limited number of specimens did not support statistical analysis of Ct value distribution, the results clearly indicate that the issue with false-negative results may persist even with the updated ID NOW COVID-19 specimen requirement. Consistent with our findings, the FDA6 recently issued a warning about the unstable performance of ID NOW COVID-19 testing, further illustrated the severity of its sensitivity issue.
With the full understanding that clinical sensitivity is different from analytical sensitivity, our study focused mainly on understanding analytical sensitivity of different assays that were performed at our institution. Here we demonstrate that although there are significant differences in the sensitivity of front-line testing systems, it may be possible to understand the hierarchy of sensitivities and deploy them strategically to achieve optimal clinical utility.
Interestingly, as briefly mentioned in the results section, the NxTAG Extended Panel has a relatively high threshold for positive result calling that may negatively impact assay performance. The threshold of multidimension detection values for its 3 targets, ORF1/ab, N gene, and E gene, were set at 35, 95, and 25, respectively. Although it requires only one target above threshold to be called positive, none of the target has a background higher than 5 among 35 negative controls, pooled from negative clinical specimens, which we ran during this period of time. These 3 targets have signals ranging from 0 to 5, −3 to 4, and −2 to 3.5, respectively, with means of 1.226, −0.7903, and −0.5645.
Using the raw data of the 101 positives that we tested, if we decrease the multidimension detection threshold for all 3 targets to 10, we can detect 100%, 100%, 91%, 100%, 22%, and 15% of specimens in each cobas target 2 Ct category (<33, 33–34, 34–35, 35–36, 36–37, and >37). Overall, this allows us to detect 94.2% of the specimens that are called positive by cobas SARS-CoV-2. Compared with its current performance, which detects 89.1% of total positives called by cobas SARS-CoV-2, this will increase the sensitivity by 5%. To our knowledge, data for relative analytical sensitivity for the NxTAG assay have not been published previously.
ID NOW COVID-19, however, has performance variation that is less well correlated with cobas target 2 Ct values. It detected some specimens with cobas target 2 Ct values up to 37 but also missed some specimens with high viral burdens (Ct values in the 20s). Note that the results that we obtained were all from clinical settings and the variation on performance could be dependent on the individual operator or sample-specific characteristics. Nevertheless, an understanding of the estimated false-negative rate provides more insight when deciding which assays to be used on which type of patient.
Our estimation of the sensitivity of the ID NOW COVID-19 assay is in general agreement with several recent publications comparing this test with other SARS-CoV-2–targeting molecular assays. Compared with other sample-to-answer–type testing platforms, Xpert Xpress SARS-CoV-2 (Cepheid, Sunnyvale, California) and ePlex SARS-CoV-2 Test (GenMark Diagnostics, Inc, Carlsbad, California), and using nasopharyngeal swabs in viral transport medium, it was estimated to provide 88% sensitivity of the other 2 assays.7 Another 5 similar studies were also conducted to compare SARS CoV-2 detection assays including Simplexa COVID-19 Direct Kit (Diasorin Molecular LLC, Cypress, California), RealTime SARS-CoV-2 (Abbott), Panther Fusion SARS-CoV-2 (Hologic, Inc, Marlborough, Massachusetts), and RealStar SARS-CoV-2 RT-PCR Kit (Altona Diagnostics GmbH, Hamburg, Germany).8–12 With Xpert Xpress, ePlex, and RealStar showing the highest agreement with the CDC FDA EUA method,7,11,12 cobas SARS-CoV-2, Panther Fusion, and Simplexa had a positive agreement of ∼96% when compared with the CDC assay.8,10,11 ID NOW, on the other hand, gave the worst concordance compared with both CDC FDA EUA and RealTime SARS-CoV-2, offered by the same vendor.7–9 Detailed conclusions and comparisons are shown in Table 3.
One unique aspect of this study is using close to 1800 SARS-CoV-2–positive clinical specimens to demonstrate that these patients exhibit a wide range of viral burden with a significant portion at low levels. In addition, the analytical approach applied in this study provides a more quantitative insight into the analysis of specimens with low levels of positivity and why the analytical sensitivity of testing platforms is particularly critical for reliable detection of SARS-CoV-2.
Given the high transmissibility of the SARS-CoV-2 virus and the important public health ramifications of false-negative results,13,14 it is important to understand diagnostic test performance to inform decisions regarding testing platform acquisition and implementation decisions, as well as prioritization of specific patient populations to different testing options. We describe here clear differences in performance for 3 commercially available SARS-CoV-2/COVID-19 molecular diagnostic tests. Although sensitivity limitations on the order described herein and in other published studies may not preclude clinical utility of tests with lower analytical sensitivity, careful consideration and test evaluation are recommended for SARS-CoV-2/COVID-19 testing algorithm development and implementation and performance monitoring.
The authors have no relevant financial interest in the products or companies described in this article.