Context.—

The use of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) serologic tests detects antibodies in the host, contributing to the identification of individuals who have been exposed to coronavirus disease 2019 (COVID-19).

Objective.—

To critically evaluate 2 commercially available SARS-CoV-2 serology tests.

Design.—

A total of 333 unique, nonduplicated serum samples obtained from COVID-19 patients (n = 170) and negative controls (n = 163) obtained before December 2019 were used in the study. Samples were tested on the Roche E411 and Abbott Architect i4000SR platforms, and results were correlated to reverse transcription polymerase chain reaction (PCR) results and clinical symptoms.

Results.—

There was a strong level of agreement in the qualitative results between both assays, with a Cohen κ value of .840, P < .001. The specificity for both Roche and Abbott were excellent at 100%. Roche exhibited marginally better performance in the 21 days or more group with a sensitivity of 90.6% (95% CI, 75.8%–96.8%) versus an Abbott sensitivity of 84.4% (95% CI, 68.3%–93.1%), as well as in the 14- to 20-day group with a sensitivity of 85.7% (95% CI, 65.4%–95.0%) versus an Abbott sensitivity of 81.0% (95% CI, 60.0%–92.3%). Less than 14 days of symptoms groups exhibited poor sensitivity at less than 50% for both assays. The areas under curve (± standard error) for Roche (0.894 ± 0.025, P < .001) and Abbott (0.884 ± 0.026, P < .001) were very similar. Potential confounders for negative serologic results include antiretroviral medication use and pauci-symptomatic patients.

Conclusions.—

Specificities for high-throughput Roche and Abbott immunoassays are excellent, but users need to be cautious to interpret serologic test results after 14 days of symptoms to avoid false negatives.

Coronavirus disease 2019 (COVID-19) has now been declared a public health pandemic by the World Health Organization (WHO) after its initial outbreak in Wuhan, China.1  COVID-19 patients may present with nonspecific symptoms ranging from mild respiratory tract illness to severe pneumonia requiring intensive care support, hence posing diagnostic difficulties to the clinician.2  The gold standard laboratory diagnosis of COVID-19 relies on the detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) nucleic acid by real-time reverse transcription polymerase chain reaction (RT-PCR) based on gene targets such as N, E, RdRp, orf1a, and orf1b genes.3  Polymerase chain reaction–based molecular testing turnaround time varies from a few hours to days, taking into consideration the expertise of the laboratory in question, as well as the workload, type, and throughput of the analyzers in operation. Coupled with limited RT-PCR reagent/kit availability, there have been some initiatives to use SARS-CoV-2 serology as a screening test or for surveillance within a population. Serologic tests detect antibodies in the host as immunoglobulin (Ig) G, IgM, and/or IgA to SARS-CoV-2, contributing to the identification of individuals who have been exposed to COVID-19, possibly obtaining immunity and further assisting in containment or isolation strategies or even the idea of “immune passports.” Zhao and colleagues4  have recently shown that the median seroconversion times for IgM and IgG were 11 days and 12 days, respectively, and that specific IgG antibodies to SARS-CoV-2 remain detectable in COVID-19 patients during the symptomatic phase of the disease even after RNA becomes undetectable.4,5  A plethora of serology kits have now entered the market, often with claims of good performance, many also having obtained provisional US Food and Drug Administration approval or Conformitè Europëenne marking. In this study, we sought to compare the diagnostic accuracy of two commercially available, automation-track–compatible SARS-CoV-2 serology assays, namely the Roche Elecsys and Abbott Architect Anti-SARS-CoV-2 assays. Chew et al6  and Tang et al7  had previously shown that Abbott performed reasonably well, with a duration of 14 days being suggested as a minimum period to test symptomatic COVID-19 patients for the presence of antibodies. Because the COVID-19 situation is rapidly evolving, there is a need to inform laboratory users of potential differences in the interpretation of SARS-CoV-2 serology assays.

MATERIALS AND METHODS

A total of 333 unique, nonduplicated serum samples obtained from COVID-19–confirmed patients (n = 170) and negative controls (n = 163) obtained before December 2019, before the COVID-19 pandemic, were used in the study. We prospectively selected samples between March 30, 2020, to May 15, 2020, from COVID-19 patients in our institution on the basis of at least 1 positive RT-PCR respiratory sample being positive on our cobas 6800 SARS-CoV-2 assay (Roche Diagnostics, Rotkruez, Switzerland), with the cycle threshold value being lower than cutoff. Samples were collected in serum separator tubes (Beckton Dickinson, Franklin Lakes, New Jersey), centrifuged at 1811g for 8 minutes, and after clinical testing residual sera were collected in accordance with previously described laboratory protocols for COVID-19 sample handling.8  These serum samples were then concurrently analyzed on the cobas e411 (Roche Diagnostics) and Abbott Architect i4000SR (Abbott Diagnostics, Chicago, Illinois) analyzer using their respective assays. Days of symptoms were recorded based on first day of onset of COVID-19 symptoms as recorded by managing clinicians to the time of blood collection. Patients who were asymptomatic at the time of PCR testing were excluded. A repository of archived negative controls were used, with samples taken from patients prior to December 2019. These include patients with and without other positive serologic tests that exhibited the following: anti–extractable nuclear antigen antibodies (9); anti-glomerular basement membrane antibodies (4); anti–smooth muscle antibody (3); hepatitis A IgM (3); Epstein-Barr virus IgM (3); anti-intrinsic factor (5); cytomegalovirus IgM (4); cytomegalovirus IgG (3); syphilis Treponoma pallidum antibody (5); hepatitis B E antigen (2); Epstein-Barr virus IgA (7); Leptospira IgM (3); hepatitis C (9); hepatitis B surface antigen (7); anti–double strand DNA (3); rubella IgM (4); ANA (3); hepatitis A IgG (3); dengue IgG (1); varicella zoster IgM (1); human immunodeficiency virus (8); and varicella zoster virus IgG (6). In brief, both are chemiluminescent immunoassays detecting antibodies to the nucleocapsid protein of SARS-CoV-2, producing a qualitative result (reactive versus nonreactive) but with a quantitative signal cutoff index value. The key difference between both assays is that Abbott detects only IgG antibodies, whereas Roche detects total antibodies (both IgM and IgG) to SARS-CoV-2. Prior to analyses on patients' sera, calibration was performed and quality controls were passed as per manufacturers' instructions. For the Abbott assay, a signal cutoff index (S/C) ratio of 1.4 or greater was interpreted as reactive and an S/C ratio of less than 1.4 was interpreted as nonreactive; for the Roche assay, a signal cutoff index (COI) of 1.0 or greater was interpreted as reactive and a COI of less than 1.0 was interpreted as nonreactive, in accordance with the manufacturers' product insert.

Statistical analyses were performed using the Statistical Package for Social Sciences (SPSS) 25.0, with statistical significance set at P < .05. The relationship between the 2 immunoassays and their qualitative agreement were assessed using Pearson correlation and Cohen κ, respectively. A 1-way analysis of variance (ANOVA) was conducted to compare the mean levels of SARS-CoV-2 antibody signals, with Tukey honest significant difference (HSD) corrections for post hoc comparisons. A χ2 test was performed to compare the sensitivities of both assays on a reactive signal, and an area under the curve (AUC) was presented to compare the diagnostic capabilities of both assays. Our study was approved by the National Healthcare Group Domain Specific Review Board (Singapore, NHG ROAM reference Nos.: 2020/00337 and 2020/00407).

RESULTS

Correlation Between Roche and Abbott

We compared the quantitative signal output of Roche and Abbott, where Figure 1 shows a moderate Pearson correlation coefficient of .734 (P < .001) between both assays. Signal intensities were in general greater in the Roche assay, which is not unexpected because it is a total antibody assay as compared with the Abbott assay. Figure 2 depicts the Bland-Altman plot showing the positive bias in the Roche assay. A total of 69 of 170 confirmed COVID-19 cases (40.6%) were reactive for the Roche assay, whereas 66 of 170 (38.8%) were positive in the Abbott assay. There was a strong level of agreement in the qualitative results between both assays, with a Cohen κ value of .840, P < .001 (Table 1).9 

Figure 1

Linear regression and correlation between Roche and Abbott assays showing that there was a moderate correlation between both assays, with Roche having generally higher quantitative signal outputs.

Figure 1

Linear regression and correlation between Roche and Abbott assays showing that there was a moderate correlation between both assays, with Roche having generally higher quantitative signal outputs.

Figure 2

Bland-Altman plot, showing that the Roche assay has a proportional positive bias compared with Abbott assays when signal values increase.

Figure 2

Bland-Altman plot, showing that the Roche assay has a proportional positive bias compared with Abbott assays when signal values increase.

Table 1

Qualitative Comparison Between Roche and Abbotta

Qualitative Comparison Between Roche and Abbotta
Qualitative Comparison Between Roche and Abbotta

Signal/Cutoff Values

Next, we proceeded to look at the quantitative signal cutoff values generated for each assay which showed that peak signals in both assays were generated between 10 and 15 days and troughed by 45 days, although signals remained detectable. The Roche assay also showed greater signal values generated across all levels as compared with the Abbott assay, demonstrated in the scatterplots in Figure 3. We stratified the positive COVID-19 patients into 4 groups: 21 or more days of symptoms (n = 32), 14 to 20 days of symptoms (n = 21), 7 to 13 days of symptoms (n = 37), and less than 7 days of symptoms (n = 80). The mean signal cutoffs are presented in Tables 2 and 3, and their respective histograms are given in Figure 4. A 1-way ANOVA showed a significant difference across the different days of symptoms for the Roche (F(3,166) = 55.5, P < .001) and the Abbott (F(3,166) = 38.9, P < .001) assays. We had previously shown that post hoc comparisons using the Tukey HSD test indicated that there was a significant difference (P < .001) in mean S/CO IgG levels in the Abbott assay for patients with 7 to 13 days versus 14 to 20 days of symptoms, suggesting that IgG antibodies peaked at the latter period.6  The 14 to 20 days group also produced the greatest mean signal intensities for Abbott IgG titers. Interestingly, in the Roche data we saw that the “total antibody” assay detects the greatest mean signal intensities in the 21 or more day group (mean, 44.18 COI), which was significantly greater than in the 14- to 20-day group (mean, 13.68 COI) from the post hoc Tukey HSD test (P < .001), indicating that total antibody production in the COVID-19 patients continued well past the 14-day period.

Figure 3

A, Scatterplot of Roche total antibody signal values plotted against day of symptom presentation. B, Scatterplot of Abbott IgG antibody signal values plotted against day of symptom presentation.

Figure 3

A, Scatterplot of Roche total antibody signal values plotted against day of symptom presentation. B, Scatterplot of Abbott IgG antibody signal values plotted against day of symptom presentation.

Table 2

Average Signal Values Generated by Roche Assay

Average Signal Values Generated by Roche Assay
Average Signal Values Generated by Roche Assay
Table 3

Average Signal Values Generated by Abbott Assay

Average Signal Values Generated by Abbott Assay
Average Signal Values Generated by Abbott Assay
Figure 4

A, Histogram of mean signal values for Roche total antibody, showing that peak signal values are recorded in the >21 days symptoms group. B, Histogram of mean signal values for Abbott IgG, showing that peak signal values are recorded in the 14- to 20-days symptoms group.

Figure 4

A, Histogram of mean signal values for Roche total antibody, showing that peak signal values are recorded in the >21 days symptoms group. B, Histogram of mean signal values for Abbott IgG, showing that peak signal values are recorded in the 14- to 20-days symptoms group.

Diagnostic Accuracy

The specificity for both Roche and Abbott were excellent at 100% and demonstrated no cross-reactivity to any of the seropositive viruses or autoimmune disorders. Table 4 shows the overall sensitivity was 40.6% (95% CI, 33.5%–48.1%) in Roche and 38.8% (95% CI, 31.8%–46.3%) in Abbott. Roche exhibited marginally better performance in the 21 or more days group, with a sensitivity of 90.6% (95% CI, 75.8%–96.8%) versus Abbott's sensitivity of 84.4% (95% CI, 68.3%–93.1%), as well as the 14- to 20-day group, with a sensitivity of 85.7% (95% CI, 65.4%–95.0%) versus Abbott's sensitivity of 81.0% (95% CI, 60.0%–92.3%). The less than 14 days of symptoms groups exhibited poor sensitivity at less than 50% for both assays. There were no significant differences in overall sensitivity (P = .74), nor were there any between individual subgroups. The AUC (± standard error) for Roche (0.894 ± 0.025, P < .001) and Abbott (0.884 ± 0.026, P < .001) were very similar (Figure 5). This implies that diagnostic sensitivities are comparable between both assays for assessing seroconversion status of COVID-19 patients after 14 days.

Table 4

Sensitivity Between Roche and Abbott Assays

Sensitivity Between Roche and Abbott Assays
Sensitivity Between Roche and Abbott Assays
Figure 5

Receiver-operator characteristic curve for Roche and Abbott, showing similar areas under the curve for both assays.

Figure 5

Receiver-operator characteristic curve for Roche and Abbott, showing similar areas under the curve for both assays.

Negative Serology Patients Who Were Confirmed COVID-19 Positive

We reviewed the patients in the 14- to 20-day and 21 or more days groups who were negative on serology but were confirmed COVID-19 cases on PCR. In the 14- to 20-day group, 3 patients were negative on both Roche and Abbott. Two of them had only mild acute respiratory illness without any evidence of immunocompromise or respiratory distress. However, 1 patient had to be intubated, had lymphopenia, and lopinavir-ritonavir (Kaletra) was administered but otherwise was not on any other immunosuppressive regimens, nor were they given transfusions. Interestingly, 1 patient in this group was nonreactive (S/CO, 0.62) on the Abbott but reactive (15.71 COI) on the Roche. This patient had end-stage renal failure secondary to IgA nephropathy, had a previous renal transplant, and was on mycophenolate (Myfortic) immunosuppressant but otherwise did not have any intensive care unit stay. In the 21 or more days group, 3 patients were negative in both the Abbott and the Roche assay. Two patients were negative for the Abbott assay (SCO values 1.11 and 1.28), but they were positive on the Roche assay (COI values 3.55 and 1.4, respectively). All 5 of these patients in this group were followed up in the infectious disease clinic, and all only had mild acute respiratory illness symptoms in the beginning, without immunocompromise and did not require intubation.

DISCUSSION

Current data suggest that there is limited utility for serology assays in the diagnosis of acute clinical SARS-Cov-2 infections.7  Serology testing may be able to detect patients who have been infected, particularly when testing is performed at least 14 days after onset of infection. However, this may not be able to differentiate current versus previous infection. COVID-19 serologic assays may have utility in seroprevalence studies, contact tracing strategies, screening and essential workers and travelers, and may have a role in ascertaining response to potential vaccines. Both assays in our study have demonstrated excellent performance in terms of specificity. No cross-reactivity was observed in the 103 samples of seropositive viruses or autoimmune disorders, nor was it seen in the 60 healthy sera collected pre-December 2019. To our knowledge, this is the first study that has critically assessed the diagnostic performance and correlation of 2 commercially available immunoassays for SARS-CoV-2 antibodies using a broad number of samples from COVID-19 patients and negative controls. Roche and Abbott are leading manufacturers in many clinical chemistry and microbiology laboratories, and their respective assays are laboratory automation system compatible. With COVID-19 having been declared by the World Health Organization as a global pandemic,1  such high-throughput platforms incorporating relatively fast (<30 minute) assays will likely become a mainstay in most laboratory medicine departments.

Our sensitivity data show that Roche marginally outperforms Abbott at critical time points of 14 days and 21 days, albeit with overlapping 95% confidence intervals. This could be due to the Roche Elecsys assay measuring total antibodies and the Abbott assay specifically detecting IgG. However, there were no significant differences in the earlier time points, and overall sensitivities when incorporating early time points remain poor. Our data have shown that sensitivities for more than 21 days only reached 90.6% in the Roche assay and 84.4% in the Abbott assay. Interestingly, if we inspect the validation data from the Roche insert, they use days after PCR confirmation and describe a 100% sensitivity in more than 14 days for n = 29 samples. The use of post-PCR days as a criterion by Roche for classifying days of symptoms may lead to a bias in reporting, because the true number of days of symptoms for the patient could very well be much more than reported. Our Roche data are consistent with those found by Public Health of England, which had reported a sensitivity of around 87% for days of symptoms more than 14 days or more than 21 days.10  The Abbott assay's product insert also claimed a sensitivity of 100% after 14 days (n = 88 sera). However, our study had more unique patients (n = 170); Abbott had only 31 unique patients and Roche had only 69 patients, which suggested that the manufacturers had used duplicated patient sera for the study. Use of duplicated patients' samples across their natural history of illness will serve as a better cohort for investigating seroconversion instead of diagnostic accuracy, which was the aim of our study. Having duplicated patients and the use of multiple sera from the same patient across different time points may allow an early seroconversion patient to continue demonstrating the presence of antibodies, hence skewing the results to falsely elevated sensitivities across groups.

Based on the mean signal intensities, our results suggested that IgG alone peaked earlier than total antibodies (IgM and IgG), and this is in line with results by Long et al,11  who showed that IgG seroconversion occurred earlier than IgM in their subset of patients. Clinicians have to be cognizant that testing earlier than the 14-day time point could risk a higher likelihood of false negatives because patients may not have had sufficient time to develop antibodies in their immune system. Our findings are further corroborated by the report by Zhao et al,4  who showed that SARS-CoV-2 IgG takes a minimum of 12 days to show up in COVID-19 patients' serum.

It must be appreciated that both assays in this study have antigens directed against the nucleocapsid protein, and as such both results may be sufficiently correlated to a large extent based on their Cohen κ score. Users should be aware of the existence of other commercially available assay configurations directed against different antigens, which may include spike protein, receptor-binding domain protein, and viral neutralizing antibody assays. In particular, neutralizing antibodies confer a status of “true immunity” upon the host, and they function by inhibiting receptor-binding domain from binding to their receptors, and hence prevent spike protein 2–mediated entry into the host cell.12  Many vaccine trials are now focusing on the spike protein target for SARS-CoV-2 to elicit an immune response of B and T cells in the host.

We had several limitations in our study. First, we did not have access to sera from patients infected with other human coronaviruses, such as SARS-CoV-1 or MERS-CoV, which would serve to strengthen our cross-reactivity data. In light of limited reagent availability and the short on-board stability, we also had difficulty in performing imprecision and interference studies which could be presented at a later time. Lastly, in view of limited residual sera and the need to analyze results on both platforms, we had fewer n-numbers in the 14- to 21-day subgroup.

In conclusion, our report shows very good specificity for both Abbott and Roche SARS-CoV-2 antibody assays, although poorer than expected sensitivities compared with the manufacturer's claims. Some of the potential factors that may influence a negative serology result in a COVID-19–positive patient include that of immunosuppressive regimens or drugs such as lopinavir-ritonavir, as well as mild acute respiratory illness, which we postulate could cause the host to mount an insufficient immune response. Lastly, patients who have had less than 14 days of symptoms prior to testing could be in a preseroconversion state. Physicians should interpret such cases with caution. Nonetheless, our data suggest there is strong correlation between the 2 assays, which is valuable information for different laboratories reporting different results.

We would like to thank Temasek Holdings Pte Ltd for sponsoring the laboratory testing kits used in this study.

References

References
1.
Cucinotta
D,
Vanelli
M. WHO
Declares COVID-19 a pandemic
.
Acta Biomed
.
2020
;
91
(1)
:
157
160
.
2.
Zhu
N,
Zhang
D,
Wang
W,
et al.
A novel coronavirus from patients with pneumonia in china, 2019
.
N Engl J Med
.
2020
;
382
(8)
:
727
733
.
3.
Hong
KH,
Lee
SW,
Kim
TS
et al.
Guidelines for laboratory diagnosis of coronavirus disease 2019 (COVID-19) in Korea
.
Ann Lab Med
.
2020
;
40
(5)
:
351
360
.
4.
Zhao
J,
Yuan
Q,
Wang
H,
et al.
Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019
[published online ahead of print, March 28, 2020].
Clin Infect Dis.
2020
.
5.
Xiao
AT,
Gao
C,
Zhang
S.
Profile of specific antibodies to SARS-CoV-2: the first report
.
J Infect
.
2020
;
81
(1)
:
147
178
.
6.
Chew
KL,
Tan
SS,
Saw
S,
et al.
Clinical evaluation of serological IgG antibody response on the Abbott Architect for established SARS-CoV-2 infection
[published online ahead of print, June 9, 2020].
Clin Microbiol Infect.
2020
.
7.
Tang
MS,
Hock
KG,
Logsdon
NM,
et al.
Clinical performance of two SARS-CoV-2 serologic assays
.
Clin Chem
.
2020
;
66
(8)
:
1055
1062
.
8.
Tan
SS,
Yan
B,
Saw
S,
et al.
Practical laboratory considerations amidst the COVID-19 outbreak: early experience from Singapore
[published online ahead of print March 20, 2020].
J Clin Pathol.
2020
.
9.
McHugh
ML.
Interrater reliability: the kappa statistic
.
Biochem Med (Zagreb)
.
2012
;
22
(3)
:
276
282
.
10.
Duggan
J,
Brooks
T,
Migchelsen
S.
Evaluation of Roche Elecsys Anti-SARS-CoV-2 serology assay for the detection of anti-SARS-CoV-2 antibodies
.
2020
.
11.
Long
QX,
Liu
BZ,
Deng
HJ,
et al.
Antibody responses to SARS-CoV-2 in patients with COVID-19
.
Nat Med
.
2020
;
26
(6)
:
845
848
.
12.
Jiang
S,
Hillyer
C,
Du
L.
Neutralizing antibodies against SARS-CoV-2 and other human coronaviruses
.
Trends Immunol
.
2020
;
41
(5)
:
355
359
.

Author notes

Tambyah has received grants paid to the National University Hospital from Roche (CP40617, MV40618), Johnson & Johnson (63623872FLZ3001, 63623872FLZ2002), Sanofi Pasteur (H-030-014), GlaxoSmithKline, and Shionogi (1601T0831). The other authors have no relevant financial interest in the products or companies described in this article.