Interlaboratory Agreement of Anti–Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Serologic Assays in the Expedited College of American Pathologists Proficiency Testing Program

Data were collected from the initial CAP SARS-CoV-2 Serology Survey (COVS-A 2020). Three individual samples, each from single donors, (2 that pretested positive and 1 negative) were sent to 1195 subscribing laboratories on June 22, 2020, along with kit instructions and the result reporting form. Each laboratory received a 0.5-mL aliquot for each sample, sent in an insulated container with a cool pack and instructions to store samples at 2°C to 8°C until testing could be performed. Laboratories were instructed to perform serology testing by using the methodology routinely performed on clinical specimens and report the results to the CAP by July 14, 2020. Reporting fields for qualitative and quantitative (ie, numeric values such as signal to cutoff ratio or index value) results were available for total antibody, IgG, IgM, and IgA. A reporting field for titer results was also available for each antibody class and instructions were given to use this reporting field only for neutralization assays. The result reporting form also included fields for method and manufacturer codes.

A supplemental questionnaire developed by the working group was also distributed with the COVS-A 2020 survey. This questionnaire consisted of 4 questions (Supplemental Table 1, available at https://meridian.allenpress.com/aplm in the May 2021 table of contents) and was designed to assess the state of SARS-CoV-2 serology testing in participating laboratories.

Results were compiled and analyzed in Microsoft Excel (Redmond, Washington). For results indicating the manufacturer code of “Other,” manufacturer was manually assigned by using write-in responses and answers to supplemental questions. Reponses from 17 participants were recorded in the fields for “Total Antibody” but used manufacturers that only had commercially available assays for individual isotypes when the survey was distributed. For example, the only Abbott Diagnostics antibody assay available for SARS-CoV-2 in June 2020 was an IgG test, yet 7 laboratories listed Abbott for a “Total” antibody result. For our analysis, we manually assigned these responses to IgG. Chi-square tests of association (SAS, Cary, North Carolina) were performed to compare differences in the reported results for each sample across lateral/non–lateral flow methods and across manufacturer. Fisher exact test was used if any table cell had fewer than 5 results. Only those manufacturers with greater than 15 results per sample were tested.

Development of a new proficiency testing survey is often a lengthy process (averaging about 18 months from conception to market availability) involving assessment of need, feasibility of obtaining appropriate material, and anticipating and addressing potential issues surrounding testing, reporting, and return of results. Early in the COVID-19 pandemic, clinical laboratories in the United States and worldwide quickly evaluated and validated SARS-CoV-2 serology assays and began reporting patient results. CAP responded by rapidly developing a proficiency testing program, and the product was available to participants within 3 months of initial conception. The proficiency testing program was developed with input from an ad hoc working group composed of members of the CAP Microbiology and Diagnostic Immunology and Flow Cytometry Committees. These committees provide expert scientific support in diagnostic microbiology, immunology, and flow cytometry, and define and monitor the state of the art and emerging technologies in these fields. These committees also support CAP proficiency testing surveys in their respective areas of expertise, including identifying opportunities for new surveys or analytes.

Candidate SARS-CoV-2 serology proficiency testing materials from 2 commercial reference material vendors were identified and distributed to CAP members laboratories for pilot testing using anti–SARS-CoV-2 antibody assays from the following manufacturers: Abbott, Beckman, Diasorin, Epitope Diagnostics, Euroimmun, Ortho, and Roche, as well as a laboratory-developed test. Three samples (2 that pretested positive, and 1 negative) from unique individuals were sent to subscribing laboratories in the COVS-A 2020 survey.

Responses from 1110 laboratories were returned by the reporting deadline. Most laboratories were located in the United States, but 115 (10.4%) were from other countries, including Canada (n = 16, 1.4%), Japan (n = 16, 1.4%), Singapore (n = 14, 1.3%), the United Arab Emirates (n = 8, 0.7%), and 31 other countries (2.8%). For each of the 3 samples, participants had the option to provide qualitative, quantitative, and titer results for total antibody, IgG, IgM, and IgA. A total of 4125 discrete qualitative results were received.

The kit instructions specified to report titer results only when using a neutralization assay. Only 4 of the 1110 participants reported results for antibody titer; 2 reported titer results for total antibody and 2 for IgG. The limited number of laboratories reporting titer information precludes further analysis and indicates that at the time this survey was performed, very few clinical laboratories were reporting results from neutralization assays.

Total antibody qualitative results from 387 laboratories were analyzed. Across all manufacturers, sample COVS-01 was reported as “detected” (positive) by 383 participants (99.0%), sample COVS-02 was reported as “detected” (positive) by 386 participants (99.7%), and sample COVS-03 was reported as “not detected” (negative) by 381 of 386 responding participants (98.7%). No laboratories reported “indeterminate/borderline” for any of the samples. These results suggest that overall agreement of total anti–SARS-CoV-2 antibody assays for these proficiency testing samples was excellent.

Results from individual manufacturers are summarized in Table 1. Roche was the most commonly used manufacturer, followed by Ortho and Siemens. A smaller number of laboratories reported using assays from Bio-Rad, other manufacturers, or laboratory-developed tests. Manufacturers reported by fewer than 5 respondents and included in the “Other” category were Beckman, GenScript, NOWDiagnostics, PHASE Scientific, Wantai, and Wondfo.

A total of 772 laboratories reported qualitative results for anti–SARS-CoV-2 IgG antibody assays. Across all manufacturers, sample COVS-01 was reported as “detected” (positive) by 701 participants (90.8%), “indeterminate/borderline” by 14 participants (1.8%), and “not detected” (negative) by 57 participants (7.4%). Sample COVS-02 was reported as “detected” (positive) by 769 participants (99.6%), “indeterminate/borderline” by 1 additional participant (0.1%), and “not detected” (negative) by only 2 laboratories (0.3% of participants). Sample COVS-03 was reported as “not detected” (negative) by 764 of 768 responding participants (99.5%), with only 4 laboratories (0.5%) reporting “detected” (positive). Together, these results suggest that overall agreement of anti–SARS-CoV-2 IgG antibody assays for these proficiency testing samples was also very good.

Anti–SARS-CoV-2 IgG antibody results from individual manufacturers are summarized in Table 2. Abbott was the most commonly used manufacturer, followed by Ortho, Diasorin, and Beckman. A smaller number of laboratories reported using assays from other manufacturers or laboratory-developed tests. Manufacturers reported by fewer than 5 respondents and included in the “Other” category were ACROBiosystems, Ansh Labs, Artron Laboratories, Bio-Rad, BioMedomics, bioMérieux, Boston Molecules, Clarity, ET Healthcare, Meridian Bioscience, PerkinElmer, PHASE Scientific, Premier Biotech, Quansys Biosciences, Savyon Diagnostics, Ringbio, SD Biosensor, Siemens, SNIBE, TheraTest, UCP Biosciences, Viramed, Vircell, and ZEUS Scientific.

Quantitative results (ie, numerical values such as signal to cutoff ratio or index value) for anti–SARS-CoV-2 IgG antibodies were reported by 113 laboratories. The mean and median numerical result for COVS-01 was consistently lower than that reported for COVS-02 for each manufacturer. This suggests that sample COVS-01 contained a lower level of anti–SARS-CoV-2 IgG antibodies than COVS-02, and the decreased performance in qualitative reporting for COVS-01 may be related to interpretive cutoff and/or analytic sensitivity. Although COVS-01 was reported as “detected” (positive) by more than 90% of participants overall, laboratories using assays from Autobio, Cellex, Diazyme, Epitope, Gold Standard, Orient Gene, Syntron, and laboratory-developed tests were less likely to have detected anti–SARS-CoV-2 IgG antibodies in this sample (P < .001) and had more than 1 participant report this sample as “not detected” (negative).

Fewer participants reported results for anti–SARS-CoV-2 IgM and IgA assays than for IgG and total antibody assays. Sample COVS-03 was reported as “not detected” (negative) for IgM by 188 of 190 participants (98.9%) and “not detected” (negative) for IgA by 26 of 27 participants (96.3%). This suggests that there were few false-positive results with this proficiency testing sample in assays for anti–SARS-CoV-2 IgM and IgA.

In contrast, there was less consensus for the detection of anti–SARS-CoV-2 IgM and IgA antibodies. Of 191 responding participants, sample COVS-01 was reported as “detected” (positive) for IgM by 88 (46.1%) and “not detected” (negative) for IgM by 95 (49.7%). Sample COVS-02 was reported as “detected” (positive) for IgM by 39 participants (20.4%) and “not detected” (negative) for IgM by 144 participants (75.4%). Thus, IgM detection for these 2 samples was highly variable. Table 3 summarizes the anti–SARS-CoV-2 IgM antibody results from individual manufacturers. Manufacturers reported by fewer than 5 respondents and included in the “Other” category were ACROBiosystems, Ansh Labs, Artron Laboratories, BioMedomics, bioMérieux, Boston Molecules, Clarity, ET Healthcare, Meridian Bioscience, NBGS, PHASE Scientific, Premier Biotech, Savyon Diagnostics, Ringbio, SD Biosensor, SNIBE, TheraTest, UCP Biosciences, Viramed, Vircell, and ZEUS Scientific.

Although the overall consensus between manufacturers for COVS-01 and COVS-02 was poor, users of some individual assays provided consistent results. For example, all participants using assays from Autobio, Epitope, and Gold Standard reported COVS-01 as “not detected” (negative), while all laboratories using assays from Aytu, Orient Gene, and Syntron reported COVS-01 as “detected” (positive). The differences between manufacturers were statistically significant for both COVS-01 and COVS-02 (P < .001). Thus, the lack of consensus for IgM detection most likely results from assay differences and not individual laboratory performance.

Similarly, sample COVS-01 was reported as “detected” (positive) for IgA by 17 of 27 participants (63.0%) and “not detected” (negative) for IgA by 8 participants (29.6%). Sample COVS-02 was reported as “detected” (positive) for IgA by 17 of 27 participants (63.0%) and “not detected” (negative) for IgA by 7 participants (25.9%). Table 4 illustrates that this lack of consensus results from differences between manufacturers, as all laboratories using Euroimmun assays reported “detected” (positive) results for COVS-01 and COVS-02 for anti–SARS-CoV-2 IgA antibodies, while none of the laboratories using the assay from Gold Standard reported these samples as positive. Together, these results suggest variability in clinical assays for detection of IgM and IgA antibodies to SARS-CoV-2.

Recent data have called into question the accuracy of anti–SARS-CoV-2 lateral flow antibody assays^9,10 ; thus, we sought to assess performance of lateral flow assays compared to other methodologies. Results were reported from 107 and 109 laboratories using lateral flow assays for anti–SARS-CoV-2 IgG and IgM, respectively. Although the overall consensus for COVS-01 IgG was greater than 90%, a reduced percentage of participants (82 of 107, 76.6%; P < .001) using lateral flow assays detected anti–SARS-CoV-2 IgG antibodies in this sample (Table 5). Detection of anti–SARS-CoV-2 IgG antibodies in the strongly positive COVS-02 sample was similar between laboratories using lateral flow (106 of 107, 99.1%) versus other methodologies (663 of 665, 99.7%). Detection of anti–SARS-CoV-2 IgM antibodies was more frequent with lateral flow assays than other methodologies for samples COVS-01 and COVS-02 (P < .001 for both). There were no false-positive results for IgG or IgM antibodies reported for COVS-03 from participants using lateral flow devices. Only 3 participants provided results for total antibody assays using lateral flow methodology; the limited number of results precluded further comparison to non–lateral flow methodologies. Together, these data suggest that the lateral flow assays used by participating laboratories may be less sensitive for the detection of anti–SARS-CoV-2 IgG antibodies than other methodologies, but are potentially more sensitive for detecting IgM.

Most laboratories reporting results (935 of 1110; 84.2%) provided answers to the supplemental questionnaire that accompanied the survey. Question 1 was designed to assess indications for testing at performing laboratories. The most common responses were “diagnosis of past infection” (N = 615) and “immune status check” (N = 458). Many laboratories (N = 129) responded “Other”—research, clinical trials, employee screening, in-development/validation, and “not yet determined” were listed among these. “Diagnosis of acute infection” was chosen by 175 respondents, with some also choosing “Other” and specifying use in symptomatic patients with negative SARS-CoV-2 PCR results.

Regarding Question 2—monthly approximate volume of SARS-CoV-2 serology orders—there was broad variability in responses (Figure). While most respondents (N = 328) reported between 0 and 100 tests per month, 254 survey respondents provided no answer to Supplemental Question 2. When plotted as a Pareto Chart (Figure), 80% of survey respondents either reported 1000 or fewer serology orders per month, or provided no volume information. However, volumes greater than 100 000 per month were reported by 9 participants.

Regarding Question 3—how many different manufacturer kits were in use in participating laboratories—most responding laboratories reported using 1 or 2 discrete manufacturer assay(s) (Table 6). However, as many as 25 discrete in-use kits for SARS-CoV-2 antibody serology were reported by a single laboratory. This may present a challenge for future proficiency testing events, as regulations allow reporting of only 1 result per analyte.

Regarding Question 4—which manufacturers' kits were represented by respondents—most respondents used kits sold by Abbott (N = 353), Roche Diagnostics (N = 169), and Ortho Clinical Diagnostics (N = 125); each other manufacturer was named by fewer than 50 respondents.

The SARS-CoV-2 pandemic has placed clinical laboratory testing in the public eye and raised awareness of the importance for reproducible and reliable test results. Currently, more than 50 SARS-CoV-2 serology assays have received emergency use authorization (EUA) from the US Food and Drug Administration. This list may continue to change over time as new assays are authorized, while other authorizations are revoked. The wide range in methodology and variability of target antigen used by these assays has raised concerns that inconsistent test performance may limit the overall utility of serology testing.

Here, we describe results from the first CAP SARS-CoV-2 serology proficiency testing survey, which included results from 1110 individual participating laboratories. Overall, we found excellent qualitative result agreement (>90% consensus for all 3 samples) across all participating laboratories for assays that detect IgG and total antibodies against SARS-CoV-2. Consensus for IgM and IgA was very high for 1 sample, which nearly all participants reported as negative. However, for the 2 samples with consensus positive results using total antibody and IgG assays, IgM and IgA were variably detected.

Although the proficiency testing survey also included reporting fields for quantitative and titer results, these were reported by a smaller number of participants and values were not standardized across platforms. These factors limited our ability to analyze the results. At the time this study was performed, most EUA-authorized tests were qualitative rather than quantitative. Thus, although many assays produce a numeric value, the clinical utility of these values remains largely uncertain. Further study may reveal antibody threshold levels that correlate with protection via passive immunity (therapeutic plasma transfusion) or active immunity, thus providing rationale for utility of quantitative antibody assessment. Development of reference materials could support efforts to standardize and harmonize quantitative antibody measures.

A primary strength of our study is the large number of results gathered; these results represent clinical assay outputs from more than 1000 clinical laboratories around the world. Further, this large sample size represents diverse testing and detection methodologies, manufacturers, and target antigen epitopes. However, a notable limitation is that this proficiency testing survey included only 3 samples, and we cannot necessarily gauge assay performance with this small number. In addition, the samples chosen for proficiency testing were pretested by using assays from multiple manufacturers and exhibited good qualitative result agreement. This pretesting effort likely contributed to the generally high result consensus. Future surveys will continue to monitor performance of clinically used assays with proficiency testing samples.

The results from our supplemental questionnaire shed light on the current indications and volumes for clinical SARS-CoV-2 serology testing. We also characterized the state of testing in participating laboratories in June/July 2020, finding that most laboratories were performing 1000 or fewer serologic tests per month, using 1 or 2 manufacturers' assays. Some of our respondents indicated use of serology testing for diagnosis of acute infection. We refer to guidelines from the Infectious Diseases Society of America and other authorities for emerging evidence regarding use of serology as an adjunct to nucleic acid amplification tests for diagnosis of acute disease.²  These guidance documents also contain information about the necessary testing interval post symptom onset to maximize clinical utility of anti–SARS-CoV-2 antibody assays.

Most respondents stated that the primary indications for serology testing at their sites were detection of past SARS-CoV-2 infection and/or to check immune status. Importantly, it remains unclear whether antibody test results correlate with protective immunity and how long potential immune-mediated protection lasts. The observation that severe and fatal COVID-19 can occur in the setting of high antibody measurements^11–13  suggests that an antibody response alone may not be sufficient to prevent severe disease and/or death, at least in the acute setting. Further studies to define whether antibody detection predicts protection from reinfection after exposure, or primary infection in the setting of vaccine administration, will help to answer these questions. We predict that clinical testing patterns may be dynamic as these data emerge. Furthermore, as vaccine studies enroll increasing numbers of participants, the role of serologic testing may further shift. Our understanding of the clinical utility and relevance of testing for SARS-CoV-2 antibodies is likely to continue to develop in the coming months and years.