Results
Characteristics of
participants
Between April and November 2020, 3’658 consecutive individuals were
included, comprising 1’085 inpatients with suspected SARS-CoV-2
infection and 2’573 medical personnel (Figure 1). Among those, 195 were
COVID positive (prevalence 5.3%). The median age was 46 years (standard
deviation, SD, 16.2); 68% of the individuals were female (a high
proportion of female nursing staff, in particular). The mean time since
PCR was 59 days (SD 47). Detailed patient characteristics are given in
Table 1.
Diagnostic accuracy for the presence of previous
COVID-19
The distribution of results obtained with various serological tests in
patients with and without previous COVID-19 is given in Figure 2; the
respective cut-off levels are depicted as a grey line. The most
significant overlap between COVID-19 positive and negative individuals
was observed in the case of anti-N ELISA. In contrast, little overlap
can be seen in anti-S1 ELISA and anti-N ECLIA.
The diagnostic accuracy of all tests in terms of receiver operating
characteristics (ROC) curves is given in Figure 3. High area under the
ROC curves (AUC) were observed for anti-S1 ELISA (0.97; 95% confidence
interval [CI] 0.95, 0.98), anti-RBD ELISA (0.95; 95% CI 0.93,
0.97), anti-S1/S2 CLIA (0.95; 0.92, 0.97), and anti-N ECLIA (0.94; 0.91,
0.97). Lower AUC values were seen for anti-N ELISA (0.90; 0.86, 0.93)
and anti-RBD+ LFI (0.92; 0.88, 0.95).
The differences in diagnostic accuracy of various serological tests are
illustrated in Figure 6 (panel A); detailed results are given in Table
2. The sensitivity to detect patients with previous COVID-19 was ≥85%
in anti-N ECLIA (86.8%; 95% CI 81.1, 91.3) and anti-S1 ELISA (86.2%;
80.5, 90.7), corresponding to 25 and 27 false-negative results.
Sensitivity was 84.7% in anti-S1/S2 CLIA (78.7, 89.5), 84.0% in
anti-RBD+ LFI (76.6, 89.8), 81.0% in anti-N CLIA (74.6, 86.3), 79.2%
in anti-RBD ELISA (72.7, 84.7), and 65.6% in anti-N ELISA (58.4, 72.3).
The corresponding numbers of false-negative results were 29 (anti-S1/S2
CLIA), 21 (anti-RBD+ LFI), 36 (anti-N CLIA), 40 (anti-RBD ELISA), and 66
(anti-N ELISA). Detailed diagnostic accuracy measures for all tests are
given in Table 2.
The specificity was 98.4% in anti-N ECLIA (98.0, 98.8), 98.3% in
anti-N CLIA (97.8, 98.7), 98.2% in anti-S1 ELISA (97.7, 98.6), 97.7%
in anti-N ELISA (97.2, 98.2), 97.6% in anti-S1/S2 CLIA (97.0, 98.1),
97.2% in anti-RBD ELISA (96.5, 97.7), and 96.1% in anti-RBD+ LFI
(95.3, 96.9). The corresponding numbers of false-positive results were
54 (anti-N ECLIA), 60 (anti-N CLIA), 62 (anti-S1 ELISA), 79 (anti-N
ELISA), 84 (anti-S1/S2 CLIA), 98 (anti-RBD ELISA), and 95 (anti-RBD+).
Diagnostic accuracy for the presence of neutralizing
antibodies
The accuracy of serological immunoassays for the presence of
neutralizing antibodies was observed in a subgroup of complex patients
(n=201). The association between the antibody response (z-scored) and
serum dilutions at full neutralization of live SARS-CoV-2 is depicted in
Figure 4.
The differences in diagnostic accuracy of various serological tests are
illustrated in Figure 6 (panel B); detailed results are given in Table
3. The sensitivity to detect neutralizing antibodies was ≥85% in
anti-S1 ELISA (92.7%; 95%CI 87.3, 96.3), anti-N ECLIA (91.7%; 86.0,
95.6), anti-S1/S2 CLIA (90.3%; 84.3, 94.6), anti-RBD+ LFI (87.9%;
80.3, 93.4), and anti-RBD ELISA (85.8; 91.0, 79.1), corresponding to 11,
12, 14, 13, and 21 false-negative results, respectively. In contrast,
sensitivity was 84.1% in anti-N CLIA (77.2, 89.7), and 66.2% in anti-N
ELISA (58.0, 73.8). The corresponding numbers of false-negative results
were 23 and 50.
The specificity was ≥ 97% in anti-N CLIA (100%; 91.8, 100), anti-S1/S2
CLIA (97.7%; 87.7, 99.9), and anti-RBD+ LFI (97.9%; 89.2, 100),
corresponding to 0, 1, and 1 false-positive results, respectively.
Specificity was 95.9% in anti-RBD ELISA (86.0, 99.5), 93.0% in anti-N
ECLIA (80.9, 98.5), 92% in anti-S1 ELISA (80.8, 97.8), and 65.3% in
anti-N ELISA (50.4, 78.3). The corresponding numbers of false-positive
results were 2, 3, 4, and 17 respectively.
Accuracy in salient subgroups of
patients
The antibody response in salient subgroups of patients is illustrated in
Figure 5 (only COVID-19 positive individuals are shown). Significant
higher antibody concentrations were observed in males, older
individuals, inpatients, and patients admitted to intensive care units,
including ventilated patients. As a sensitivity analysis, we calculated
the diagnostic accuracy of the anti-S1 ELISA for the presence of
COVID-19 in various subgroups (Table 4). The sensitivity was higher in
males (93.1%), probably reflecting more severe disease in this subgroup
of patients. Other significant differences were not observed (Table 4).
Biobank samples
Analyzing 102 anonymized biobank samples collected from inpatients
between December 2018 and February 2019, serological test results were
negative in all samples in case of anti-S1 ELISA, anti-N ECLIA, and
anti-S1/S2 CLIA. One positive test result was observed in the case of
anti-RBD ELISA and anti-N CLIA (1.0%). Four positive test results were
observed in case of anti-N ELISA and anti-RBD+ LFI (3.9%).
Discussion
We conducted a large prospective cross-sectional study in a real-life
clinical setting stringently fulfilling the requirements of a diagnostic
accuracy study and comparing all significant serological testing
strategies. Sensitivities and specificities varied remarkably between
different tests and were substantially different from manufacturer’s
specifications. The diagnostic accuracy in detecting patients with
previous COVID-19 was high in anti-N ECLIA and anti-S1 ELISA
(sensitivity ≥ 85%; specificity ≥97%). The accuracy in detecting
neutralizing antibodies was high in anti-S1/S2 CLIA and anti-RBD+ LFI
(sensitivity ≥ 85%; specificity ≥97%). Sensitivities and specificities
obtained were consistent across various patient subgroups. With these
diagnostic accuracy measures obtained in a real-life clinical setting,
we were able to fill a critical gap in knowledge, identified by many
previous authors, systematic reviews, and institutions such as the WHO
[2, 5, 9, 10, 20, 30-32].
The study presented here adds important value as it was designed (1) as
an adequately powered cross-sectional study conducted in a real-life
clinical setting, (2) to answer clearly defined clinical questions, (3)
to include a representative study population, (3) to conduct a
head-to-head comparison of all significant serological testing
strategies, (4) to select and determine the reference standard test
rigorously (e) apply optimal flow and timing. Specifically, we assessed
whether different serological testing strategies may (a) accurately
identify patients with previous COVID-19 and (b) correctly predict
neutralizing antibodies against SARS-CoV-2. However, several potential
limitations can be discussed. One might argue that we might have missed
COVID-19 in some asymptomatic healthcare workers because individuals
were asked to perform a nasopharyngeal swab in case of symptoms
consistent with COVID-19. However, we do not believe that this might
have affected the interpretation because it would not alter sensitivity
and does not affect differences between different assays. Besides, a
pre-specified complex subset of patients was selected for the live
neutralization assay. This procedure ensures that there is no
overestimation of performance.
The sensitivities and specificities obtained in clinical practice were
considerably lower compared to previous publications. For example, the
manufacturer of the anti-N ELICA claims a sensitivity of 100% (Elecsys
Anti-SARS-CoV-2 package insert; Roche diagnostics, Rotkreuz,
Switzerland), and previous studies reported sensitivities between 96 and
100% [33-38]. However, these study populations do not reflect
real-life clinical practice, and the diagnostic accuracy measures can
therefore not be applied to routine practice. In contrast, the
sensitivity in our study, which was strictly designed to reflect
clinical settings, was 87%. These differences in sensitivity translate
into a completely different interpretation of serological test results
in seroprevalence studies and individual patients. Applying the
sensitivity provided by the above-mentioned manufacturer (anti-N ECLIA)
to our study population, the number of COVID-19 patients missed by the
tests would be zero. In contrast, 25 COVID-19 patients were missed by
the same test in our population (13.2% of RT-PCR positive individuals).
The number of missed COVID-19 patients was even higher in other tests
(e.g. anti-N ELISA; n=66; 34.3%). Accordingly, the specificity of the
anti-N ELICA was stated to be 99.8%, corresponding to 7 patients
falsely claimed to have had COVID-19 in our study cohort
(false-positives). In contrast, we observed 54 false-positive
individuals (falsely claimed to have COVID-19). These values are very
similar in the case of the other serological tests.
The sensitivities and specificities must be taken into account to
interpret test results correctly. We would like to give two examples to
illustrate how this could be done. In a seroprevalence study in a
setting similar to our study cohort, one can estimate the true
prevalence by adding the numbers of false-negatives and subtracting the
number of false-positives as calculated using the diagnostic accuracy
measures determined from our study. Likewise, the probability of
neutralizing antibodies in individual patients can be estimated in a
similar calculation.
Our data support previous knowledge that the majority of patients with
COVID-19 develop antibodies against epitopes of the SARS-CoV-2 and that
these antibodies can be detected with a range of serological
immunoassays. However, this does not apply to all patients and the
clinical performance varies remarkably between different assays. A high
diagnostic accuracy in terms of previous COVID-19 was observed for
anti-N ECLIA and anti-S1 ELISA. In terms of neutralizing antibodies, the
accuracy was high in anti-S1/S2 CLIA and anti-RBD+ LFI. Our data further
confirm that the concentration of antibodies detected is strongly
associated with the intensity of neutralizing antibodies, irrespective
of assay technique. However, major questions remain which must be
addressed in future studies: (1) can serological assays be used to
distinguish between previous COVID-19 and vaccination, and (2) what is
the accuracy of serological assays to predict protective immunity at
certain serological cut-off levels.
Conclusions
In conclusions, sensitivities and specificities varied remarkably
between different tests and were substantially lower than the
manufacturer’s specifications. These diagnostic accuracy measures can be
used to calculate the virus burden within a specific population and
determine the likelihood of protection against re-infection. Thus, our
data might inform researchers, health professionals, and authorities to
interpret seroprevalence studies and test results in individual
patients.