Efficiency evaluation of a SARS-CoV-2 diagnostic strategy combining high throughput quantitative antigen immunoassay and real time PCR

Introduction

The current coronavirus pandemic (COVID-19) has challenged healthcare systems around the world. Laboratory testing of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) played a frontline role in the effort to prevent and contain local outbreaks. However, the overwhelming and unprecedented amount of SARS-CoV-2 molecular and antigenic test requests has driven most laboratories to the limit of testing capacity. The gold standard assay for the diagnosis of COVID-19 still remains the nucleic acid amplification test (NAAT) performed in swab specimen from the upper respiratory tract [1]. Although direct detection of viral RNA particles by real-time reverse transcription polymerase chain reaction (RT-PCR) represents the most sensitive diagnostic tool at our disposal, it constitutes a major bottleneck in the diagnostic process. Namely, it is laborious, has a long turnaround time, and requires trained personnel and expensive equipment. In addition to this, positive RT-PCR SARS-CoV-2 results do not always indicate contagiousness.

The detection of the SARS-CoV-2 antigen (e.g. nucleocapsid) by rapid diagnostic tests (Ag-RDTs) is mainly based on lateral flow immunochromatography and represents a valid, quick, and simple diagnostic alternative [2]. However, also these assays suffer from drawbacks. They are mainly performed manually, binding high laboratory personnel resources and show lower diagnostic performances compared to NAAT. To date a large number of Ag-RDTs with different analytical performances have been developed by different manufacturers, and a constantly growing body of literature evaluating their performance is available. According to a recent meta-analysis, Ag-RDTs show an overall sensitivity of 76% if compared to the gold standard RT-PCR [3]. Nevertheless, Ag-RDTs detect the vast majority of SARS-CoV-2-infected persons within the first week of symptom onset and those with high viral load [4].

Recently, a new generation of high-throughput quantitative laboratory-based antigen assays has become available. The SARS-CoV2 antigen immunoassay (CoV2Ag) on the high-throughput Siemens Atellica IM 1300 analyzer targets epitopes of the viral nucleocapsid protein and is a fully automated quantitative immunoassay based on acridinium ester chemiluminescent technology (CLIA). High-throughput laboratory-based antigen tests represent a breakthrough for the integration of the COVID-19 diagnostic into the central laboratory automation. They offer many potential advantages compared to Ag-RDTs, including generation of quantitative results, 24/7 availability, lower costs, higher throughput, reduced hands-on time and automated result transmission into electronic patient health records [5]. Notably, the quantitative output of these assays can support clinical decision making, as higher viral loads have been convincingly associated with increased contagiousness of patients [6, 7].

This study aimed to verify the diagnostic accuracy of the CoV2Ag assay and to explore the reliability of a tentative diagnostic algorithm combining high-throughput antigen and conventional molecular tests to address potential future challenges of the SARS-CoV-2 pandemic.

Materials and methods

Results

Overall performance of the CoV2Ag assay

The Spearman’s rank correlation analysis of SARS-CoV2 positive samples showed a strong, negative relationship between quantitative results measured by RT-PCR and CoV2Ag (R=−0.826 [95% CI: −0.847 to −0.801]; Figure 1). The median CoV2Ag levels (Index) of the RT-PCR positive and negative samples were 13.56 and 0.25 respectively (IQR range 1.46–139.4 vs. 0.16–0.33; p<0.0001).

Figure 1:

Scatter plot comparing SARS-CoV2 viral loads from nasopharyngeal swabs obtained by RT-PCR [Ct-value] and COV2Ag assay [Index]. Data were plotted only for RT-PCR positive samples (Ct<40; n=795). Dashed line indicates reference range value of the COV2Ag assay.

Receiver operating characteristic (ROC) analysis of the CoV2Ag assay according to RT-PCR outcome is shown in Figure 2. The area under the curve (AUC) was 0.947 (95% CI 0.933–0.959). Overall sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) at the manufacturer cut-off (Index=1) were 79.1% (95% CI 75.6–81.3), 97.8% (95% CI 95.8–99), 69.2% (95% CI 66.3–71.9) and 98.6% (95% CI 97.4–99.3), respectively. Since the objective of our study was to evaluate the use of the CoV2Ag assay as screening test to quickly rule-in SARS-CoV2-positive patients, we aimed for maximum specificity. Therefore, a modified cut-off (Index=1.2) was applied, which increased specificity to 99.75% (95% CI 98.6–100) and PPV to 99.8% (95% CI 98.9–100), however at the expense of lower sensitivity and NPV, which decreased to 76.5% (95% CI 66.3–71.9) and 67.6 (95% CI 64.9–70.3), respectively. When the analysis was restricted to cases with Ct values ≤30 and ≤25, the sensitivity of the assay improved to 98.1% (95% CI 95.4–100) and 100%, respectively (Tables 2A and 2B). In contrast, the CoV2Ag assay showed low sensitivity in patients with very low viral loads (Ct values between 30–34 and 35–39), ranging from 50.9 to 18.3%, respectively. The positive and negative likelihood ratios (LR+; LR−) of interval-specific CoV2Ag results are summarized in Table 3. A LR+ of 10 was achieved at a CoV2Ag cut-off value of 1.2. Furthermore, all samples with CoV2Ag results above a cut-off value of 2.2 where RT-PCR positive (LR+=∞).

Figure 2:

CoV2Ag receiver operating curve (ROC) analysis. Dotted line: 95% confidence intervals (n=1,224).

Table 2A:

Sensitivity and specificity of COV2Ag according to RT-PCR Ct-intervals (n=1,172).

Ct values	n	COV2Ag positive	COV2Ag negative	Sensitivity, %	Specificity, %
<20	12	12	0	100	–
20–24	224	224	0	100	–
25–29	303	293	10	96.7	–
30–34	163	83	80	50.9	–
35–<40	93	17	76	18.3	–
≥40, neg	377	9	368	–	97.6
Total	1,172	638	534

Table 2B:

Sensitivity of COV2Ag according to RT-PCR Ct cumulative ranges (n=1,172).

Ct values	n	COV2Ag positive	COV2Ag negative	Sensitivity, %	Specificity, %
<20	12	12	0	100	–
<25	236	236	0	100	–
<30	539	529	10	98.1	–
<35	702	612	90	87.2	–
<40	795	629	166	79.1	–
≥40, neg	377	9	368	–	97.6

Table 3:

LR(+) for disease (defined as RT-PCR positive) by COV2Ag interval (n=1,224).

COV2Ag range	RT-PCR positive	RT-PCR negative	LR(−)	LR(+)
<0.1	8	50	0.16	na
0.1–0.99	168	344	0.48	na
1.0–9.99	198	9	na	22.1
10–99.9	218	0	na	∞
>100	229	0	na	∞

CoV2Ag test performance according to symptom onset

The results of the CoV2Ag assay in patients presenting at the hospital’s COVID-19 test center for whom clinical data and time of symptoms onset were recorded (n=711) are shown in Figure 3. The highest CoV2Ag values were observed in patients presenting after 2–4 days from symptom onset (median 20.1; IQR 2.81–218). The sensitivity of the CoV2Ag was comparable in patients presenting within one or two to four days from symptom onset (80.5 vs. 84.8%). Overall sensitivity of the CoV2Ag in symptomatic patients was 82.1%. However, sensitivity was significantly decreased in patients presenting after 5 days from symptom onset (73.3%) and in asymptomatic patients (68%; Table 4). Nine patients with negative RT-PCR had false positive CoV2Ag results ranging from Index 1.03 to 2.2. Using the modified cut-off (1.2 Index) led to one single false positive CoV2Ag result, corresponding to an overall specificity of 99.6%.

Figure 3:

COV2Ag Index and RT-PCR results in patients without (asymptomatic) or presenting after 1, 2–4 and more than 5 days after symptoms onset (n=711).

Table 4:

Sensitivity of COV2Ag by RT-PCR positive asymptomatic and symptomatic patients (n=487).

Symptoms onset, days	n	RT-PCR +	RT-PCR + COV2Ag +	RT-PCR + COV2Ag −	Sensitivity, %
Asymptomatic	89	25	17	8	68.0
0–1	192	123	99	24	80.5
2–4	382	309	262	47	84.8
>5	48	30	22	8	73.3
Overall	711	487	400	87	82.1

Combined diagnostic approach for SARS-CoV2 screening

To manage the overwhelming amounts of SARS-CoV2 test requests during potential future pandemic peaks, we assessed a two-steps diagnostic strategy based on the sequential use of antigen and molecular assays. Patients meeting the local criteria for molecular testing should be screened first by the fully-automated high throughput CoV2Ag assay. RT-PCR testing would be performed exclusively for CoV2Ag negative samples. This algorithm takes advantage of the excellent PPV of the CoV2Ag assay. High PPV are particularly met during pandemic phases with high disease prevalence (>20%; Figure 4). In fact, as the prevalence of the disease increases, the PPV of the CoV2Ag test also increases from 98.75 to 99.86% (Table 5). To assess the efficiency of this diagnostic approach we performed a retrospective simulation taking into consideration the disease prevalence registered in our hospital from December 2020 to June 2022 and assessed the diagnostic performance of the CoV2Ag assay. The analysis showed a 36.1% mean reduction of the number of RT-PCR tests during the acute phase of the COVID-19 pandemic between 25.12.2021 and 1.4.2022 corresponding to savings in reagent costs and technician workload up to 8,000 €/day and 10.5 h/day respectively (Figures 5A, B).

Figure 4:

Predictive values of the COV2Ag assay against disease prevalence. Dotted line: positive predictive value (PPV); straight line: negative predictive value (NPV); thin lines: 95% confidence intervals.

Table 5:

Positive (PPV) and negative (NPV) predictive values according to the prevalence of disease.

Prevalence, %	PPV, %	NPV, %
20	98.75	95.00
30	99.27	91.72
40	99.53	87.69
50	99.68	82.61
60	99.79	76.00
70	99.86	67.06

Figure 5:

(A) SARS-CoV2 positivity rate (dashed line); amount of SARS-CoV2 test requests (gray line). Simulation data: relative amount (%) of RT-PCR tests potentially avoided following the COV2Ag/RT-PCR sequential algorithm (solid line). Threshold of 20% (positivity rate) and 200 test requests (dotted horizontal line). (B) Laboratory efficiency expressed in technician’s time in hours (solid line) and reagent costs in € (dotted line) for SARS-CoV2 analysis following the COV2Ag/RT-PCR sequential algorithm (simulation).

Discussion

The objectives of this study were twofold: first, to evaluate the analytical performance of the CoV2Ag assay compared to RT-PCR, and second, to investigate the efficiency of a two-step diagnostic strategy in order to increase the overall testing capacity.

The evaluation of the CoV2Ag assay was performed in accordance with the criteria proposed by the IFCC, which consist of performing a prospective, independent study using a suitable sample type (i.e., nasopharyngeal swabs) of symptomatic and asymptomatic patients. In addition, the antigen test should be compared to the gold standard method (NAAT-based test) immediately after collection (within 24 h) using the same or paired specimens collected consecutively and strictly following the procedures recommended by the manufacturer [2]. The diagnostic performance of the CoV2Ag assay at the manufacturer cut-off (Index=1) demonstrated an overall sensitivity and specificity of 79.1 and 97.8%, respectively. Lippi et al. and Brummer et al. reported similar results for the CoV2Ag assay and Ag-RDTs respectively [4, 11]. Across all meta-analyzed samples in their study, the pooled Ag-RDT sensitivity and specificity were 71.2% (95% CI 68.2–74.0%) and 98.9% (95% CI 98.6–99.1%), respectively. Interestingly, a comparison study of Osterman et al. including four commercial antigen assays on automated platforms and one manual assay showed low sensitivities ranging from 17.7 to 52.3%, but comparable specificities between 97.0 and 99.7% in samples of hospitalized patients. However, the study was performed at the end of the second pandemic wave, when the number of newly infected individuals was declining, probably leading to difficulties to include patients at the time of initial diagnosis and high viral loads [12]. Recently, Lippi et al. performed a pooled analysis of the diagnostic performance of the Siemens CoV2Ag assay, reporting cumulative sensitivity of 79% and a specificity 98% [11]. The analytical performances described in the literature appear to be partially divergent, underscoring the fact that the sensitivity of the test is highly dependent on the distribution of viral load in the test population [13], [14], [15], [16]. This may be influenced by several factors, such as inclusion criteria (asymptomatic vs. symptomatic patients), timing of specimen collection, type of specimen (i.e., nasopharyngeal swab or nasal swab), and prevalence of disease at inclusion. We used Ct-values as a semiquantitative surrogate marker of viral load. The viral load distribution in our sample collective was right-skewed, indicating an overrepresentation of samples with very high Ct-values between 35 and 40 (Supplementary Figure 1). This might explain the lower sensitivity observed in our study compared with that reported by Hörber et al. [14]. Indeed, despite showing outstanding specificity, the overall sensitivity of the CoV2Ag assay did not fully meet the WHO minimal performance requirements for antigen tests compared to a reference NAAT, requiring at least 80% sensitivity and ≥97% specificity [1, 2]. However, these prerequisites were fulfilled when calculation of sensitivity was limited to samples with Ct-values lower than 37. Considering only samples with low Ct-values (<30 and <25) increased the estimated sensitivity to 98.1 and 100% respectively (Table 2B). These results highlight that the CoV2Ag assay is particularly reliable in samples from patients with moderate to high viral loads (<30 Ct-values), as previously demonstrated for similar automated chemiluminescence methods [17]. Notably, individuals with Ct-values above 30 seem to have such a low viral load that may be insufficient to transmit the virus [18]. In addition, several lines of evidence suggest that Ag-RDTs may correlate better with virus culture-based test results than RT-PCR [19]. Thus, it appears that the diagnostic features of the CoV2Ag test may be sufficient to rapidly identify SARS-CoV2-infected individuals with high viral loads and effectively contribute to stem the spread of the virus.

Additionally, we estimated the LR+ of interval-specific CoV2Ag results. The LR+ is defined as the fraction of the true positive rate with a specific CoV2Ag outcome divided by the false positive rate with the same outcome. A high LR+ (10 or greater) will result in a significant increase in the probability of a positive SARS-CoV2 RT-PCR test outcome, given a positive CoV2Ag assay result [20]. In our hands, a LR+ of 10 was reached at a CoV2Ag assay Index of 1.2, qualifying this cut-off for the prediction of a SARS-CoV2 PCR positive outcome with a very high degree of certainty.

The utilization of an antigen assay in the diagnostic strategy strongly depends on the specific clinical settings [21]. As expected, the performance of the CoV2Ag assay was superior in symptomatic patients and those presenting within four days from symptom onset (Table 4). In these groups, sensitivity of the CoV2Ag was above 80%, meeting the minimal WHO requirements. Similarly, Burmmer et al. described a decreased sensitivity in asymptomatic individuals and in those presenting later in disease course (>7 days from symptom onset) [4]. Overall, our data support the use of the CoV2Ag assay to rapidly identify symptomatic patients with high SARS-CoV2 viral loads within the first four days of symptom onset.

With respect to a future COVID-19 pandemic, there remains a significant need to explore new COVID-19 testing strategies, allowing mass screening, a reliable and quick identification of infected people and a reduced economic impact. NAAT-based testing strategies still demonstrate the highest diagnostic performance. However, they are expensive, time-consuming, and often have a time lag of 24–48 h between specimen collection and result generation. These factors limit their widespread use and effectiveness in preventing disease transmission. On the other hand, antigen assays, especially the new generation of highly automated tests, are suitable for mass screening, have a short turnaround time and exhibit very high specificity. However, they show lower diagnostic sensitivity than molecular tests. A diagnostic approach combining both the new high-throughput antigen- and classical NAAT-based assays might potentially obviate these problems. In their recent work, Halfon et al. proposed an optimized stepwise analysis that combines Ag-RDTs and RT-PCR for the screening of patients for SARS-CoV2 [22]. This approach reduced the number of RT-PCRs performed by only 4%. However, the calculations were performed on a sample collective showing a low SARS-CoV2 positivity rate of 11%, which may strongly affect the efficiency of the two-step analysis strategy. Pighi et al. performed a cost-effectiveness analysis of different screening strategies based on Ag-RDT or laboratory-based SARS-CoV2 antigen testing [23]. Similarly to our findings, they showed that combining a laboratory based antigen assay followed by RT-PCR in negative samples, was equally accurate but reasonably cheaper than using alternative approaches. However, as shown in our retrospective analysis, to benefit from such an algorithm certain conditions, such as a SARS-CoV2 positivity rate above 20% and a number of test requests per day above 200, must be met. According to our simulation, the use of a screening algorithm combining the automated CoV2Ag assay and RT-PCR during one of the acute phases of the pandemic (25.12.2021–01.04.2022; 97 days) would have dramatically improved the efficiency of SARS-CoV2 laboratory testing by reducing the analytical costs of €320′590 and the staff workload of 370 h. At the same time, an excellent mean turnaround time for positive SARS-CoV2 results could be achieved (3.1 h) reducing the number of SARS-CoV2 RT-PCR performed by 36.1%.

In summary, our results show an excellent performance of the automated high-throughput CoV2Ag assay. The diagnostic algorithm proposed in our study can offer an alternative approach for an efficient screening procedure of SARS-CoV2 during critical pandemic phases. This procedure would allow laboratories to improve their testing capacity and rapidly deliver accurate diagnosis.