Measuring accuracy of the neutralizing activity of COVID-19 convalescent plasma

To the Editor,

Plasma collected from individuals recovered from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, called hyperimmune or convalescent plasma (CP), has been utilized worldwide during the pandemic Coronavirus Disease 2019 (COVID-19) period with mixed results [1]. Assuming that the main determinant of CP effectiveness is its level of anti-SARS-CoV-2 neutralizing antibodies, it is reasonable that most investigators focused their research on the identification of the optimal neutralization tests. For instance, Bal and colleagues recently evaluated the neutralizing activity of nine high-throughput SARS-CoV-2 commercial serological assays and concluded that, although with some differences, they cannot replace the reference virus neutralization test (VNT) [2]. Several studies have tried to validate various serological tests as surrogate of VNT [3, 4], which is currently considered the gold standard for assessing the neutralization potency of CP but which is quite demanding (utilizes live SARS-CoV-2 virus and thus require a biosafety level 3 laboratory) and time-consuming (5–7 days for results) [5]. This argument is particularly critical in the setting of COVID-19 CP donation, where there is the need to collect rapidly high-titer CP units. To further elucidate this still unsolved issue, we have compared the traditional VNT₅₀ (performed at the Molecular Virology Unit of the University Hospital of Pavia and based on the determination of the in vivo cytopathic effect, as previously described) [6] and the chemiluminescent immunoassay (CLIA) DiaSorin (LIAISON SARS-CoV-2 S1/S2 IgG, DiaSorin, Vercelli, Italy) in 495 consecutive CP donors (median age 42.6 years, range 19–65 years; 401 males and 94 females).

The VNT₅₀ was used as “gold standard” and a titer ≥80 was considered positive (negative if < 80). The Pearson’s rho as measure of the correlation and the non-parametric Spearman rho were calculated. Moreover, the relationship between the two tests was investigated by linear regression, adjusting for age and gender. Adequacy of linear model was verified by visual inspection of data and linear prediction. Then, the non-parametric receiver operating characteristic (ROC) curve method was used to assess the predictive value of Diasorin SARS-CoV-2 S1/S2 IgG assay as a potential substitute of the more demanding neutralization assay. For the search of the cut-point, the following methods were used: (1) maximization of the sum sensitivity + specificity, (2) maximization of the product sensitivity × specificity, and (3) minimization of the absolute difference sensitivity – specificity. Area under curve (AUC), sensitivity, specificity, accuracy, true positives (TP), false negatives (FN), false positives (FP), true negatives (TN) and diagnostic odds ratio (DOR) at each cut-point were reported as well. Stata 16.1 and R version 4.0.3 were used to perform calculations.

The in vitro neutralization test detected 340 subjects with a titer ≥80 (neutralization-positives), and 155 subjects with a lower titer (neutralization-negatives). By linear multiple regression method on 495 observations, the linearity of the relationship between the quantitative VNT₅₀ and the DiaSorin SARS-CoV-2 S1/S2 IgG assay was far from ideal (Figure 1, top). A locally weighted scatterplot smoother (LOWESS) was added to the figure, for comparison to the linear prediction. Using the ROC curve method, the area under the curve was 0.6471 (95% CI 0.5957, 0.6985) (Figure 1, bottom). The optimal cut-point was 89 with methods (1), 70 with (2), and 63 with (3). As showed in Table 1, at the first cut-point (89), the sensitivity was 43%, the specificity was 81%, and the fraction of the correctly classified patients (accuracy) was 55%. At the third cut-point (63), the sensitivity, specificity and accuracy were 59%.

Figure 1:

Correlation between viral neutralization and Diasorin tests by linear multiple regression (top) and ROC curve (bottom) methods.

The ROC curve method appeared fit to evaluate the predictive value of the DiaSorin test. Based on the AUC, the DiaSorin assay showed a modest diagnostic accuracy using the traditional VNT₅₀ as the reference test. Its capability to predict the neutralization assay cannot be considered adequate due to the overlap of negative (<80) and positive (≥80) results. This imperfect separation explains the low AUC value.

In conclusion, the suboptimal sensitivity, specificity and accuracy of the Diasorin SARS-CoV-2 S1/S2 IgG assay with respect to VNT₅₀ observed in our study is in agreement with the results of the study by Bal and colleagues [2] and confirms that this method cannot replace viral neutralization for the correct determination of the neutralizing activity of CP.