Performance evaluation of SARS-CoV-2 antigen detection in the post-pandemic era: multi-laboratory assessment

Introduction

Coronavirus Disease 2019 (COVID-19) has taken the world by storm. In response to the pandemic, various diagnostic strategies for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), including molecular methods [1], antigen detection [2], and serological immunological methods [3], have been developed to address different application scenarios [4, 5]. At the preliminary stage, real-time reverse-transcription polymerase chain reaction (rRT-PCR) was applied for infection confirmation [6, 7]. Then gradual consensus was that asymptomatic or presymptomatic individuals posed a high risk of virus transmission [8, 9], highlighting the need for large-scale screening to identify transmission chains. This increased burden of molecular detection, substantial costs, labor-intensive efforts, and delays in results. Rapid antigen detection was initially implemented as complementary testing to block virus transmission due to its simplicity, rapidity, and cost-effectiveness [4, 10] and was stipulated to be applicable to symptomatic individuals, personnel under quarantine and community residents with self-test demands [11]. During the pandemic peak in late December 2022, the daily average of antigen tests reached a staggering 1.89 million [12], leaving people in the situation of “hard for one antigen testing”. In the post-pandemic era, as China experiences a substantial number of secondary infections [13, 14], the shortage of antigen test kits is now relieved, but there are still substantial demands for antigen testing. Antigen detection is now mainly used as a low-cost rapid detection method by individuals with suspicious symptoms or used in fever clinics for primary screening, whereas PCR tests are not as commonly employed. Antigen detection now serves as a valuable “strategic reserve” for monitoring disease trends and diagnosing those at a high risk of secondary infections.

The World Health Organization (WHO) outlined that the “acceptable” and “desirable” analytical sensitivities for rapid antigen tests are equivalent to 10⁶ and 10⁴ genomic viral copies/mL, respectively [15]. According to the quality evaluation requirements issued by China, approved antigen detection kits should have an analytical sensitivity of at least 5 × 10⁵ copies/mL [16, 17]. However, studies have illustrated that some commercially available kits, particularly in terms of their sensitivities, cannot meet the required criteria [18, 19]. As of December 30th, 2022, the National Medical Products Administration (NMPA) in China has approved 50 SARS-CoV-2 antigen detection assays, including three fluorescence immunochromatography assays, nine latex chromatography assays and 38 colloidal gold assays [20]. Full-scale external quality assessment (EQA) of antigen detection assays is crucial but scarce.

In this pilot study, a nationwide EQA was carried out by the National Center for Clinical Laboratories (NCCL) of China to shed light on the characteristics and performance of the antigen detection products, offer guidance for potential product selection and encourage manufacturers to enhance their kits.

Materials and methods

Sample preparation and panel design

Inactivated cell culture supernatants of the SARS-CoV-2 Omicron BA.1 and BA.5 strains were provided by Sinovac Biotech Co., Ltd. (China) and Sinopharm Group Co., Ltd. (China), respectively. For quantification, viral RNAs were extracted with the QIAamp Viral RNA mini kit (Qiagen, Hilden, Germany) and then reverse transcribed into cDNAs with the PrimeScript RT Reagent Kit (Perfect Real Time; TaKaRa, Japan). Subsequently, the corresponding cDNAs were subjected to real-time quantitative polymerase chain reaction (RT-qPCR) on the Applied Biosystems 7,500 Fast real-time PCR system (Thermo Fisher Scientific, USA) and droplet digital PCR (ddPCR) on the QX-200 System (Bio-Rad, USA) (Figure 1A). The primers/probe set was modified from the N (nucleocapsid) gene assay recommended by China’s Centers for Disease Control (CDC) because of the mutation in the forward primer binding site (Table 1).

Figure 1:

Flow charts of the EQA scheme. Sample preparation (A), suitability tests (B), homogeneity tests (C), stability tests (D), and geographical distribution and types of laboratories (E).

Table 1:

Primers/probe set for viral RNA quantification.

Assays	Primers/probe sets
China CDC recommended N gene reference assay	Forward primer: 5′-GGGGAACTTCTCCTGCTAGAAT-3′ Reverse primer: 5′-CAGACATTTTGCTCTCAAGCTG-3′ Probe: 5′-TTGCTGCTGCTTGACAGATT-3′
Modified N gene assay	Forward primer: 5′-AACGAACTTCTCCTGCTAGAAT-3′ Reverse primer: 5′- CAGACATTTTGCTCTCAAGCTG-3′ Probe: 5′-TTGCTGCTGCTTGACAGATT-3′

Detailed information on the sample panel is shown in Table 2. The test panel included ten samples containing serial 5-fold dilutions of inactivated SARS-CoV-2-positive supernatants of the Omicron BA.1 and BA.5 strains (2 × 10⁷ copies/mL, 4 × 10⁶ copies/mL and 8 × 10⁵ copies/mL) and negative samples. Positive samples with 8 × 10⁵ copies/mL and negative samples were set in duplicate to evaluate the repeatability of participants’ procedure. The samples were diluted with phosphate buffer solution (PBS, VivaCell, Shanghai, China) and immediately lyophilized after being dispensed in 0.2-mL aliquots (Supplementary Materials). Prior to distribution, suitability tests were carried out by detecting lyophilized samples with five NMPA-approved antigen tests to confirm that the samples were suitable for the EQA, including tests from BGI Biotech Co., Ltd. (fluorescence immunochromatography), Hangzhou ACON Biotech Co., Ltd. (latex chromatography), Hangzhou AllTest Biotech Co., Ltd. (latex chromatography), Shenzhen YHLO Biotech Co. Ltd. (colloidal gold), and Nanjing Vazyme Biotech Co., Ltd. (colloidal gold) (Figure 1B; Supplementary Materials Figure S1). For homogeneity tests, ten positive samples of each coded number were randomly selected and tested with the Hangzhou AllTest Biotech Co., Ltd., which was proven to have high sensitivity in a previous study (Figure 1C). For stability experiments, samples coded with 2023A03 (Omicron BA.1) and 2023A10 (Omicron BA.5) with 8 × 10⁵ copies/mL were tested using the same kit (Figure 1D). Samples were incubated at 37 °C, room temperature, 4 °C and −20 °C for 1, 3, 5, and 7 days, respectively. The results showed that samples incubated at 37 °C degraded from the third day, while samples remained stable for at least one week at room temperature, 4 °C and −20 °C (Supplementary Materials Figure S2).

Table 2:

SARS-CoV-2 antigen testing EQA scheme – composition of panel and overall results.

Sample content	Sample number	Sample category	Viral RNA concentration (ddPCR copies/mL)a	Percent correct qualitative results, %
				PPA	NPA
Omicron BA.1 variant	2023A02	Validating	2 × 107	100.00	–
	2023A04	Validating	4 × 106	91.27	–
	2023A03	Educational	8 × 105	24.87	–
	2023A07	Educational	8 × 105	22.75	–
Omicron BA.5 variant	2023A06	Validating	2 × 107	99.47	–
	2023A08	Validating	4 × 106	93.12	–
	2023A01	Educational	8 × 105	25.66	–
	2023A10	Educational	8 × 105	27.78	–
PBS	2023A05	Validating	–	–	99.74
	2023A09	Validating	–	–	99.74
Overall EQA performance
OPA, % PPA, % NPA, %				68.44
				60.62
				99.74

1. OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. ^aViral RNA concentrations were quantified with ddPCR N gene modified assay.

Results

Percent agreements of overall results

The qualitative results were analyzed in combination with the original images submitted by the participants, and both were consistent. The color depths of the T-lines by the naked eye matched the virus loads, which became lighter as the viral loads decreased. The performances were found to be competent in 90.56 % (307/339) of the participants and 90.21 % (341/378) of the datasets. The accuracy of laboratory test results was evaluated by calculating the overall percent agreement (OPA), positive percent agreement (PPA) and negative percent agreement (NPA), which were expressed as the ratios of correct results and total results reported for all, positive and negative samples, respectively. For all ten EQA samples, the overall OPA, PPA and NPA were 68.44 % (2,587/3,780), 60.62 % (1,833/3,024) and 99.74 % (754/756), respectively (Table 2). For positive samples with various concentrations, the PPA was greater than 99 % for samples of 2 × 10⁷ copies/mL and 92.20 % (697/756) for samples of 4 × 10⁶ copies/mL. However, the PPA was significantly lower at 25.26 % (382/1,512) for educational samples compared to the two other samples with higher concentrations (p<0.0001) (Figure 2A). Out of the 378 datasets, 21.96 % (83/378) reported positive results for all four educational samples, while 69.05 % (261/378) failed to detect any of the educational samples. The remaining datasets were able to detect one to three out of the four educational samples.

Figure 2:

The results of panel samples submitted by the participants. Overall results (A), results of fluorescence immunochromatography (B), latex chromatography (C) and colloidal gold (D) methods.

Percent agreements of results for positive samples

Detection methods

In this EQA scheme, one fluorescence immunochromatography, seven latex chromatography and 30 colloidal gold methods were used by the participants. Table 3 provides an overview of the datasets produced by the three methods. For all EQA samples, the OPAs were 92 % (46/50), 83.02 % (440/530) and 65.66 % (2,101/3,200) for the fluorescence immunochromatography, latex chromatography and colloidal gold methods, respectively. Among validating samples, 100 % (5/5), 92.45 % (49/53) and 89.69 % (287/320) of datasets using fluorescence immunochromatography, latex chromatography and colloidal gold methods reported correct results, which were identified as competent. All three methods demonstrated 100 % PPAs for samples of 2 × 10⁷ copies/mL and PPAs exceeding 90 % for samples of 4 × 10⁶ copies/mL. However, for the results of a viral load of 8 × 10⁵ copies/mL, the PPAs varied considerably, and the performance of colloidal gold (18.52 %, 237/1,280) was far behind those of fluorescence immunochromatography (60.85 %, 129/212) and latex chromatography (80 %, 16/20) (p<0.0001). The results of the colloidal gold method had a significantly higher rate of false-negatives in comparison with the other two methods (p<0.0001) (Figure 2B–D).

Table 3:

SARS-CoV-2 antigen testing EQA scheme – performance of different antigen testing methods.

Samples	Method performance, % for:
	Fluorescence immunochromatography (n=5)	Latex chromatography (n=53)	Colloidal gold (n=320)
Positive samplesa	Percent correct qualitative results (PPA, %)
2023A02 Omicron BA.1 variant 2 × 107 copies/mL	100	100	100
2023A04 Omicron BA.1 variant 4 × 106 copies/mL	100	94.34	90.63
2023A03 Omicron BA.1 variant 8 × 105 copies/mL	80	60.38	18.13
2023A07 Omicron BA.1 variant 8 × 105 copies/mL	80	56.60	16.25
2023A06 Omicron BA.5 variant 2 × 107 copies/mL	100	96.23	100
2023A08 Omicron BA.5 variant 4 × 106 copies/mL	100	98.11	92.19
2023A01 Omicron BA.5 variant 8 × 105 copies/mL	80	58.49	19.38
2023A10 Omicron BA.5 variant 8 × 105 copies/mL	80	67.92	20.31
Negative samples	Percent correct qualitative results (NPA, %)
2023A05 PBS	100	98.11	100
2023A09 PBS	100	100	99.69
Overall EQA performance
OPA, %	92	83.02	65.66
PPA, %	90	79.01	57.11
NPA, %	100	99.06	99.84

1. OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. ^aViral RNA concentrations quantified with ddPCR N gene modified assay.

Detection assays

For most antigen assays used in the scheme, the number of reported datasets was too limited to provide strong evidence for the detection performance of individual assays. Therefore, 11 antigen assays employed by >10 laboratories herein were analyzed (Table 4). Assays manufactured by ACON, YHLO, Kehua and Kanghua demonstrated PPAs of 92.42 % (244/264), 73.21 % (205/280), 68.75 % (77/112) and 64.47 % (98/152) for all positive samples, respectively, whereas the other seven assays showed PPAs lower than 60 % (Figure 3). For samples with 2 × 10⁷ copies/mL, the PPAs of the 11 assays were all >95 %. However, for 4 × 10⁶ and 8 × 10⁵ copies/mL, the rates were variable. For the concentration of 4 × 10⁶ copies/mL, Vazyme and Wantai had PPA values inferior to 80 %, while the remaining assays all displayed PPA values greater than 80 %. Moreover, Vazyme, Wantai, Orient Gene and Bioscience were essentially unable to detect educational samples, indicating that the LODs of these four test kits were beyond 8 × 10⁵ copies/mL. Notably, only ACON showed a higher positive rate (>80 %) when detecting educational samples, resulting in a prominently increased OPA and PPA compared to the other ten antigen assays.

Table 4:

SARS-CoV-2 antigen testing EQA scheme – performance of antigen testing assays employed by >10 laboratories.

Items	Assay performance, % for:
	Zybio (n=71)	YHLO (n=35)	ACON (n=33)	Vazyme (n=28)	Wondfo (n=24)	Kanghua (n=19)	Wantai (n=16)	Kehua (n=14)	EasyDiagnosis (n=14)	Orient gene (n=11)	Bioscience (n=11)
LOD (TCID50/mL)	70	250	160	50	850	64	137	500	500	Not available	850
LOD (copies/mL)	7 × 105	2.5 × 106	1.6 × 106	5 × 105	8.5 × 106	6.4 × 105	1.37 × 106	5 × 106	5 × 106	Not available	8.5 × 106
Positive samples a	Percent correct qualitative results (PPA, %)
2023A02 Omicron BA.1 variant 2 × 107 copies/mL	100	100	100	100	100	100	100	100	100	100	100
2023A04 Omicron BA.1 variant 4 × 106 copies/mL	98.59	97.14	100	67.86	87.5	100	50	100	100	81.82	100
2023A03 Omicron BA.1 variant 8 × 105 copies/mL	14.08	45.71	84.85	0	12.5	31.58	0	42.86	14.29	0	0
2023A07 Omicron BA.1 variant 8 × 105 copies/mL	12.68	45.71	81.82	0	12.5	26.32	0	28.57	14.29	0	0
2023A06 Omicron BA.5 variant 2 × 107 copies/mL	100	100	96.97	100	100	100	100	100	100	100	100
2023A08 Omicron BA.5 variant 4 × 106 copies/mL	97.18	97.14	100	75	91.67	100	56.25	100	100	100	100
2023A01 Omicron BA.5 variant 8 × 105 copies/mL	14.08	48.57	84.85	0	16.67	26.32	0	42.86	28.57	0	9.09
2023A10 Omicron BA.5 variant 8 × 105 copies/mL	18.31	51.43	90.91	0	12.5	31.58	0	35.71	14.29	0	0
Negative samples	Percent correct qualitative results (NPA, %)
2023A05 PBS	100	100	96.97	100	100	100	100	100	100	100	100
2023A09 PBS	100	100	100	100	100	100	93.75	100	100	100	100
Overall EQA performance
OPA, %	65.49	78.57	93.64	54.29	63.33	71.58	50	75	67.14	58.18	60.91
PPA, %	56.87	73.21	92.42	42.86	54.17	64.47	38.28	68.75	58.93	47.73	51.14
NPA, %	100	100	98.48	100	100	100	96.88	100	100	100	100

1. LOD, limit of detection; TCID₅₀, tissue culture infectious dose required to infect 50 % of the cells; OPA, overall percent agreement; PPA, positive percent agreement; NPA, negative percent agreement. Zybio, Zybio, Inc.(colloidal gold); YHLO, Shenzhen YHLO Biotech Co. Ltd. (colloidal gold); ACON, Hangzhou ACON Biotech Co., Ltd. (latex chromatography); Vazyme, Nanjing Vazyme Biotech Co., Ltd. (colloidal gold); Wondfo, Guangzhou Wondfo Biotech Co. Ltd. (colloidal gold); Kanghua, Shandong Kanghua Biotech Co., Ltd. (colloidal gold); Wantai, Beijing Wantai Biological Pharmacy Enterprise Co., Ltd. (colloidal gold); Kehua, Shanghai Kehua Bio-engineering Co., Ltd. (colloidal gold); EasyDiagnosis, Wuhan EasyDiagnosis Biomedicine Co., Ltd. (colloidal gold); Orient Gene, Zhejiang Orient Gene Biotech Co., Ltd. (colloidal gold); Bioscience, Tianjin Bioscience Bio-technology Co., Ltd. (colloidal gold). ^aViral RNA concentrations quantified with ddPCR N gene modified assay.

Figure 3:

Datasets for antigen testing assays employed by >10 laboratories.

BA.1 and BA.5 variants

For the Omicron BA.1 and BA.5 variants, the OPAs were 59.72 % (903/1,512) and 61.51 % (930/1,512), respectively, and no significant difference was observed (p=0.315). For 2 × 10⁷ copies/mL and 4 × 10⁶ copies/mL, the PPAs of BA.1 and BA.5 were all >90 %. However, the PPAs of the BA.1 and BA.5 variants were 23.81 % (180/756) and 26.72 % (202/756) for 8 × 10⁵ copies/mL, respectively. Similar PPAs were shown for the two variants at the three concentrations.

Discussion

In this study, an EQA program was implemented by the NCCL to assess the testing capabilities of laboratories using antigen detection assays that have been approved by the NMPA. Regarding the intrinsic characteristic of lower analytical sensitivity for antigen detection (10⁵–10⁶ copies/mL) compared to rRT-PCR [21, 22], inactivated cell culture supernatants of the SARS-CoV-2 Omicron BA.1 and BA.5 strains with high (2 × 10⁷ copies/mL), medium (4 × 10⁶ copies/mL), and low (8 × 10⁵ copies/mL) concentrations were prepared as EQA-positive samples. Validating samples were used to categorize the detection performance, and educational samples were set in duplicate to enable a more comprehensive sensitivity assessment and evaluate the repeatability of the participants’ procedure.

In the EQA scheme, 378 effective results were collected from 339 laboratories in China. According to the scoring principals, 90.56 % (307/339) of the participants and 90.21 % (341/378) of the datasets were identified as competent. As the sample concentrations decreased from 2 × 10⁷ copies/mL to 8 × 10⁵ copies/mL, the PPAs of the datasets decreased from 99.74 % (754/756) to 25.26 % (382/1,512). This trend indicated that the LODs for most assays were beyond 8 × 10⁵ copies/mL, making it challenging to detect weak positive samples. Next, we will delve into the discrepancies in results among three detection methods, different detection kits, and different laboratories.

The three detection methods exhibited significant disparities in terms of usage rates and PPAs. Colloidal gold was the most frequently used (84.66 %, 320/378) but showed the lowest PPA (57.11 %, 1,462/2,560). In contrast, fluorescence immunochromatography was the least popular (1.32 %, 5/378) but had the highest PPA of 90 % (36/40). Latex chromatography showed an intermediate application rate (14.02 %, 53/378) and PPA (79.01 %, 335/424). The limited adoption of fluorescence immunochromatography may be attributed to the need for a compatible reader, which adds complexity compared to visual interpretation and incurs additional costs for technical maintenance [19, 23]. Despite its lower prevalence, the integration of the fluorescence immunolabeling technique with fluorescence photometric signal detection has significantly enhanced the analytical sensitivity by 1–2 orders of magnitude, surpassing the sensitivity of the other two methods [24]. Discrepancies in results among the three detection methods were evident for educational samples, where fluorescence immunochromatography demonstrated a noteworthy 80 % (16/20) PPA, while colloidal gold only displayed a PPA of 18.52 % (237/1,280), which was consistent with the findings of our previous study [25]. However, it is important to emphasize that the conclusions drawn regarding the fluorescence immunochromatography method may be influenced by the limited dataset generated from only five laboratories, which could have introduced statistical bias.

To minimize analysis bias, we focused on 11 antigen assays performed daily in more than 10 laboratories, as most antigen assays used in this study were adopted by only a few laboratories. For all positive samples, ACON stood out as the exceptional reagent kit with a PPA of 92.42 %. Additionally, YHLO, Kehua, and Kanghua exhibited relatively strong PPAs of over 60 %. The 11 assays reported almost 100 % PPAs for samples with 2 × 10⁷ copies/mL; however, Vazyme and Wantai showed obviously decreased PPAs for samples with 4 × 10⁶ copies/mL. Notably, Vazyme, Wantai, Orient Gene and Bioscience were essentially unable to detect the educational samples at all, while ACON showed PPAs greater than 80 %, indicating that ACON was the most sensitive of the 11 assays. According to the study conducted by Sender R et al. [26], the claimed LODs of the 11 test kits varied between 5 × 10⁵ and 8.5 × 10⁶ copies/mL after conversion from TCID₅₀ (tissue culture infectious dose required to infect 50 % of the cells). Among these 11 kits, Zybio, Vazyme, Kanghua, and Wantai produced false-negative results when the virus loads were at or above their specified LODs, while the remaining seven tests showed detection performances that matched their claimed LODs.

In the EQA scheme, discordant results were observed in datasets from laboratories using the same kits, indicating inconsistency between laboratories, which might be caused by the analytical sensitivity of assays, operational factors, as well as between-batch differences. For educational samples with 8 × 10⁵ copies/mL, eight kits showed laboratory inconsistency to varying degrees. Among them, YHLO exhibited the highest variation, with a PPA of 47.86 % (67/140), followed by Kehua and Kanghua, with PPAs of more than 20 %. The inter-laboratory inconsistency may be due to the fact that 8 × 10⁵ copies/mL was below their LODs, and it was expected to see a significant decrease in positive results for educational samples. For samples with 4 × 10⁶ copies/mL, Wantai and Vazyme showed higher laboratory inconsistency with PPAs of 53.13 % (17/32) and 71.43 % (40/56), respectively. Because the LODs for the two kits were below 4 × 10⁶ copies/mL, the inconsistency between laboratories may be caused by operational factors or between-batch differences. To minimize batch-to-batch variability, routine reagent batch validation is recommended using negative, weak positive and medium positive samples across different time points and operating conditions, utilizing different reagent lots and involving different operators.

In terms of negative samples coded with 2023A05 and 2023A09 composed of PBS, all 11 kits showed NPAs of 100 % except for ACON and Wantai. By analysis, the reason for false-positive results by ACON was that one laboratory accidentally reversed the results of samples coded as 2023A05 (PBS) and 2023A06 (Omicron BA.5 with 2 × 10⁷ copies/mL). For Wantai, considering the relatively poor detection performance, which had an approximately 50 % PPA for samples of 4 × 10⁶ copies/mL and could not detect educational samples at all, positive results were unexpectedly reported for negative samples. After tracing the original images and excluding the possibility of mistakes in completing the questionnaire, we speculated that the false-positive results were possibly due to poor quality of the kit batch, occasional operational errors or incorrect result interpretation.

Based on the aforementioned statements, the analytical sensitivities of the methods and reagent kits were the major cause of performance differences [18, 19]. The intrinsic properties of assays, such as the antigens targeted, the antibodies coated, and the method of coloration, are the main determinants for analytical sensitivity [27]. Research efforts are being dedicated to targeting antigens other than the N protein, such as the spike protein [28, 29] or a combination of multiple proteins [30, 31], in order to enhance detection sensitivity. The density, affinity, and specific epitopes targeted by coated antibodies also play a crucial role in determining the detection sensitivity. Additionally, fluorescent labeling combined with instrumentation significantly improves the sensitivity of the analysis compared to the less objective naked-eye interpretation of colloidal gold and latex methods [24]. Considering the considerable risk of false-negative rates, it is important to rectify the public perception that negative results definitively rule out infections.

In this study, the inclusion of educational samples was highly valuable, as performance differences were mainly displayed by educational samples with low concentrations among detection methods, detection kits, and laboratories. EQA conducted here is the initial step into post-market surveillance. Regular participation in well-designed EQAs can alert technicians to pay attention to proper sample handling and accurate result interpretation. However, relying solely on data from one EQA and two types of variants would provide limited information. Therefore, follow-up research encompassing a wider range of variants is required to gather dynamic data and provide a holistic view of antigen detection kits in the post-pandemic era.