Evaluation of seven commercial SARS-CoV-2 RNA detection kits based on real-time polymerase chain reaction (PCR) in China

To the Editor,

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a new, rapidly spreading human beta coronavirus. It was first identified in Wuhan and caused a disease named Coronavirus Disease 2019 (COVID-19) [1]. COVID-19 patients may commonly present with fever, cough, myalgia, or fatigue. The severe cases progress rapidly to acute respiratory distress syndrome, shock, organ dysfunction, and even death [2]. According to the latest data, the WHO confirmed a total of 80,565 cases infected with SARS-CoV-2 and 3,015 deaths in China on March 05, 2020. And the virus has infected 95,333 individuals globally and spread to at least 79 countries, such as Korea, Italy, and Iran [3]. The entire viral genome is more than 29,000 nt in length and has 12 protein coding regions (1ab, S, 3, E, M, 7, 8, 9, 10b, N, 13, 14) [4], [5]. Based on the Diagnosis and Treatment Program of COVID-19 (seventh trial version) formulated by the National Health Commission of China, real-time polymerase chain reaction (PCR) assays have been regarded as the reference standard to make a definitive diagnosis of COVID-19 (see Supplementary Table 1). Therefore, various commercial kits were available in China during the outbreak of disease. Most of them targeted the viral genes so far include the ORF1ab, N, and E genes, as in the world health organization (WHO) Laboratory guidance document (https://www.who.int/emergencies/diseases/novel-coronavirus-2019/technical-guidance/laboratory-guidance). Notably, different countries have distinct target genes for SARS-CoV-2 detection. The China centers for disease control and prevention (CDC) requires that the ORF1ab and N genes should be detected simultaneously in the positive samples (see Supplementary Table 2). At the time of submission of this paper, in consideration of the essential time diagnosis of COVID-19, various commercial kits were authorized for emergency use in the clinical laboratories. However, they lack validation for diagnostic use in the clinic. Hence, we selected seven representative commercial kits to evaluate their performance in SARS-CoV-2 RNA detection.

A total of seven domestic diagnosis kits from seven biopharmaceutical companies were included in our study: kit A (BGI Biotech Co., Ltd, Wuhan, China), kit B (Outdo Biotech Co., Ltd, Shanghai, China), kit C (Sansure Biotech Inc., Changsha, China), kit D (Perkin Elmer Medical Diagnostic Products, Co., Ltd, Shanghai, China), kit E (Daan Gene Co., Ltd. Of Sun Yat-Sen University, Guangzhou, China), kit F (Jiangsu Bioperfectus Technologies Co., Ltd, Taizhou, China), and kit G (Fosun Long March Medical Science Co., Ltd, Shanghai, China). Among these, all of them have been used in the clinic. But only the kit A, kit C, and kit E were approved by the National Medical Products Administration (NMPA). In different regions of China, the clinical laboratories selected assays from diverse manufacturers. For instance, the kit C and kit F had a higher frequency of clinical use in Hunan Province. Additionally, the kit C has even been applied in Japan. 42 archived SARS-CoV-2-positive samples (34 throat swabs and eight fecal samples) and 200 randomly selected SARS-CoV-2-negative samples were collected from the Second Xiangya Hospital. The reference results not only consider the clinical diagnosis of the case but also consider the laboratory nucleic acid testing results (see Supplementary Tables 1, 2). The interpretation criteria for SARS-CoV-2-positive samples included: 1) The result in both SARS-CoV-2 ORF1ab and N gene was positive (two commercial kits (kit C and kit F) were performed to validate the detection results); 2) Epidemiologic history, clinical situation, laboratory tests, and imaging examinations accord with COVID-19 diagnostic criteria. Total nucleic acid extractions were performed on the NP968-S Nucleic Acid Extraction System (Tianlong Science & Technology Co., Ltd, Xi’an, China) with paramagnetic particle method, using the Virus DNA/RNA Isolation Kit (Tianlong Science & Technology Co., Ltd, Xi’an, China) and the default instrument settings. The residual specimens and nucleic extracts were retained and restored immediately after initial clinical testing at −80 °C. These samples were thawed once before testing. Then, we diluted the unfrozen nucleic acid according to the instructions of the selected kits and actual sample volume. Seven kits tested samples in parallel and according to the manufacturer’s introductions. RNA extraction procedures have been qualified and validated for quality and purity before trial. All of the samples were conducted using strict quality control and quality assurance procedures. The researcher performing testing was blinded to the original SARS-CoV-2 screening outcomes of these 242 specimens.

Table 1 summarizes the specifications of SARS-CoV-2 commercial kits included in this study. Most of them were suitable for respiratory specimen’s detection. The kit D could also be applied for blood samples. The kit D required the highest volume of the nucleic acid, and the RNA volume was only 5 μL for kit E and kit F. The discrepancy of template volume may significantly contribute to the clinical performance of detection methods. The target genes determined by seven kits were not the same. All the kits were designed to target the virus ORF1ab gene, but kit F and kit G could detect three genes simultaneously. Based on the different results interpretation criteria of the seven kits, we categorized them into three groups: group Ⅰ: SARS-CoV-2-positive: ORF1ab gene-positive or N gene-positive in samples; group Ⅱ: ORF1ab gene-positive and N gene-positive in samples; group Ⅲ: E gene is included in the kit. The kit D and E kits in group Ⅱ were consistent with the results interpretation standard of China CDC.

Subsequently, we evaluated the diagnostic performances of the seven kits in the clinical samples (Table 2). Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Youden index were calculated. Cohen’s Kappa coefficient, McNemar test, chi-square (χ²) values were carried out using SPSS software (version 26.0). As shown in Table 2, the diagnostic specificity of seven kits was all 100%. The concordance of kit A, kit B, kit D, and kit F and “reference result” was robust (Cohen’s Kappa coefficient>0.900, p<0.001). The diagnosis results of kit C, kit E, and kit G and “reference result” were not identical (p<0.05). The clinical sensitivity for three assays in group Ⅰ was 90.48, 92.86, and 83.33%, respectively (see Supplementary Table 3). The difference between them is not statistically significant. As for group Ⅱ and group Ⅲ, kit D and kit F demonstrated superior sensitivity (p<0.05).

We also calculated the detection rate on different target genes (ORF1ab, N, and E) of these assays (see Supplementary Table 4). It has to be mentioned that the kit B used only one fluorescent channel (FAM) to detect two target genes simultaneously, so we couldn’t differentiate which gene of the SARS-CoV-2 was discovered in the positive samples. Apart from the kit B, we observed that the kit D had the highest positive detection rate of the virus ORF1ab gene (92.86%). Though the kit C and kit G were able to detect the N gene of SARS-CoV-2, the detection rates of the ORF1ab gene for them were only 23.81 and 30.95%, respectively. The kit D and F had the same positive detection rate of the N gene (90.48%). The detection rate of the E gene for the group Ⅲ kit was 66.67 and 85.71%, respectively.

To sum up, we summarized and compared the detailed information of seven SARS-CoV-2 RNA detection kits in the present study. The assessment of commercial kits in the clinical application involves many factors such as specimen types, target genes, the limit of detection, interpretation criteria, etc., As a result of time to market, not all kits have satisfied with China CDC standards or been approved by NMPA. So far, more than 100 pharmaceutical companies have successfully developed SARS-CoV-2 RNA detection reagents in China. We discovered that all seven kits have excellent specificity, but there were differences in their clinical sensitivity. If the clinical laboratories need to perform virus single gene detection for COVID-19 screening, the clinical performances of group Ⅰ kits were similar. If selecting a kit that meets the China CDC standard is necessary, the kit D was better. Considering the E gene detection, the kit F had higher clinical sensitivity. Otherwise, the detection ability of the ORF1ab gene for some kits was weak. We speculated that this might be related to the primer design methods for different kits. Interestingly, we proposed that though respiratory samples have the greatest yield, the virus can be detected in other specimens, such as stool. Recently, the Chinese researches pointed out that they successfully isolated the SARS-CoV-2 in the urine. It remains to verify whether the kits in our study could also be used in urine samples detection.

It is worth noting that our study just made a simple comparison of the detection capacity of each reagent in a specific batch. The tested samples had been preserved at −80 °C refrigerator in days. Consequently, the results may be interfered with by freshness of samples, sample preservation solution, sample dilution ration, operation process, and other factors. Though efforts were made to design kits to conserved viral genomic regions, variability resulting in mismatches between the primers and probes and the target sequences can result in diminished assay performance. Therefore, our study does not represent the overall detection performance of these kits. Another limitation of our study was the inability to test seven kits with a certified reference material due to a lack of availability of such material at the time of this study. The commutability of the results requires accredited reference materials that can be used to determine performance across different platforms. In general, the clinical laboratory should select the most appropriate kit according to the locally epidemic trend of disease, the actual situation of clinical samples, the purpose of detection, and laboratory conditions.