A look at the precision, sensitivity and specificity of SARS-CoV-2 RT-PCR assays through a dedicated external quality assessment round

To the Editor,

In an external quality assessment (EQA) round for SARS-CoV-2 virus genome detection in early 2021, we found first indications of the precision of individual SARS-CoV-2 RT-PCR assays in the linearity of the C_t values obtained for three samples from a dilution series (1:1, 1:10, 1:100) [1]. To further investigate the reproducibility of the C_t values and thus the precision of the participating assays, two identical samples were employed in the latest round of this EQA scheme of the Austrian Association for Quality Assurance and Standardization (ÖQUASTA) and the Center for Virology of the Medical University Vienna.

On August 16, 2021, a panel of six samples was sent to participant laboratories. Production, testing, shipping of samples and instructions for participant laboratories followed the same procedures as in the previous rounds [2]. The panel consisted of two SARS-CoV-2 negative and four positive samples, of which two were identical (S2 and S6; B.1.617.2, Delta variant, C_t∼30.6). The third positive sample (S3) was B.1.1.7, Alpha variant, C_t∼32.2, and the fourth (S4) was B.1.617.2, Delta variant, C_t∼32.8. One negative sample (S5) consisted of virus genome free NaCl, the second SARS-CoV-2 negative sample (S1) contained a recent clinical sample of human betacoronavirus OC43 (HCoV-OC43, C_t∼29.9) to assess potential cross-reactivity of assays. All samples were spiked with human cells to give positive results for housekeeping genes for internal control in RT-PCR. A total of 114 laboratories reported results before August 31, 2021. They employed a total of 139 assays, including 53 different, for detection of SARS-CoV-2 genomic RNA.

Among the two identical samples, 136/139 (98%) assays reported S2 as positive, 1/139 (1%) as negative, and 2/139 (1%) as not determinable; S6 was reported positive by 134/139 (97%) assays, negative by 2/139 (1%), and not determinable by 3/129 (2%). S3 (C_t∼32.2) was reported positive by 138/139 (99%) assays and negative by one (1%). S4 (C_t∼32.8) was reported positive by 131/139 (94%), negative by four (3%), and not determinable by four (3%) assays. Among them was one assay that reported all four positive samples as negative, the other four false negative results were reported from four different assays. No false positive results were reported for S1 and S5, however 7/139 (5%) assays found S1 and S5 not determinable (data not shown). The rates of false negative results depending on the C_t values of samples correspond to earlier rounds of this EQA scheme [1], [2], [3]. Of the total of 109 different participating assays, 70 did not show a single false negative result in any of the four rounds. The rate of false negative in all reported results ranged from 0/105 to 4/4 (Figure 1, Supplementary Material). We did not observe any difference in the rates of false negative results between the two variants of the virus used in this round. While the B.1.1.7, Alpha variant, sample (S3) was reported negative by one (1%) assay, the B.1.617.2, Delta variant, samples (S2, S4, S6) were reported negative by one (1%), four (3%), and two (1%) assays. These small differences in the rates of false negative results appear to be more due to the different C_t values of the samples, namely about 1% for samples with C_t∼32.2 (S3) and C_t∼30.6 (S2=S6), and 3% for the sample with the highest C_t∼32.8 (S4).

Figure 1:

Percentages of false negative results increased with increasing C_t values of 15 positive samples in four rounds of an EQA scheme to detect SARS-CoV-2 virus genome.

Percentages are labeled beside plotted diamonds if they deviate from 0%. Data came from four rounds of an EQA scheme (blue, May 2020; gold, October 2020; green, February 2021; and red, August 2021).

For the paired samples (C_t values for both samples S2 and S6), 123 assays were reported targeting one (n=47), two (n=70), three (n=5) or four (n=1) gene targets. Mean C_t value reported for the identical samples S2 and S6 was 33.2 (range 23.6–39.4; SD 3.2); and sample-specific C_t values were 33.5 (range 22.8–40.8; SD 3.2) for S2 and 32.7 (range 23.1–43.8; SD 3.3) for S6. The differences in C_t values reported for each individual gene target were calculated per participant as [S6 C_t value − S2 C_t value] and ranged from −5.3 to +6.5 cycles. C_t value discrepancy was not associated with a specific RT-PCR target region. The mean values of the differences between both results for all gene targets reported by individual assays ranged from −4.7 to +4.7 cycles (Figure 2). Seven assays reported a C_t value for one of the target genes they used for only one of samples S2 and S6 and none for the second. The reported C_t values were 35.7 mean, in detail 32.6, 35.0, 37.0, 37.5 for S2, and 34.7, 36.6, 36.7 for S6. These single results are not included in the data but shown and highlighted in Figure 2.

Figure 2:

Differences of C_t values reported for two identical samples of SARS-CoV-2.

The top box-and-whiskers plot shows global median and quartiles of individual C_t values for two paired samples. The bottom box-and-whiskers plot shows median and quartiles of the differences in C_t values comparing the same laboratory. The dots are colored according to three groups based on the differences in C_t values – identical results (red), similar results (green) and deviating results (dark blue) – and the means of each group are displayed as colored vertical lines on the top figure. Seven results that reported a C_t value for only one of the paired samples are shown in light blue (dots and vertical line). The assays that fall into each of the three groups are listed in the table.

To better visualize the differences between the paired samples and detect assay-specific deviations, we grouped the results according to the differences in C_t values: (i) identical results (difference <1 cycle, 77/123, 63%); (ii) similar results (difference >1 and ≤3 cycles, 42/123, 34%); and (iii) deviating results (difference >3 cycles, 4/123, 3%). Means and ranges of C_t values in each group were 32.9 (22.8–40.6) for group 1, 33.8 (24.1–40.8) for group 2, and 36.8 (31.7–43.8) for group 3. Participants with identical results (group 1) reported using 37 different assays, those with similar results (group 2) reported 26 assays, and those with deviating results (group 3) reported four assays. Ten assays are represented in both groups 1 and 2, three are represented in groups 1 and 3, and one assay is represented in all three groups (Figure 2).

The results presented here indicate good reproducibility and thus high precision of the C_t values of most of the RT-PCR assays represented in this EQA round. As we previously noted, the detection rate of SARS-CoV-2 decreases with increasing C_t values [1], [2], [3]. Additionally, we observed that higher C_t values are more likely to be associated with greater differences between the two measurement results and thus with lower precision. By including paired samples and grouping the results based on the deviation in C_t values, we noted that only a few of the assays were categorized in more than one performance group (e.g., giving both identical results and deviating results). From this, we conclude that overall precision seems to rely on the performance of the individual assays and less on their operation. The fact that no false positive results were reported in this round confirms our earlier findings of low rates of false positive results in SARS-CoV-2 viral genome detection [3]. The high specificity of the test systems included in this EQA round was indicated by no false positive results obtained in the sample with the human betacoronavirus OC-43.

In summary, the performance of most SARS-CoV-2 virus genome detection assays in this EQA round is good, however some appear to tend to report comparatively high C_t values and thus less accurate and consequently false negative results. The use of such assays should be subject to risk analysis and assessment by their operators. The results presented here underline once again the great importance of EQA schemes for monitoring the performance of laboratories as well as of test systems. This applies in particular, if the infectious agent to be detected is a rapidly changing virus, the newly emerging variants of which have different clinical significance and a manifold of different test systems are in use.