The versatility of external quality assessment for the surveillance of laboratory and in vitro diagnostic performance: SARS-CoV-2 viral genome detection in Austria

Introduction

Regulation (EU) 2017/746 on In Vitro Diagnostic Medical Devices (IVDR) requires performance of in vitro diagnostics (IVD) to be monitored by external quality assessment (EQA) schemes [1]. While the contribution of EQA schemes to post-market surveillance is not yet fully developed and standardized, EQA for medical laboratories has a tradition going back decades [2]. In such schemes, EQA providers distribute samples with known but undisclosed properties or content. Participating laboratories analyze these samples and report results to the EQA provider. Reported results are compared with assigned targets and scored as either matching (correct) or non-matching (false). Participants learn in individual reports how comparable their results are with those of their peers and gain additional educational effects from the general report [3]. Therefore, their motivation to participate in EQA is to have results assessed by an independent third party and to compare them with those obtained with both identical and alternative analytical systems. However, mere assessment of performance does not fully exploit the real potential of EQA schemes. The underutilized potential of such schemes lies in the aggregation of results, which provides an overview of the analytical performance and competencies of laboratories and test systems. This data is therefore useful for both post-market surveillance and epidemiological purposes.

Molecular diagnostics for SARS-CoV-2 infection from nasopharyngeal, nasal or oro-pharyngeal swabs were available in early 2020 and were quickly adopted by laboratories around the world. SARS-CoV-2-specific real time reverse transcription (RTq-) PCR is currently the gold standard for confirmation or exclusion of infection, monitoring COVID-19 therapy and infectious and epidemiologic follow-up testing [4, 5]. According to manufacturers’ specifications, the reliability of such test results should be high. By and large, previously reported EQA results have confirmed that the various participating systems perform well for SARS-CoV-2 viral genome detection [6], [7], [8], [9], [10], [11]. However, we believe an EQA scheme offers an opportunity to further probe the specific skills and competencies of participants by including samples with specific properties; and to our knowledge this approach has not yet been reported.

During a pandemic, the preparedness of laboratories to face arising challenges is of great interest. For example, it has been reported that several mutations in the SARS-CoV-2 genome are associated with reduced sensitivity of routine RT-qPCR systems [12], [13], [14]. To reduce the risk of positive samples escaping detection, the International Federation for Clinical Chemistry and Laboratory Medicine (IFCC) recommended the use of at least two SARS-CoV-2-specific gene targets in test systems [15]. An EQA can check to what extent this recommendation has been implemented. Another interesting point that EQA schemes can clarify is the preparedness of laboratories to interpret and report divergent or unexpected results (e.g., target gene “dropout”). This may include, for example, further examinations of putative SARS-CoV-2-positive samples for possible mutations or including amplification of human RNA as an internal control of sample quality [16].

The Austrian Association for Quality Assurance and Standardization (ÖQUASTA) is the national EQA provider in Austria and serves most of the relevant laboratories in the country. The EQA scheme “SARS-CoV-2 virus genome detection” was established in cooperation with the National Reference Laboratory for Respiratory Viruses at the Center for Virology of the Medical University Vienna, Austria, in early 2020, and two rounds have since been performed. For the third round (the first round in 2021), samples were selected so that the measurement results provided information in addition to detection of viral nucleic acid. Herein, we evaluated the current nationwide performance in SARS-CoV-2 genome detection, and we assess specific properties of laboratories and test systems using a specifically designed panel of sample materials.

Materials and methods

Results

Analytical performance

A total of 124 out of 134 test systems (93%) reported correct results for all five positive and both negative samples (Table 2). A total of 1,006 individual C _t values were reported from 122 test systems targeting the following viral gene targets: E (250 C _t values); N (337); S (68); RdRP (112); ORF1ab (105); or “other” undisclosed gene targets (128), which may have included a single result obtained from two or more gene targets. Summary statistics of C _t values were calculated for individual targets and “other” target genes (Table 2 and Figure 1).

Figure 1:

SARS-CoV-2-specific and human RNA C _t values obtained for each sample and the indicated target genes.

Boxplots of five test samples (S2 and S4 are patient-derived swabs; S6, S3, and S7 are a 1:1, 1:10, 1:100 dilution series, respectively, of virus culture supernatant) in the left column show 50% inter-quartile range and median within the box for various test systems (x-axis) targeting specific viral gene targets (subheading in italics). The plot titles indicate the sample number, the reference C _t value, and the template copy number as determined by digital droplet PCR. “Other” = results obtained from two or more gene targets in one channel and therefore not shown separately. The histogram in right column displays the distribution of reported C _t values (y-axis is the same as in left column boxplots) for each target, indicating the mode and total sample number.

First, we note that the reported individual C _t values from each laboratory corresponded well with viral loads determined by the reference laboratory RT-qPCR and ddPCR (Figure 1). Generally the mode of the reported Ct values (all targets/test systems combined) were within 0.2–3.3 C _t values of the target mean C _t values (S2 = ∼252,000/mL, mean target C _t  = 27.2, mode = 27; S3 = ∼22,300/mL, mean target C _t  = 32.1, mode = 34; S4 = ∼345,300/mL, mean target C _t  = 26.7, mode = 30; S6 = ∼96,100/mL, mean target C _t  = 29.6, mode = 31; and S7 = ∼640/mL, mean target C _t  = 35.9, mode = 37) (Table 2, Figure 1).

We calculated the mean differences of C _t values in three test systems that target more than one gene (Figure 2). For GeneXpert/Xpert Xpress SARS-COV-2, mean difference between C _t for the N- and E-gene was 2.6; for Roche Cobas 6800,8800/cobas SARS-CoV-2 it was 0.1 for the E- and the ORF1ab-gene; for Simplexa Liaison MDS Realtime PCR/Simplexa COVID-19 Direct Reaction Mix it was 0.4 for the ORF1ab- and the S-gene.

Figure 2:

Differences of C _t values in test systems employing more than one gene target.

Barplots show the difference in C _t values for each reported pair, ordered from least to greatest difference, within the following test systems: GeneXpert, C _t N-gene – C _t E-gene; Roche Cobas, C _t E-gene – C _t ORF1ab-gene; Simplexa Liaison, C _t ORF1ab-gene – C _t S-gene. The horizontal lines display the mean difference, indicated in the text annotation.

The three samples from the dilution series were used to assess the linearity of C _t values (Figure 3). While we did not perform statistical analysis of these data, we note that approximately linear results were obtained from many test systems, but some were obviously non-linear (see Figure 3).

Figure 3:

Linearity of C _t values in S6, S3, and S7 of the most frequently used gene targets for a dilution series derived from a virus isolate grown in cell culture.

Lines connect the C _t values reported by each participant from a dilution series of nCoV-19/Austria/CeMM0360/2020 [EPI_ISL_438123] (S6 undiluted, S3 = 10-fold dilution, S7 = 100-fold dilution) from various test systems for five viral gene targets (E, N, S, RdRP, and ORF1b) or “other” for systems which calculate C _t based on two gene targets.

All test systems that included an internal control correctly reported the presence/absence of human RNA in all samples except for one participant reporting a false negative result. The sensitivity for human RNA (inferred from C _t values) varied between samples (coefficient of variation of C _t values for a given sample <16%). We found no differences in C _t values between targets for the same sample (Mann–Whitney U tests, p>0.05) (Table 2 and Figure 1).

The sample containing B.1.1.7 variant virus (S4) was reported positive by all participants, but only 20 (22%) reported evidence of a mutation. Of these, 19 participants specifically reported S4 as either positive for the substitution in the spike protein at site N501Y and/or the deletions at amino acid positions H69/V70 or identified the sample as a lineage B.1.1.7 virus. One participant indicated they suspected variant lineage B.1.351 or P1 in Samples 3, 4, and 7. We noted that the S-gene target PCR of the test system of two participants (2× TaqPath) did not detect the lineage B.1.1.7.

Discussion

Evaluation of results of this round revealed good analytical performance of participants. There were no false positives and only a few false negatives, which were mostly in the sample with the highest mean C _t value (“weakly positive”). As this was the third EQA round, we note the improvement in the false negative ratio over the two previous rounds: in the first and the current third round the weakly positive samples had roughly similar C _t values (38 and 36, respectively) and yet the rate of false negatives fell from 40.3 to 6.7% (per 67 and 134 reported tests, respectively) [22].

In some cases, we could deduce the possible cause for the false negative results. One participant reported false “negative” for S3 and S7 using a test system that requires 400 µL of primary sample, but the participant reported that only 30 µL was used. Each C _t value reported by this participant was 3–4 cycles higher than other participants’ results, and therefore it is likely that low template led to a false negative result for two samples. Two participants reported the interpretation “negative” for S7, and yet reported C _t values between 36.8 and 39.5. We assume these were interpreted “negative” according to an internal laboratory cut-off value. For another five, no cause for the false negative result could be identified based on the data provided.

IFCC’s recommendation to use at least two different target genes in SARS-CoV-2 detection systems was met by only about two thirds of participant test systems. We were able to analyze three test systems that utilize two gene targets for differences in C _t values between the targets. These differences may be useful for laboratory quality control, for example, if the difference deviates beyond defined limits there may be indirect evidence of possible mutations in the target sequences. We note that one test system had a uniformly large difference in C _t values between the targets for the same sample (mean 2.6), the differences in targets between the other two test systems were similar (Figure 2). This may have implications for the use of C _t values as a qualitative assessment of patient status.

Reporting of C _t values of positive results is still requested by medical authorities and may offer additional qualitative assessment of a virus-positive result [23]. However, our results clearly demonstrate that caution should be taken when comparing and interpreting C _t values. First, we observed the well-documented property of RT-qPCR, that primer-probe sets specific for different genomic targets produce different C _t values for the same sample. This is likely related to reaction efficiency and/or the relative abundance RNA template for the different viral genes. Second, the range of C _t values reported for the same target, in the same sample, by different participants was on average approximately 10 C _t units. However, the variability of C _t values was less than in the previous round of this EQA scheme [20]. Finally, in the previous EQA scheme, the data seemed to suggest that laboratories were at least consistently over- or under-estimating the mean C _t value for a given sample. We therefore tested this hypothesis with a dilution series of a virus culture supernatant. The results revealed that many test systems reported approximately linear changes in C _t according to template concentration, but some were notably non-linear (Figure 3).

Possible reasons for the high variability revealed by individual C _t values in general, and departures from expected efficiency in the dilution series, include differences in assay performance and/or in EQA materials characteristics. Commutability of an EQA material describes its ability to properly mimic the behavior of real patient samples that have not been modified [24, 25]. Although a formal commutability analysis could not be performed, the fact that similar results were obtained in both swab and virus culture samples suggests that assays are likely not greatly impacted by matrix effects, at least for the materials involved in this study. Although the amplification of human RNA as an internal control (e.g., as evidence of proper sampling via swab) was reported by some labs, it does not appear to be very widespread. However, if laboratories do not supervise correct sampling, including such an internal control may be helpful to identify invalid samples and prevent the reporting of false negative results. As mentioned, we noted variability in the volume of sample used for extraction, and believe this, combined with assay efficiency related to PCR reagents (primers, polymerases, buffer systems), may have had the greatest impact on the variability of individual Ct values.

In conclusion, this round of EQA was beneficial in simultaneously serving several interested parties. Participant laboratories have received feedback on the quality of their individual results, which had improved over prior rounds. Supervisory authorities can read information about the general analytical performance and laboratory preparedness; and here we note that, from a technical standpoint, considerations should be made concerning the use of C _t values to assess patient status. Finally, we demonstrated that EQA may be useful to IVD manufacturers (as well as diagnostic laboratories) to compare the performance of products in routine use.