Serological diagnostic for SARS-CoV-2: an experimental External Quality Assessment Scheme

Introduction

The current situation due to the COVID-19 pandemic has had a strong impact on professional and management issues in laboratory processes. To meet the high demand incurred by test requests ascertain the presence of COVID-19, clinical laboratories have reorganized their routine work flows, employing novel diagnostic assays made available on the market. Different types of tests have been issued on the market for different purposes: i) diagnostic tests that detect components of the SARS-CoV-2, which can be used to diagnose SARS-CoV-2 infection – these include molecular and antigen tests; ii) serological tests that detect antibodies (e.g., IgM, IgG, IgA, total Ig) to the SARS-CoV-2 virus – these cannot be used to diagnose a current infection, but can identify subjects who have had SARS-CoV-2 infection, or have recovered from COVID-19 infection, these serological tests also playing an important role in current vaccination strategies; iii) tests for the appropriate management of COVID-19 patients, using, for example, biomarkers related to inflammatory responses and to disease severity [1, 2]. In the case of serological tests, the topic of this paper, a considerable number of immunometric methods [3, 4] have been developed to meet the enormous demand for rapid tests designed to detect specific antibodies to SARS-CoV-2, characterized by high biological matrix (e.g., venous, capillary blood, saliva) heterogeneity; antibody class (IgM, IgG, IgA, total Ig); antigenic determinant target (protein N or S, RBD); analytical detection technique (e.g., colloidal gold immunochromatographic assay [CGIA], chemiluminescent immunoassay [CLIA], enzyme-linked immunosorbent assay [ELISA], lateral flow assay [LFA], immunochromatography); analytical sensitivity (i.e., limit of detection, functional sensitivity); diagnostic performances (sensitivity, specificity, agreement with neutralization tests); availability for decentralized systems, possibility of automation; expression of the result (qualitative, semi-quantitative, quantitative) and, finally, type of validation and/or certification (e.g., diagnostic use, research use only, in-house).

Laboratory professionals are required to ensure that accurate results are released in a very short space of time. In order to guarantee the provision of reliable information, of the highest possible standard using commercially available tests, and to ensure that patients are correctly clinically classified, laboratory professionals have implemented several quality assurance procedures. In compliance with the requirement of the ISO 15189:2012, the following steps are crucial:

1. Validation of examination procedures for the purpose, developed in-house;

2. Verification of the performance characteristics stated by the manufacturers in the validation process, of the examination procedures;

3. Definition of criteria for interpretation of results, in collaboration with clinical specialists, on the basis of clinical information, symptom onset, disease severity;

4. Definition of the decision-making algorithms, in relation to the different types of examination procedures and specific target groups of patients/citizens;

5. Implementation of quality assurance systems, in particular internal quality control (CQI) and External Quality Assessment (EQA) programs, to monitor the test performances and internal laboratory procedures.

EQA/Proficiency Testing (PT) schemes are an important part of quality management systems and of strategies to assure the reliability of laboratory information. They enable a laboratory to become aware of any unsatisfactory performances, and to evaluate the degree of comparability of results between laboratories, the analytical systems used by participants and related performances. The final aim of EQAS is to identify areas for improvement and to ensure comparability of results between laboratories. However, in order to make participation in EQA schemes effective, participants must accurately analyze the information provided in EQA reports, make a root cause analysis and undertake improvement action. EQA schemes are not designed to evaluate the correct implementation of internal laboratory procedures (e.g., verification or validation procedures of tests), but to highlight any problems.

In 2020, the EQA for serological assays for SARS-CoV-2 organized by INSTAND e.V, an interdisciplinary non-profit, scientific and medical society with headquarters in Düsseldorf, Germany, was proposed by the Centre of Biomedical Research for Quality in Laboratory Medicine (CRB), a specialized centre of the Veneto Region of Italy, for all Veneto Region Laboratories. The main focus of this experimental EQA is on assessing the ability of laboratories to correctly perform serological tests for SARS-CoV-2.

The aim of the present study was to summarize the overall results collected in this experimental EQA scheme for the serological diagnosis of SARS-CoV-2.

EQA scheme design

In May, October and November, 2020, CRB distributed control samples (supplied by INSTAND e.V) to medical laboratories in the Veneto Region, Italy, for three different respective surveys. The numbers of CRB laboratories participating in each survey were 34 (first survey), 27 (second survey) and 19 (third survey). The results of CRB laboratories were added to those of INSTAND e.V. participants.

Participants were requested to use their routine examination procedures to detect anti-SARS-CoV-2 antibodies. The results, and the information on the analytical systems used, were collected via the CRB website (www.centroricercabiomedica.net), and accessed using a confidential username and password provided to each participant. The CRB results were processed with those of laboratories pertaining to the INSTAND e.V, and a periodical report for each survey was issued and distributed to the laboratories by INSTAND e.V. In the report, the number of collected results with respect to the analytical systems used, was specified.

Results

A total of 4,867 results were collected, different diagnostic systems being used (Supplementary Table 1). For each control sample, the quantity of collected data differed depending on the specific antibody and sample (Table 2).

Most participating laboratories used different commercial assays to detect the concentration of SARS-CoV-2 antibodies based on qualitative, semi-quantitative and quantitative methodologies (e.g., CLIA, ELISA, immunochromatographic rapid test). The results of IgG, IgM, IgA and total Ig, expressed as positive, negative, borderline, for all sample types are shown in Figures 1 –4, respectively.

Figure 1:

Percentage of laboratory answers for IgG.

Sample code: 416001 to 001; 416002 to 002; 416003 to 003; 416004 to 004; 416006 to 006; 416007 to 007; 416008 to 008; 416009 to 009; 416010 to 010; 416012 to 012; 416017 to 017; 416018 to 018; 416019 to 019; 416020 to 020; 416029 to 029; 416030 to 030; 416031 to 031; 416032 to 032.

Figure 2:

Percentage of laboratory answers for IgM.

Sample code abbreviated: 416001 to 001; 416002 to 002; 416003 to 003; 416004 to 004; 416006 to 006; 416007 to 007; 416008 to 008; 416009 to 009; 416010 to 010; 416012 to 012; 416017 to 017; 416018 to 018; 416019 to 019; 416020 to 020; 416029 to 029; 416030 to 030; 416031 to 031; 416032 to 032.

Figure 3:

Percentage of laboratory answers for IgA.

Sample code abbreviated: 416001 to 001; 416002 to 002; 416003 to 003; 416004 to 004; 416006 to 006; 416007 to 007; 416008 to 008; 416009 to 009; 416010 to 010; 416012 to 012; 416017 to 017; 416018 to 018; 416019 to 019; 416020 to 020; 416029 to 029; 416030 to 030; 416031 to 031; 416032 to 032.

Figure 4:

Percentage of laboratory answers for total Ig.

Sample code abbreviated: 416001 to 001; 416002 to 002; 416003 to 003; 416004 to 004; 416006 to 006; 416007 to 007; 416008 to 008; 416009 to 009; 416010 to 010; 416012 to 012; 416017 to 017; 416018 to 018; 416019 to 019; 416020 to 020; 416029 to 029; 416030 to 030; 416031 to 031; 416032 to 032.

The mean percentage of agreement for IgG positive result for SARS-CoV-2 samples was 86.6% (Figures 1 –4). On considering sample 416003 an outlier, eliminating it from the mean, agreement attained 92.4%. The mean percentage for the IgM results was: 34.9% positive; 58.4%, negative; 6.7%, borderline. The highest agreement was found for total Ig results, for which the mean percentage of positive results was 98.2%, whereas, for IgA, distribution of results was: 64.7% positive, 23.4%, negative and 11.9%, borderline.

The observed incongruity in results does not appear to be associated with either a specific laboratory or a specific analytical system. Only for total Ig were discordant results observed when an in-house and/or an undeclared procedure were used.

A greater/higher agreement between laboratories’ results was observed for negative control samples. The mean percentage for a negative result was: 98.2%, for IgG; 98.6 for IgM; 97.6% for IgA; 98.8, for total Ig, respectively. A further observation is that, in the cases of IgM and total Ig, the more incongruous result was related to sample 416017 rather than to an undeclared analytical procedure.

Regarding specificity, agreement between laboratory results can be considered optimal for IgG (98.9%). For IgM, the mean percentage of agreement was 95.7% for negative results; 2.9%, borderline and 1.4%, positive, respectively. The less satisfactory result was found for IgA (86.4%), while complete agreement was observed only for total Ig (100%).

Discussion

The measurement of antibodies to SARS-CoV-2 is widely employed for purposes such as the detection of late infection, sero-epidemiological studies, vaccine clinical trials and monitoring of immune response [5]. Currently, the availability of various kits on the market is increasing, as is their use by clinical laboratories; their analytical efficiency is also constantly improving. The implementation of an EQA scheme to evaluate and monitor the antibodies involved in COVID-19 infection is of crucial importance in the current pandemic. A well-designed EQA scheme, which is a quality assurance tool, may support clinical laboratories in achieving excellent quality performance. The information collected by EQA underpins improvement actions, and is brought to the attention of process stakeholders. In 2020, the CRB undertook an experimental EQA scheme using the EQAS proposed by INSTAND e.V., with the aim of providing a tool to control the performance of participating laboratories and analytical systems used. The analysis of EQA data demonstrates a greater agreement between laboratory results for total Ig than for other single immunoglobulins (IgG, IgM, IgA) in the case of samples positive for SARS-CoV-2 (Figures 1 –4). This finding does not appear to be related to sample characteristics such as severity of symptoms and time from disease onset (Table 1). Therefore, it has not been proven that disease severity influences the extent and course of detectable antibody responses.

The wide divergence observed between IgM results in positive samples (only 34.9% are correct), confirms the limited utility of this class of antibody for the detection of SARS-CoV-2, as reported by other authors [6, 7]. The results observed for IgM indicate potential cross reactions with viruses other than SARS-CoV-2s. Furthermore, false-positive test results may depend on interference from, for example, rheumatoid factors, pregnancy and other viruses (e.g., HIV) [8].

However, the incongruous results highlighted are not attributable to a specific analytical system or laboratory. Results concerning negative control and specificity control demonstrate overall a greater agreement than those in positive samples. Laboratory professionals must therefore rise to the challenge of achieving a better harmonization of results and developing well-defined protocols, the first step being the validation of analytical procedures complying with ISO 15189 requirements, and working in collaboration with the IVD manufacturers. In fact, as laboratory test results are of crucial importance in medical and public health decision-making, it is essential to validate the test performance for fit-for-purpose and guarantee the testing capacity in ensuring optimal quality. A number of actions need to be undertaken for this goal, in particular:

1. Performance evaluation and conformity assessment against ISO 15189 of analytical assays;

2. Sharing of information on the test performance with IVD manufacturers in order to promote improvement in tests;

3. Notification of analytical assays performance to institutional authorities in order to guarantee that the pathway in the health choice is adequate.

Further action is required in order to achieve the harmonization of results. Studies in literature highlight the use by laboratories of different cut-offs in analytical systems, as well as arbitrary measurement units, both of which could compromise the interpretation of results, and explain the discrepancies found and the reasons for this variation [9, 10]. The availability of a suitable reference material could, moreover, aid the harmonization process in establishing comparability and accuracy of analytical results between different analytical assays and, over time, playing an essential role for many tasks in daily laboratory routine (e.g., in the calibration of analytical procedures and their validation, the performance in the EQA schemes, quality assurance protocols). The First WHO International Standard for anti-SARS-CoV-2 immunoglobulin (NIBCS code: 20/136) (version 2 Dec 17, 2020) might allow the accurate calibration of assays to binding antibody units (BAU)/mL, and be used to assist in the comparison of assays detecting the same class of immunoglobulins with the same specificity (e.g., anti-RBD IgG, etc.), thereby reducing inter-laboratory variation and creating a common language for reporting data. This is a fundamental step in setting out projects aiming to increase harmonization and comparability between different SARS-CoV-2 antibody assays [11].

Conclusions

During the COVID-19 pandemic, numerous diagnostic systems, internal laboratory procedures (in-house) and commercial products, have been implemented in order to identify positive subjects. In this context, EQA programs, such as that presented in the present study, have proven very useful in verifying the suitability of analytical procedures used and in identifying possible problems. The data reported demonstrate that, through the continuous monitoring of performances granted by EQA reports in which high quality control samples are used, it is possible to define the state-of-the-art in results interpretation and the more widely used methodologies. For example, of the antibody classes evaluated, IgM is expected to be of limited utility, since inter-laboratory agreement is poor. Moreover, the regular participation in this, and other, EQAS schemes will provide important information concerning the validation and harmonization of both qualitative and quantitative tests with the same specificity, and should enable the monitoring, over time, of laboratory performances for tests conducive to establishing the immunological status of patients and vaccinated individuals [12].

However, the competence of laboratory professionals is a prerequisite for both the design and management of reliable EQA schemes, and for the analysis of data provided in EQA reports [13, 14]. Participation in EQA schemes, above all when new methods become available on the market in response to the emergency, is mandatory, as is the undertaking of corrective and improvement actions whenever necessary, in order to guarantee reliable results.