Evaluation of eleven rapid tests for detection of antibodies against SARS-CoV-2

Introduction

In December 2019, Wuhan city in Hubei Province, China, became the center of an outbreak of a severe pneumonia, later named Coronavirus Disease 2019 (COVID-19), and identified as caused by a novel coronavirus SARS-CoV-2 [1]. The coronavirus was isolated, and full-genome sequencing showed a betacoronavirus in the subgenus sarbecovirus [2]. Human-to-human transmission of SARS-CoV-2 occurs primarily through respiratory droplets. Due to the rapid spread of the virus, the World Health Organization declared COVID-19 a worldwide pandemic by February 2020. The clinical presentation of COVID-19 varies from asymptomatic disease, via mild upper respiratory infection to severe pneumonia with respiratory failure and death.

Molecular diagnostic tests with real-time PCR are used to identify SARS-CoV-2 RNA in respiratory samples. PCR is performed at medical microbiology laboratories, requiring advanced analytical instruments and trained personnel. Shortage of sampling equipment and necessary reagents has periodically limited the number of people being tested for COVID-19 in Norway.

Detecting humoral immune response to the virus is a different analytical approach. Generally, immunoglobulin type M (IgM) is produced during the early stages of an infection, usually followed by production of immunoglobulin type G (IgG). For infection with SARS-CoV-2, however, there is some evidence that IgG may be detected at the same time as IgM, or even earlier [3], [4]. Several enzyme immune assays (EIA) or chemiluminescent immunoassays (CLIA) for detection of antibodies against SARS-CoV-2, both commercial and in-house, will are becoming available in Norwegian hospital laboratories. At the same time, a substantial number of point-of-care rapid tests (lateral flow immunoassays) are currently being marketed. Even though most of the rapid tests are CE/IVD approved, they generally come with very limited documentation on test performance, and only rarely with any manufacturer independent evaluation [5], [6], [7].

Our aim was to perform a limited evaluation of the diagnostic performance of a selection of rapid test for COVID-19 in different clinical settings, and in particular to evaluate if the tests could be used to confirm past infection. Further, we wanted to assess user-friendliness.

Materials and methods

The evaluation was organized as a quality assurance study in a collaborative effort between the municipality of Kristiansand, the Norwegian Institute of Public Health, and the Norwegian Organization for Quality Improvement of Laboratory Examinations (Noklus). The eleven rapid tests chosen for evaluation were a convenience sample, consisting of the tests that could be delivered to Noklus before the set deadline of April 1st, 2020 (Table 1). Suppliers provided their tests free of charge to Noklus and did not pay for the evaluation. In sending the tests, they consented to having the results published.

The rapid tests were for professional use and designed to qualitatively detect IgM and/or IgG antibodies against SARS-CoV-2, with results read visually after 10–15 min. All rapid tests were performed by experienced biomedical laboratory scientists and in accordance with manufactures’ instructions (Supplemental Table) under optimal and standardized conditions, using venous blood samples with K₂-EDTA anticoagulant. A test was considered invalid if the control line did not appear, and inconclusive if it was not possible to read the result (for instance due to blood drawn into the test area).

Patients were enrolled in three study arms: 1) 20 patients with PCR-confirmed COVID-19, hospitalized at Oslo University Hospital, Ullevål, 2) 23 recovered outpatients in the municipality of Kristiansand with previously PCR-confirmed COVID-19, and 3) 49 patients with suspected COVID-19 presenting at the primary care emergency room of Bergen municipality. All patients were tested with rapid test for detection of antibodies, and with PCR against SARS-CoV-2 in samples from upper airways. In study arm 1, rapid test analyses were performed on surplus K₂-EDTA whole blood left over from hematology analyses. In study arms 2 and 3, participants consented to having one tube of K₂-EDTA whole blood drawn for the analyses. In study arms 1 and 2, PCR-confirmed COVID-19 was the inclusion criterion, but in arm 3, PCR results were collected in addition to the rapid test results. We also collected the date and laboratory used for the PCR test, and the number of days since onset of symptoms (in arm one in categories <7, 7–13 and 14+ days).

PCR results from the clinical microbiology laboratories were used as comparison when investigating diagnostic accuracy of the rapid tests. IgM and IgG test results were evaluated separately, except for test E, which detected “total antibodies”. In all study arms, we calculated the tests’ sensitivities (positivity rates). Sensitivity was defined as the proportion of patients with antibodies detectable by the rapid test among those with PCR-confirmed COVID-19. In study arm 3, we also calculated the proportion of participants with negative PCR tests who were antibody negative. Further, in study arm 3, we stratified positivity rates according to days since onset of symptoms (<7 or 7+ days). Because sample sizes were small, we computed 80% confidence intervals for binomial proportions, and we used the adjusted Wald method (8).

User-friendliness was reported by the biomedical laboratory scientists performing the tests.

Results

In the 20 hospitalized patients in study arm 1, the number of days since onset of symptoms was <7 for one patient, 7–13 days for three patients, and 14+ days for 16 patients. In study arms 2 and 3, the median number of days since onset of symptoms was 30 (range 27–36) and 8 (range 2–34), respectively. Of the 23 participants in study arm 3 with a positive PCR test, thirteen had <7 days since onset of symptoms, six had 7–13 days, and four had 14+ days.

Results from hospitalized patients (study arm 1) showed that all the eleven rapid tests detected SARS-CoV-2 IgM and/or IgG antibodies in this population, though with varying sensitivities (Tables 2 and 3). Study arm two consisted of participants who had recovered from PCR-confirmed COVID-19 without requiring hospitalization. In this population, tests A, B, C, and D had higher IgG positivity rates than tests E, F, H I, and J. Confidence intervals (80%) for test K were overlapping with the others (Table 3). Five of the tests had a sensitivity below 65% for IgG.

Of the 49 participants with suspected COVID-19 (study arm 3), 23 had positive PCR tests, and 26 tested negative. In this population, the rapid tests had very low sensitivities when compared to PCR (Tables 2 and 3). Positivity rates increased with increasing number of days since onset of symptoms, especially for IgM (Table 4). However, even for IgM, none of the tests had a positivity rate above 50% among patients with confirmed COVID-19 and less than 14 days of symptoms, although numbers were too small to draw firm conclusions. Few rapid tests were positive in participants with negative PCR tests.

Not all participants with PCR confirmed COVID-19 had detectable SARS-CoV-2 antibodies. Also, despite comparable sensitivities, the rapid tests did not necessarily give the same result in all participants (Supplemental Figures 1–3). For test K, IgM and IgG results were identical in all participants in study arms 1 and 2, and in 46 out of 49 participants in arm 3. Tests F, I and J had higher positivity rates for IgM than IgG in study arm 2, even though all samples were collected more than 14 days after onset of symptoms.

Tests G and H were judged as less user-friendly (Table 5), both when performing the tests and interpreting the results. These tests also had higher proportions of inconclusive or invalid results. Tests C and E were judged easy to perform but difficult to interpret. Test C additionally had a high proportion of inconclusive or invalid results.

Discussion

All the evaluated rapid tests detected SARS-CoV-2 antibodies in participants with PCR-confirmed COVID-19, although positivity rates varied and were generally higher in the study arm of more severely affected patients. In the population with suspected COVID-19, none of the rapid tests had any diagnostic value, but further studies are needed to assess the usefulness of antibody rapid tests in the acute phase of COVID-19 as a supplement to PCR.

PCR for detection of viral RNA and antibody detection tests use different test principles and are not interchangeable. The viral load in upper or lower respiratory tract is highest in early stages of COVID-19, while it takes several days for the adaptive immune system to elicit an antibody response. Early in the infection, we therefore expect PCR to be positive and antibody detection tests to be negative. As the infection progresses and clears, most patients will develop detectable antibodies, while the virus is gradually cleared from the upper airways [9]. Thus, even under the best of circumstance, PCR is far from an ideal “gold standard” for comparison of antibody detecting rapid tests.

If a participant with PCR-confirmed COVID-19 has no detectable antibodies, there are several possibilities: (i) the stage of the infection is too early for antibodies to have been formed, (ii) the level of antibodies produced is too low to be detected, (iii) the participant does not produce antibodies, (iv) a false negative rapid test result, or v) a false positive PCR result (wrong labeling for instance). Similarly, if a participant with negative SARS-CoV-2 PCR has detectable antibodies on a rapid test, there are a number of plausible explanations: (i) the participant is recovering from COVID-19 and had cleared the virus prior to PCR testing, ii) false positive rapid test result (for instance cross reaction with other antibodies), or (iii) false negative PCR result (pre-analytical or analytical issues). Comparing results from several rapid tests with each other may provide some clue as to which is the most likely explanation in each case, but does not provide a definite answer.

To determine a rapid test’s ability to detect past infection, its performance with regard to IgG antibodies is emphasized in the specification criteria for serology point of care tests published by the United Kingdom Medicines & Healthcare products Regulatory Agency [10]. Most of the included rapid tests had higher IgG positivity rates in arm 1 (hospitalized patients) than in arm 2 (recovered outpatients). More severe infection has been associated with higher levels of antibodies [3], [4]. One might speculate that antibody levels were lower in study arm 2, where participants were less severely affected compared to the hospitalized patients in study arm 1. This may have influenced rapid test performance. It is worth noticing that according to manufacturers’ information; most of the rapid tests have been evaluated in samples collected from hospitalized populations. We were not able to evaluate the tests’ performances in a population that had been through COVID-19 with very little or no symptoms.

In study arm 3, most participants had a short duration of symptoms. In this population, all of the rapid tests in our study had low sensitivities compared to PCR, confirming a priori expectations. It is possible that the rapid tests, and IgM in particular, may still have a supplemental role in diagnosing COVID-19 in the acute phase. However, among patients with confirmed COVID-19 and less than 14 days of symptoms, none of the tests in our study had a positivity rate above 50%. Thus, the diagnostic value of using any of these tests during the first two weeks of infection seems very low. Furthermore, a negative result in the acute phase of infection should be followed by a second sample at least 14 days after onset of symptoms, and it is imperative that a negative result of a rapid test is never be used to exclude COVID-19.

In study arm 3, we also calculated the proportion of participants with negative PCR tests who did not have detectable antibodies. We do not know if a participant with a negative PCR test and a positive IgG rapid test has, or has recovered from, COVID-19, or if the rapid test result is a false positive. In a population where the prevalence of COVID-19 is low, there could be a substantial a risk of false positive results, which we cannot at present quantify because of the lack of knowledge of the tests’ specificities. Nevertheless, it is worth noticing that the few positive test results among patients with negative PCR and short duration of symptoms, were mainly IgM. In any case, an isolated positive IgM result should be followed by a second sample to detect IgG-seroconversion and thereby rule out the possibility of an unspecific IgM result.

The strengths of our study include the large number of rapid tests included, and the varied clinical settings we were able to evaluate the tests in. The small sample size is an important limitation, meaning our results must be interpreted with caution, particularly when considering the different groups and the time from the onset of symptoms. Also, PCR is not an ideal gold standard when evaluating antibody detecting rapid tests. Finally, the lack of pre-COVID-19 sera limited our ability to assess false positives and the tests’ specificities. As more rapid tests are emerging and EIA becomes increasingly available, our group will continue to evaluate tests. We will use EIA methodology as a comparison method in addition to PCR, and also analyze sera collected before the COVID-19 pandemic. This will allow us to investigate diagnostic accuracy and analytical properties of the rapid tests more thoroughly.

Conclusions

Based on our observations, results from rapid test for detection of antibodies seem to be unreliable during the first two weeks of symptoms of COVID-19. However, rapid tests with a high IgG sensitivity may be useful for confirmation of past infection. We also recommend using tests that are user-friendly and have a low proportion of invalid/inconclusive tests. In our study, it seems that tests A, B, D, and possibly K (Table 1) fulfill these recommendations, under the assumption that their specificities are high.

To allow sufficient time to develop antibodies, we recommend not using a rapid test until at the earliest two weeks after onset of symptoms. A negative test may be repeated, but not all SARS-CoV-2 infected develop antibodies, and not all antibodies are necessarily detected by the rapid test. Thus, a negative rapid antibody test does not rule out current nor past COVID-19.

Finally, we recommend performing an independent evaluation before introducing a new rapid test, and importantly, in the population it is intended for. Our results indicate that when a rapid test is to be used to detect past infection in people who have not required hospitalization, it is not sufficient to validate the test in a hospitalized population.