Evaluation of eight lateral flow tests for the detection of anti-SARS-CoV-2 antibodies in a vaccinated population

The NHS Research Ethics Committee (REC, UK) [REC reference:16/NW/0170] and the central Liverpool research ethics committee [Protocol Number: UoL001207] granted ethical approval for this work. The Integrated Research Application System (IRAS) Project ID is: 202413.

Participants were recruited from the Liverpool School of Tropical Medicine and University of Liverpool staff networks as well as members of the public through social media outreach. All participants were recruited from the Liverpool, Merseyside region of the UK. Participants were recruited onto an existing study (The Human Immune Responses to Acute Viral Infections study (AVIS), 16/NW/0160). Participants were recruited between January-December 2021. All participants gave written informed consent. Healthy individuals who had received or were due to receive their COVID-19 vaccination and were aged 18 years or over were recruited to the study. Individuals taking part in COVID vaccine trials were excluded from the study. Case record form (CRF) was completed by a trained member of staff to confirm eligibility. Participants were asked to provide a blood sample at days 21 (± 7 days), 42 (± 7 days) post dose one and two of their COVID-19 vaccine.

Venous blood (5 ml, plain serum tube) samples were collected by trained health care workers and processed on the same day of collection. Briefly, venous blood samples were centrifuged at 1500 g for 10 min and serum was aliquoted and stored at − 20 °C until testing. Following processing samples were stored at − 20 °C before testing in bulk.

LFTs were performed according to the instructions for use (IFU). Serum was allowed to thaw at room temperature for 15 min and vortexed for 5 s. A sample from each individual was tested on all brands of LFTs. According to individual IFU’s, 10–20 µl of serum was added to the sample well and 2–3 drops of manufacturer specified buffer solution was added. Tests were run for 10–15 min, according to IFU, and read independently by two readers. Where there was a disagreement a third reader was used. Failed tests were repeated once. Characteristics of the tests used are shown in Table 1. When no details on the antigen composition were provided in the IFU, the company was approached for further information. Although all tests used in this study detect both IgM and IgG antibodies, IgG was the focus of this investigation and results from IgM are not included. This is due to the longevity of IgG antibodies compared to IgM likely making them a more reliable target as proxy for immunity.

Samples were analysed by quantitative and semi-qualitative chemiluminescent microparticle immunoassay (CMIA) on the fully automated Alinity i system (Abbot, United States) as a reference standard. SARS-CoV-2 IgG II CMIA (Abbott, Ireland) was used to quantify anti-S-RBD IgG antibodies in serum samples. To distinguish between antibodies produced from natural infection from SARS-CoV-2 to those produced from vaccination the samples were also analysed using the SARS-CoV-2 IgG I assay (Abbott, Ireland), a semi-qualitative CMIA to detect anti-nucleocapsid IgG antibodies, a method that was utilised by Narasimhan et al. and found to be effective [18]. When using these assays, individuals positive for anti-nucleocapsid IgG antibodies are considered to have been naturally infected with SARS-CoV-2 and will also test positive for anti-S-RBD IgG antibodies. If an individual gives a negative test result for anti-nucleocapsid IgG antibodies but a positive result for anti-S-RBD IgG antibodies, then this individual is assumed to have not had a natural infection and has antibodies generated in response to vaccination. Following manufacturer recommendations, results higher or equal to 50 AU/ml when using the SARS-CoV-2 IgG II Quant assay were considered positive for anti-S-RBD IgG antibodies. Similarly, results higher than or equal to 1.4 S/C (Sample control index) when using the SARS-CoV-2 IgG I Qualitative assay were considered positive for anti-nucleocapsid IgG antibodies as per manufacturer’s instructions.

Sensitivity of the eight brands of AbLFT was calculated in reference to the results from the SARS-CoV-2 IgG II CMIA that quantifies anti-S-RBD IgG antibodies. Only this test was used as a reference standard as we wanted to assess the sensitivity of detecting those with antibodies from the vaccine which would be anti-spike given the main available vaccines during sample collection utilised the spike protein in their design. Sensitivity was calculated as a proportion of the number of positive and negative AbLFT results that were confirmed by CMIA. The results of which can be found in Additional file 1: Table S3.

A binomial mixed effect model was designed to provide point-estimates for sensitivity based on the data collected from the laboratory evaluation. Due to the small sample size, binomial mixed models allowed us to borrow strength information across all individuals and estimate the sensitivity of each test more reliably than the conventional approach based on simple proportions. The model also analyses the impact of the key variables on the sensitivity of the different LFTs and to determine parameters for the calculation of the sensitivities and confidence intervals of each LFT at each dose. Details of modelling methods can be found in Additional file 1.

We used binomial mixed models to account for the clustering arising from the administration of multiple tests on the same individuals. These models were used to assess the effect of the LFT brand and other factors on the risk of a positive test. More details on the binomial mixed models and a comparison with the standard approach for estimation of LFT sensitivity can be found in the Additional file 1. The statistical analysis was conducted in RStudio (Version: 2021.9.1.372).

Point estimates of sensitivity from the binomial mixed effect model and the standard percentage calculation were largely comparable and results from both are summarised in Tables 2 and 3. Sensitivity from the binomial mixed effect model for dose 1 ranged from 4.38% [CI95% 1.24, 8.40] for KHB to 95.43% [87.42, 97.42] for Fortress. For dose 2, sensitivities ranged from 20.15% [13.15, 30.02] for KHB to 99.30% [96.46, 99.73] for Fortress. Similarly, the standard percentage calculation for dose 1 ranged from 14.77% [CI95% 8.11, 23.94] for KHB to 97.72 [CI95% 92.3, 99.72] for Fortress. For dose 2, sensitivities ranged from 11.59% [CI95% 5.14, 21.57] for KHB to 100% [CI95% 94.79, 100] for CTK, Fortress and Bionote.

Both the mixed effect model and the standard percentage calculation showed that six out of eight LFTs had a statistically significant increase in sensitivity estimates from dose 1 to dose 2. This may be indicative of higher antibody titres following a second vaccine dose making it easier to detect antibodies on LFT. Fortress did not have a statistically significant increase in sensitivity however had already achieved the highest sensitivity of all eight brands at dose 1.

Sensitivities of the tests when focussing on target antigen was varied. Three tests used nucleoprotein, two used both nucleoprotein and spike and three used spike alone, with two specifying the receptor binding domain (RBD) (Table 1). The tests that achieved the highest sensitivities post dose 2, CTK, Fortress and Bionote, all used different antigens of spike, spike-RBD and nucleoprotein respectively. The LFT with the lowest sensitivity was KHB which used nucleoprotein antigen.

Our mixed effect model found that vaccine brand and day of sampling (Day 21 vs day 42) had no significant effect on overall test result and were therefore removed from the model analysis (Additional file 1: Table S1). Dose had a significant, positive effect on positivity rate with more positive results being detected after dose 2 compared to dose 1 (Additional file 1: Table S2).