Meta-analysis of the clinical performance of commercial SARS-CoV-2 nucleic acid and antibody tests up to 22 August 2020.

BackgroundReliable testing for SARS-CoV-2 is key for the management of the COVID-19 pandemic.AimWe estimate diagnostic accuracy for nucleic acid and antibody tests 5 months into the COVID-19 pandemic, and compare with manufacturer-reported accuracy.MethodsWe reviewed the clinical performance of SARS-CoV-2 nucleic acid and antibody tests based on 93,757 test results from 151 published studies and 20,205 new test results from 12 countries in the European Union and European Economic Area (EU/EEA).ResultsPooling the results and considering only results with 95% confidence interval width ≤ 5%, we found four nucleic acid tests, including one point-of-care test and three antibody tests, with a clinical sensitivity ≥ 95% for at least one target population (hospitalised, mild or asymptomatic, or unknown). Nine nucleic acid tests and 25 antibody tests, 12 of them point-of-care tests, had a clinical specificity of ≥ 98%. Three antibody tests achieved both thresholds. Evidence for nucleic acid point-of-care tests remains scarce at present, and sensitivity varied substantially. Study heterogeneity was low for eight of 14 sensitivity and 68 of 84 specificity results with confidence interval width ≤ 5%, and lower for nucleic acid tests than antibody tests. Manufacturer-reported clinical performance was significantly higher than independently assessed in 11 of 32 and four of 34 cases, respectively, for sensitivity and specificity, indicating a need for improvement in this area.ConclusionContinuous monitoring of clinical performance within more clearly defined target populations is needed.

Introduction

Testing is one of the central pillars of public health actions in epidemic and pandemic situations to allow timely identification, contact tracing and isolation of infectious cases to reduce the spread of infectious diseases. In addition, it allows estimating disease incidence, disease prevalence, and prevalence and duration of humoral immunity. Reliable testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and timely reporting of the data to public health authorities is therefore key for the management of the coronavirus disease (COVID-19) pandemic. This requires appropriate and sufficiently accurate diagnostic tests to identify individuals who are currently infected with SARS-CoV-2 as well as those who have been infected in the past. Timely access to testing, sufficient supply of testing materials, availability of tests and related reagents and consumables as well as high-throughput testing are pivotal in this context.

By August 2020, a large number of commercial tests for SARS-CoV-2 RNA detection (nucleic acid tests) were available, as well as serological tests for SARS-CoV-2-specific antibodies. The various types of tests can be used for different purposes and many of these tests have the CE certificate for in vitro diagnostics (CE-IVD) that indicates compliance with the European IVD directive (98/79/EC) and can thus be marketed in the countries in the European Union and European Economic Area (EU/EEA). In addition, the United States (US) Food and Drug Administration has granted emergency use authorisations for many commercial tests in the US, and the World Health Organization (WHO) maintains an emergency use listing of commercial tests [1,2]. It is, however, important to note that CE certification is based on a self-declaration of the test manufacturer, including the claims on performance of the test. Independent information on the clinical performance of these tests in terms of sensitivity and specificity is still limited, and yet this is critical for proper interpretation of results.

For this reason, the European Centre for Disease Prevention and Control (ECDC) launched a continuous call to EU/EEA countries and the United Kingdom (UK) on 1 April 2020 to provide any such clinical performance data for sharing with other countries. These data, provided by 12 countries, are presented in this article. In addition, we included publicly available data. Finally, minimal performance criteria for different intended uses were gathered from public sources and aided by a survey conducted among EU/EEA countries and the UK from 20 May to 1 June 2020.

Methods

Search strategy and selection criteria

Studies containing potentially usable data on the clinical performance of SARS-CoV-2 nucleic acid and antibody tests were first extracted from systematic reviews on this topic. We identified these reviews through an initial PubMed (Medline) search for systematic reviews and meta-analyses for ‘COVID-19’ and ‘SARS-CoV-2’, followed by snowballing using the ‘find similar articles’ feature. We extended the selection with the studies listed in the Foundation for Innovative Diagnostics database (FIND, www.finddx.org/covid-19/tests) and the European Commission COVID-19 In Vitro Diagnostic Devices and Test Methods Database (EC, https://covid-19-diagnostics.jrc.ec.europa.eu). Both databases attempt to exhaustively identify peer-reviewed as well as grey literature on clinical performance of COVID-19 tests and are continuously updated [3,4]. Results from the latter were further filtered for articles with a description indicating that they contain clinical performance results. We also included results produced by the US Food and Drug Administration (FDA) [5]. Finally, we searched PubMed according to the query shown in Supplement 1.

The resulting studies were subsequently assessed for eligibility. By August 2020 there were no clinical performance studies that can be judged as having low risk of bias and low applicability concerns. Systematic reviews up to that point have not used risk of bias or applicability concerns as exclusion criteria [6-9]. This was not done in this work either. Instead, we excluded studies if they did not contain data on commercial tests, or if one or more of the authors were employed by the developer or manufacturer of the index test, to avoid possible conflicts of interest. Subsequently, we also excluded studies with an ineligible design, such as blinded tests, analytical validation only, use of another threshold for positivity than in the instructions for use, comparisons between different specimen types or use of an antibody rather than nucleic acid test as reference test for any type of index test.

Further exclusions were made at sample level based on the reference test employed. Samples classified as actual negatives, i.e. used for determining specificity, had to be taken (i) before the COVID-19 outbreak, in practice before 2020, (ii) from an individual without COVID-19-compatible symptoms, or (iii) from an individual with COVID-19-compatible symptoms but who was confirmed with another respiratory illness. Samples classified as actual negatives that were taken during the outbreak and were negative according to a nucleic acid test were therefore excluded. We did this to maximally reduce misclassification as actual negatives because of known issues with sensitivity of nucleic acid tests. Such misclassified samples would artificially lower index test specificity, in particular when the index test is more sensitive than the reference test [10-16]. For the same reason, the reported sensitivity of nucleic acid index tests, based on a nucleic acid reference test, was considered to be a positive agreement instead, calculated as part of a head-to-head comparison between the two tests. For antibody index tests on the other hand, we considered a nucleic acid test to be a valid reference test to determine actual positive samples and sensitivity, in accordance with WHO interim guidelines [17].

Manufacturer-reported clinical sensitivity and specificity data were extracted from instructions for use where available, or otherwise from the manufacturer’s website. Sensitivity results derived from contrived samples spiked with purified viral RNA were excluded.

Original clinical performance data

Primary clinical performance data generated by the COVID-19 microbiological laboratories author group were assessed by the ECDC according to the same criteria as those of the literature review.

Statistical analysis

Meta-analysis of the included clinical sensitivity and specificity results was performed per test and per target, i.e. the genomic region for nucleic acid tests and the antibody isotype for antibody tests. Antibody test sensitivity results below the threshold number of days after onset were excluded. Sensitivity and positive agreement results were further stratified by case population as hospitalised cases, mild or asymptomatic cases, or unknown. We calculated pooled sensitivity and specificity values using fixed effects analysis, i.e. separately summing and dividing the number of correct predictions by the total number of samples in the group. Wilson score 95% confidence intervals (CI) were calculated for pooled results. Study heterogeneity was assessed through the I2 statistic, calculated through random effects analysis using R version 4.0.2 and the metafor package [18]. We considered I2 values < 50.0% as low heterogeneity, 50.0–74.9% as moderate and ≥ 75% as high heterogeneity.

Results

Minimum performance criteria

By 1 June 2020, minimum performance criteria for tests were publicly available from Belgium, France, the Netherlands and the UK (Supplementary Table S1). All were applicable solely to antibody tests. The intended uses included diagnosis of COVID-19, determination of exposure to SARS-CoV-2 and determination of the immune status against SARS-CoV-2. Minimum clinical sensitivity for all of the specified intended uses ranged from 85% to 98%, with a median of 95%. These thresholds applied to samples collected at least 15 days post onset of symptoms (dpo), taking into account the time to seroconversion. Minimum clinical specificity for all of the specified intended uses was 98% in three countries and 98.5% in one. For nucleic acid confirmatory tests, the draft WHO Target Product Profiles for priority diagnostics to support response to the COVID-19 pandemic state > 95% to > 98% sensitivity (acceptable/desired) and > 99% specificity [19].

We used general thresholds of > 95% sensitivity and > 98% specificity to determine if a test met the minimum performance criteria, together with a maximum 95% CI width ≤ 5%. For results on IgM antibodies only, an upper limit of ≤ 28 dpo, or the highest dpo category with an upper limit ≤ 28 dpo, was added since IgM antibodies decrease fairly rapidly and such tests are not intended to be used long after exposure [20]. These sensitivity and specificity thresholds can be converted to false positives (FP) and negatives (FN), and positive and negative predictive value (PPV, NPV) if the prevalence of the condition, i.e. SARS-CoV-2 nucleic acid or antibody positivity, is known. These metrics better express the real impact of the accuracy. For a hypothetical low prevalence of 1% in a population of 100,000 people, the PPV would be > 32.4% (FP < 1,980) and NPV > 99.9% (FN < 50). For a high prevalence of 5%, these values would be > 71.4% (FP < 1,900) and > 99.7% (FN < 250). Finally, for a high prevalence of 30%, PPV would be > 95.3% (FP < 1,400) and NPV > 97.9% (FN < 1,500).

Primary clinical performance data

We identified eight systematic reviews, including one by health technology assessment bodies not listed as a peer-reviewed study, and included the primary studies they were based on [6-9,21-24]. The full list of studies in the FIND and EC databases was retrieved on 22 August 2020. PubMed was searched on the same date. From the EC database, 268 of 385 studies were screened out because their description did not indicate that they contained clinical performance data on commercial tests. Of the remaining 117 studies, 81 were not present in the FIND database and 82 were not present in the EC database. From the PubMed results, 1,520 of 1,738 studies were screened out. From the combined list of 364 unique studies, 105 had no clinical performance data on commercial nucleic acid or antibody tests, 34 were excluded because of a potential conflict of interest and 74 were excluded because of ineligible design, leaving a total of 151 included studies. Of those, 53 were exclusively found through the Pubmed search and 15 in the FIND database. The remaining studies were listed by at least two sources.

A complete overview of the study selection is given in Figure 1. After exclusion of antibody test sensitivity results ≤ 14 dpo and ineligible specificity results, a total of 37,435 and 56,322 index test results remained for calculation of sensitivity and specificity, respectively. After addition of original, previously unpublished results provided by the authors of this study, this increased to 47,543 and 66,419 index test results, respectively, for 198 tests. A descriptive overview of the number of studies and results per country in given in Table 1. A complete overview of the studies is given in Supplementary Tables S2-S4.

Figure 1

Selection of public studies on clinical performance of SARS-CoV-2 nucleic acid and antibody tests, up to 22 August 2020 (n = 151)

Table 1

Descriptive statistics on the number of published studies on clinical performance of SARS-CoV-2 nucleic acid and antibody tests, whether we included additional original data, and number of samples included in the meta-analysis, up to 22 August 2020 (n = 151 studies)

Country	Studies	Original data	PCRsens/spec	CLIAsens/spec	ELISAsens/spec	LFIAsens/spec	Otherasens/spec	Totalsens/spec
Australia	3	No	125/59	0/0	209/0	1,511/1,012	0/0	1,845/1,071
Austria	5	No	115/75	195/2,308	421/0	220/0	0/0	951/2,383
Belgium	6	Yes	22/6	1,192/1,031	957/922	3,934/2,985	287/254	6,392/5,198
Brazil	1	No	0/0	0/0	0/0	0/100	0/0	0/100
Canada	1	No	0/0	84/150	185/150	499/450	0/0	768/750
China	17	No	364/0	3,659/1,572	1,494/726	1,038/557	0/0	6,555/2,855
Croatia	0	Yes	168/271	0/0	0/0	0/0	0/0	168/271
Cyprus	0	Yes	6/466	0/0	0/0	0/0	0/0	6/466
Denmark	2	No	0/0	1,495/4,421	195/1,403	126/62	0/0	1,816/5,886
Ecuador	1	No	33/21	0/0	0/0	0/0	0/0	33/21
Finland	3	Yes	121/75	0/82	64/238	0/242	0/0	185/637
France	13	Yes	567/324	173/165	515/154	1,160/486	154/625	2,569/1,754
Germany	9	No	85/200	643/1,597	508/568	32/13	0/0	1,268/2,378
Greece	0	Yes	0/0	0/0	139/20	0/0	0/0	139/20
Hong Kong SAR	1	No	72/114	0/0	0/0	0/0	0/0	72/114
Italy	10	No	0/0	139/37	531/203	60/97	0/0	730/337
Japan	5	No	340/435	0/0	0/0	735/245	98/111	1,173/791
Luxembourg	0	Yes	0/0	0/0	235/218	0/0	0/0	235/218
The Netherlands	4	Yes	253/210	415/1,177	2,107/3,449	2,336/1,642	0/0	5,111/6,478
Norway	1	No	0/0	0/0	0/0	207/0	0/0	207/0
Poland	0	Yes	390/662	0/0	0/0	0/0	0/0	390/662
Portugal	0	Yes	0/0	0/0	0/0	22/28	0/0	22/28
Singapore	2	No	0/0	202/878	0/0	0/0	0/0	202/878
Slovenia	1	Yes	168/641	0/0	0/0	0/0	0/0	168/641
South Korea	1	No	0/0	0/0	0/0	140/158	0/0	140/158
Spain	4	No	0/0	0/0	0/124	806/566	0/0	806/690
Sweden	2	Yes	39/4	58/113	0/0	78/248	0/0	175/365
Switzerland	6	No	1,920/3,816	0/0	312/50	129/50	100/200	2,461/4,116
Taiwan	1	No	0/0	0/0	0/0	129/0	0/0	129/0
United Kingdom	17	No	15/1710	1,975/5,247	65/0	412/200	0/0	2,467/7,157
United States	35	No	2,273/2,628	1,260/4,164	794/769	5,446/11,140	587/1,295	10,360/19,996
Total	151	NA	7,076/11,717	11,490/22,942	8,731/8,994	19,020/20,281	1,226/2,485	47,543/66,419

CLIA: chemiluminescence assay; ELISA: enzyme-linked immunosorbent assay; LFIA: lateral flow immunoassay; sens/spec: number of samples that are reference test positive/negative; NA: not applicable; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2.

a Includes loop-mediated isothermal amplification, microarray, transcription-mediated amplification, and enzyme-linked fluorescent assay.

EC: European Commission COVID-19 In Vitro Diagnostic Devices and Test Methods database; FDA: United States Food and Drug Administration; FIND: Foundation for Innovative Diagnostics database; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2.

Meta-analysis

Pooled estimates for clinical sensitivity and specificity per test, target and, for sensitivity, case population were made. For antibody tests, we restricted the results to those estimates that had a 95% CI width ≤ 5% and were derived from at least two studies, to be able to assess study heterogeneity. Based on the minimum performance criteria analysis, results ≥ 95% sensitivity and/or ≥ 98% specificity for a particular population are highlighted in Table 2. Among these results, there were two CLIA, one ELISA and no LFIA/POC that had ≥ 95% sensitivity and nine CLIA, four ELISAs and 12 LFIA/POC that had ≥ 98% specificity, including the three with ≥ 95% sensitivity. Study heterogeneity was low for four of 10 sensitivity and 53 of 69 specificity results with CI width ≤ 5%. There were few sensitivity results for IgG for mild or asymptomatic cases, for IgA and for total antibody, none of which had a CI width ≤ 5%. In four cases where the same test was used for hospitalised cases, a reduction in sensitivity was observed of 7.4%, 11.0%, 13.1% and 19.2% for IgG (Table 2). For IgA and total antibody, data were available for only one test each. A reduction of 28.8% was observed for IgA and an increase of 6.0% for total antibody. The latter increase was probably due to the small number of samples for both populations.

Table 2

Pooled sensitivity and specificity results for SARS-CoV-2 antibody tests with confidence interval width ≤ 5% for either or both and based on at least two studies, up to 22 August 2020

Category	Test	Target	Case population	Sensitivitya	Specificitya
CLIA	Abbott, SARS-CoV-2 IgG assay on Architect	IgG	Hospitalised	95.9 (93.4–97.5)n = 368BE, CA, NL, UK, US(3)	99.5 (99.3–99.6)n = 8,243AT, BE(2), CA, DE(2), DK, FI, FR(3), IT, NL, SE, SG, UK(3), US(8)
CLIA	Abbott, SARS-CoV-2 IgG assay on Architect	IgG	Mild/asymptomatic	88.5 (84.6–91.5) bn = 331NL, UK(2), US	Same as above
CLIA	Abbott, SARS-CoV-2 IgG assay on Architect	IgG	Unk	92.0 (90.4–93.3)n = 1,332AT, BE, DE, DK, FI, FR(2), SE, SG, UK(2), US(4)	Same as above
LFIA, POC	Anhui Deep Blue Medical Technology, COVID-19 (SARS-CoV-2) IgG/IgM Antibody Test Kit	IgG	Na	Nd	99.4 (96.5–99.9)n = 158CA, US
ELISA	Beijing Wantai Biological Pharmacy Enterprise, Wantai SARS-CoV-2 IgM ELISA	IgM	Hospitalised	92.8 (88.3–95.7) b,cn = 195CN(2), NL	98.7 (98.0–99.1)n = 1,505CN(2), DK, NL(2)
ELISA	Beijing Wantai Biological Pharmacy Enterprise, Wantai SARS-CoV-2 total Ab ELISA	Total Ab	Hospitalised	97.5 (95.9–98.5) cn = 603CN(2), DE, DK, NL	99.5 (99.2–99.7)n = 3,097CN(2), DE, DK(2), FR(2), NL(3)
ELISA	Beijing Wantai Biological Pharmacy Enterprise, Wantai SARS-CoV-2 total Ab ELISA	Total Ab	Unk	97.5 (94.9–98.8)n = 279AT, DK, FR	Same as above
ELISA	Bio-Rad, Platelia SARS-CoV-2 Total Ab	Total Ab	Na	Nd	96.4 (93.3–98.1)n = 250BE, FR, LU, NL
LFIA, POC	CTK Biotech, OnSite COVID-19 IgG/IgM Rapid Test	IgG	Na	Nd	98.6 (95.2–99.6)n = 148AU, NL
CLIA	DiaSorin, Liaison XL S1/S2 IgG chemiluminescence immunoassay	IgG	Hospitalised	92.9 (89.6–95.2) b,cn = 324CA, DE, NL	97.7 (97.3–98.0) cn = 5,994AT, BE(2), CA, DE(3), DK, FI, FR, NL(2), SE, UK, US(2)
CLIA	DiaSorin, Liaison XL S1/S2 IgG chemiluminescence immunoassay	IgG	Mild/asymptomatic	81.9 (76.3–86.3) bn = 226NL, UK	Same as above
CLIA	DiaSorin, Liaison XL S1/S2 IgG chemiluminescence immunoassay	IgG	Unk	90.9 (88.9–92.6) dn = 967AT(2), BE(2), DK, SE, UK, US	Same as above
CLIA	Diazyme Laboratories, DZ-Lite SARS-CoV-2 IgM and IgG CLIA	IgG	Unk	95.3 (84.5–98.7) bn = 43US(2)	99.0 (97.5–99.6)n = 414US(2)
CLIA	Diazyme Laboratories, DZ-Lite SARS-CoV-2 IgM and IgG CLIA	IgG or IgM	Unk	100.0 (91.8–100.0) bn = 43US(2)	98.6 (96.9–99.3)n = 414US(2)
CLIA	Diazyme Laboratories, DZ-Lite SARS-CoV-2 IgM and IgG CLIA	IgM	Unk	90.7 (78.4–96.3) bn = 43US(2)	99.5 (98.3–99.9)n = 414US(2)
LFIA, POC	Dynamiker Biotechnology Tianjin, 2019 nCoV IgG/IgM Rapid test	IgG or IgM	Hospitalised	100.0 (89.0–100.0) bn = 31BE, DK	97.6 (94.8–98.9)n = 248BE, DK, SE
LFIA, POC	Dynamiker Biotechnology Tianjin, 2019 nCoV IgG/IgM Rapid test	IgG or IgM	Unk	89.0 (79.8–94.3) b,dn = 73SE, TW	Same as above
ELISA	Epitope Diagnostics, EPI-KT-1032 Coronavirus COVID-19 IgG ELISA Kit	IgG	Hospitalised	94.0 (86.7–97.4) b,cn = 83CA, NL, US	97.6 (96.7–98.3) cn = 1,451AT, CA, DE(2), NL, UK, US(3)
ELISA	Epitope Diagnostics, EPI-KT-1032 Coronavirus COVID-19 IgG ELISA Kit	IgG	Mild/asymptomatic	74.8 (65.8–82.0)b,dn = 107NL, US	Same as above
ELISA	Epitope Diagnostics, EPI-KT-1032 Coronavirus COVID-19 IgG ELISA Kit	IgG	Unk	96.0 (90.1–98.4) b,cn = 99AT, DE, US	Same as above
ELISA	Epitope Diagnostics, EPI-KT-1033 Coronavirus COVID-19 IgM ELISA Kit	IgM	Hospitalised	95.5 (78.2–99.2) b,cn = 22CA, NL	98.1 (97.0–98.9)n = 810AT, CA, NL, US
ELISA	Epitope Diagnostics, EPI-KT-1033 Coronavirus COVID-19 IgM ELISA Kit	IgM	Unk	83.3 (70.4–91.3) b,cn = 48AT, US	Same as above
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgA S1 ELISA	IgA	Hospitalised	96.0 (92.5–97.9) bn = 224BE(2), CA, DK, FI, FR, GR, NL	86.7 (84.9–88.3) dn = 1,459AU, BE(2), CA, DK, ES, FI(2), FR(2), GR, LU, NL(2), US
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgA S1 ELISA	IgA	Mild/asymptomatic	67.2 (55.0–77.4) bn = 64FI, NL	Same as above
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgA S1 ELISA	IgA	Unk	94.8 (90.9–97.1) bn = 212AU, BE, FR, US	Same as above
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgG S1 ELISA	IgG	Hospitalised	92.6 (89.7–94.7)n = 431BE(3), CA, CH(2), DE, DK, FI, FR, GR, NL, US	97.9 (97.4–98.3)n = 3,954AU, BE(3), CA, CH(2), DE(6), DK(2), ES, FI(2), FR(3), GR, LU, NL(2), US(5)
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgG S1 ELISA	IgG	Mild/asymptomatic	79.5 (71.9–85.5) b,dn = 132CH, FI, NL, US	Same as above
ELISA	Euroimmun Medizinische Labordiagnostika, Anti-SARS-CoV-2 IgG S1 ELISA	IgG	Unk	89.0 (86.7–91.0) cn = 785AT, AU, BE, DE(2), DK, FR, UK, US(2)	Same as above
LFIA, POC	Getein Biotech, One Step Test for Novel Coronavirus (2019-nCoV) IgM/IgG Antibody (Colloidal Gold)	IgG	Na	Nd	100.0 (96.9–100.0)n = 120CA, US
LFIA, POC	Getein Biotech, One Step Test for Novel Coronavirus (2019-nCoV) IgM/IgG Antibody (Colloidal Gold)	IgG or IgM	Na	Nd	99.2 (95.4–99.9)n = 120CA, US
LFIA, POC	Getein Biotech, One Step Test for Novel Coronavirus (2019-nCoV) IgM/IgG Antibody (Colloidal Gold)	IgM	Na	Nd	99.2 (95.4–99.9)n = 120CA, US
LFIA, POC	Guangzhou Wondfo Biotech, Wondfo SARS-CoV-2 Antibody Test	IgG or IgM	Unk	88.0 (82.6–92.0) b,dn = 184AU, ES, TW, US	99.3 (98.3–99.7)n = 605AU, BR, ES, US(2)
LFIA, POC	Hangzhou Alltest Biotech, 2019-nCoV IgG/IgM Rapid Test Cassette	IgG	Unk	88.7 (81.6–93.3) bn = 115AU, ES	100.0 (98.5–100.0)n = 254AU, ES(2)
LFIA, POC	Hangzhou Alltest Biotech, 2019-nCoV IgG/IgM Rapid Test Cassette	IgG or IgM	Unk	92.3 (87.2–95.4) bn = 168AU, ES, TW	96.7 (93.8–98.2)n = 269AU, DK, ES(2)
LFIA, POC	Hangzhou Alltest Biotech, 2019-nCoV IgG/IgM Rapid Test Cassette	IgM	Unk	21.7 (15.2–30.1) b,dn = 115AU, ES	97.2 (94.4–98.7)n = 254AU, ES(2)
LFIA, POC	Innovita Biological Technology, 2019-nCoV Ab Test (Colloidal Gold)	IgG	Hospitalised	86.9 (76.2–93.2) bn = 61CA, JP	100.0 (98.5–100.0)n = 258CA, JP, US
LFIA, POC	Innovita Biological Technology, 2019-nCoV Ab Test (Colloidal Gold)	IgM	Hospitalised	75.4 (63.3–84.5) b,dn = 61CA, JP	98.4 (96.1–99.4)n = 258CA, JP, US
ELISA	Mikrogen Diagnostik, recomWell SARS-CoV-2 IgG	IgG	Na	Nd	96.4 (94.2–97.8)n = 445BE, DE, NL
ELISA	NovaTec Immundiagnostica, NovaLisa SARS-CoV-2 IgA ELISA	IgA	Hospitalised	88.7 (78.5–94.4) bn = 62BE(2)	95.2 (92.1–97.1) cn = 293BE(2), IT, NL
ELISA	NovaTec Immundiagnostica, NovaLisa SARS-CoV-2 IgG ELISA	IgG	Hospitalised	91.9 (82.5–96.5) bn = 62BE(2)	97.3 (94.7–98.6)n = 293BE(2), IT, NL
ELISA	NovaTec Immundiagnostica, NovaLisa SARS-CoV-2 IgM ELISA	IgM	Hospitalised	43.5 (31.9–55.9) b,dn = 62BE(2)	99.0 (97.0–99.7)n = 293BE(2), IT, NL
CLIA	Ortho Clinical Diagnostics, VITROS Immunodiagnostic Products Anti-SARS-CoV-2 IgG	IgG	Unk	93.4 (89.4–96.0) bn = 227DK, UK	99.7 (99.3–99.9)n = 1,420DK, UK, US
CLIA	Ortho Clinical Diagnostics, VITROS Immunodiagnostic Products Anti-SARS-CoV-2 Total Ab	Total Ab	Na	Nd	100.0 (99.5–100.0)n = 732DK, US
CLIA	Roche, Elecsys Anti-SARS-CoV-2	Total Ab	Hospitalised	85.7 (75.7–92.1) bn = 70CA, DE, NL	99.8 (99.7–99.9)n = 7,833AT, BE(3), CA, DE(5), DK, LU, NL, SE, SG, UK(2), US(5)
CLIA	Roche, Elecsys Anti-SARS-CoV-2	Total Ab	Mild/asymptomatic	91.7 (84.4–95.7) b,cn = 96NL, UK	Same as above
CLIA	Roche, Elecsys Anti-SARS-CoV-2	Total Ab	Unk	94.7 (93.3–95.7) cn = 1,351AT(2), BE(3), DE(2), DK, SE, SG, UK(2), US(2)	Same as above
LFIA, POC	SD BioSensor, Standard Q COVID-19 IgM/IgG Duo	IgG	Na	Nd	99.8 (99.3–99.9) cn = 1,254US(2)
LFIA, POC	SD BioSensor, Standard Q COVID-19 IgM/IgG Duo	IgM	Na	Nd	98.8 (98.0–99.3)n = 1,256US(2)
CLIA	Shenzhen New Industries Biomedical Engineering (SNIBE), Maglumi 2019-nCoV (SARS-CoV-2) IgG/IgM kit	IgG	Hospitalised	93.4 (85.5–97.2) b,cn = 76BE(2)	97.6 (96.8–98.3) dn = 1,744BE(2), CN(2), DK
CLIA	Shenzhen New Industries Biomedical Engineering (SNIBE), Maglumi 2019-nCoV (SARS-CoV-2) IgG/IgM kit	IgG	Unk	91.1 (89.2–92.6) dn = 1084CN, DK	Same as above
CLIA	Shenzhen New Industries Biomedical Engineering (SNIBE), Maglumi 2019-nCoV (SARS-CoV-2) IgG/IgM kit	IgG or IgM	Hospitalised	96.1 (89.0–98.6) bn = 76BE(2)	98.6 (96.4–99.5)n = 285BE(3)
CLIA	Shenzhen New Industries Biomedical Engineering (SNIBE), Maglumi 2019-nCoV (SARS-CoV-2) IgG/IgM kit	IgM	Hospitalised	93.4 (85.5–97.2) b,cn = 76BE(2)	99.2 (98.7–99.5) dn = 1,756BE(2), CN(2), DK
CLIA	Shenzhen New Industries Biomedical Engineering (SNIBE), Maglumi 2019-nCoV (SARS-CoV-2) IgG/IgM kit	IgM	Unk	67.8 (65.0–70.5) b,dn = 1084CN, DK	Same as above
CLIA	Shenzhen Yahuilong (YHLO) Biotech, SARS-CoV-2 IgG/IgM antibody detection kit	IgG	Na	Nd	99.0 (98.3–99.4)n = 1,313CN(2), DK, IT
CLIA	Shenzhen Yahuilong (YHLO) Biotech, SARS-CoV-2 IgG/IgM antibody detection kit	IgM	Na	Nd	98.7 (97.9–99.2) dn = 1314CN(2), DK, IT
CLIA	Siemens, Healthineers SARS-CoV-2 Total Assay on Atellica/ADVIA Centaur	Total Ab	Unk	96.7 (95.2–97.8) dn = 757DE, DK, UK	99.8 (99.5–99.9)n = 2,108DE(2), DK, UK
LFIA, POC	SureScreen Diagnostic, Covid-19 IgG/IgM Rapid Test Cassette	IgG	Na	Nd	99.0 (96.4–99.7)n = 198BE, NL
LFIA, POC	VivaChek Biotech, VivaDiag COVID-19 IgM/IgG Rapid Test	IgG	Unk	78.9 (69.7–85.9) bn = 95AU, US	98.2 (96.1–99.2)n = 334AU, BE, IT, NL, US
LFIA, POC	VivaChek Biotech, VivaDiag COVID-19 IgM/IgG Rapid Test	IgG or IgM	Hospitalised	100.0 (89.0–100.0) bn = 31BE, NL	97.5 (95.2–98.7)n = 324AU, BE, IT, US
LFIA, POC	VivaChek Biotech, VivaDiag COVID-19 IgM/IgG Rapid Test	IgG or IgM	Unk	80.0 (70.9–86.8) bn = 95AU, US	Same as above
LFIA, POC	VivaChek Biotech, VivaDiag COVID-19 IgM/IgG Rapid Test	IgM	Unk	80.0 (70.9–86.8) bn = 95AU, US	97.8 (95.6–98.9)n = 324AU, BE, IT, US
LFIA, POC	Xiamen Biotime Biotechnology, SARS-CoV-2 IgG/IgM Rapid Qualitative Test Kit	IgG	Na	Nd	98.0 (94.3–99.3)n = 150FI, US
CLIA	Xiamen Innodx Biotech, Antibody test kit for 2019-nCoV	IgG or IgM	Na	Nd	99.3 (98.0–99.8)n = 430CN(2)
LFIA, POC	Zhejiang Orient Gene Biotech, COVID-19 IgG/IgM Rapid Test Cassette	IgG	Hospitalised	96.7 (91.7–98.7) bn = 120BE, CH, NL	97.7 (96.1–98.7)n = 568BE, CH, FR, NL, SE
LFIA, POC	Zhejiang Orient Gene Biotech, COVID-19 IgG/IgM Rapid Test Cassette	IgG	Unk	92.4 (85.1–96.3) bn = 92FR, SE	Same as above
LFIA, POC	Zhejiang Orient Gene Biotech, COVID-19 IgG/IgM Rapid Test Cassette	IgM	Hospitalised	86.0 (77.5–91.6) bn = 93BE, NL	98.4 (96.3–99.3)n = 308BE, FR, SE
LFIA, POC	Zhejiang Orient Gene Biotech, COVID-19 IgG/IgM Rapid Test Cassette	IgM	Unk	82.6 (73.6–89.0) b,cn = 92FR, SE	Same as above
LFIA, POC	Zhuhai Livzon Pharmaceutical Group, Diagnostic Kit for IgM / IgG Antibody to Coronavirus (SARS-CoV-2) (Lateral Flow)	IgG	Hospitalised	86.4 (80.3–90.9) bn = 162CN(2), FR	98.0 (94.3–99.3)n = 150CN, FR, US
LFIA, POC	Zhuhai Livzon Pharmaceutical Group, Diagnostic Kit for IgM / IgG Antibody to Coronavirus (SARS-CoV-2) (Lateral Flow)	IgM	Hospitalised	75.9 (68.8–81.9) bn = 162CN(2), FR	99.3 (96.3–99.9)n = 150CN, FR, US

Ab: antibody; AT: Austria; AU: Australia; BE: Belgium; BR: Brazil; CA: Canada; CH: Switzerland; CLIA: chemiluminescence assay; CN: China; COVID-19: coronavirus disease; DE: Germany; DK: Denmark; ELISA: enzyme-linked immunosorbent assay; ES: Spain; FI: Finland; FR: France; GR: Greece; IT: Italy; JP: Japan; LFIA: lateral flow immunoassay; LU: Luxembourg; Na: not applicable; Nd: not determined, either due to no data or due to data from only one country or study; NL: The Netherlands; POC: point-of-care test; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2; SE: Sweden; SG: Singapore; TW: Taiwan; UK: United Kingdom; Unk: unknown or unclearly defined; US: United States.

a Sensitivity and specificity values given as value (confidence interval), number of samples (n = X), list of countries (number of studies per country if > 1). Value in bold if both confidence interval width ≤ 5% and value ≥ 95% (for sensitivity) or ≥ 98% (for specificity).

b Confidence interval width > 5%.

c Moderate study heterogeneity (50.0 ≤ I2 < 75.0%).

d High study heterogeneity (I2 ≥ 75.0%).

Only samples taken > 14 days post onset of symptoms are included, and ≤ 28 days post onset for IgM only as target. Rows are sorted alphabetically by test, target and case population.

For nucleic acid tests, results were restricted as for antibody tests (Table 3). Four tests, including one POC, had ≥ 95% positive agreement with a CI width ≤ 5%, and nine had ≥ 98% specificity. Study heterogeneity was low for all five sensitivity and all 15 specificity results with CI width ≤ 5%.

Table 3

Pooled positive agreement and specificity results for SARS-CoV-2 nucleic acid tests with confidence interval width ≤ 5% for either or both and based on at least two studies, up to 22 August 2020

Category	Test	Target	Case population	Positive agreementa	Specificitya
PCR	Altona Diagnostics, RealStar SARS-CoV-2 RT-PCR Kit 1.0	E	Unk	88.1 (80.4–93.1) bn = 101CH, FR, NL, US	100.0 (96.7–100.0)n = 112CH, NL
PCR	Altona Diagnostics, RealStar SARS-CoV-2 RT-PCR Kit 1.0	S	Unk	87.1 (79.2–92.3) bn = 101CH, FR, NL, US	100.0 (96.7–100.0)n = 112CH, NL
PCR	Altona Diagnostics, RealStar SARS-CoV-2 RT-PCR Kit 1.0	S or E	Unk	81.6 (75.8–86.3) b,cn = 207FR(3), NL	100.0 (98.4–100.0)n = 237FR, NL, UK
PCR	AusDiagnostics, Coronavirus Typing Assay	ORF1ab	Na	Nd	100.0 (98.5–100.0)n = 254AU, UK
PCR	BGI, Real-time fluorescent RT-PCR kit for detecting 2019 nCoV	ORF1ab	Unk	93.8 (88.7–96.7) bn = 146CH, JP, NL, PL	99.1 (95.1–99.8)n = 112CH, NL
PCR, POC	Cepheid, GeneXpert Xpert Xpress SARS-CoV-2	E or N	Unk	98.8 (97.3–99.5)n = 427BE, CH, CY, DE, FI, FR, NL, SE, US(5)	100.0 (82.4–100.0) bn = 18BE, CH, SE
PCR	CerTest Biotec, VIASURE SARS-CoV-2 Real Time PCR Detection Kit	N	Unk	96.8 (89.1–99.1) b,cn = 63CH, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	CerTest Biotec, VIASURE SARS-CoV-2 Real Time PCR Detection Kit	ORF1ab	Unk	93.7 (84.8–97.5) b,dn = 63CH, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	CerTest Biotec, VIASURE SARS-CoV-2 Real Time PCR Detection Kit	ORF1ab or N	Na	Nd	100.0 (98.2–100.0)n = 207NL, UK
PCR	DiaSorin, Simplexa COVID-19 Direct RT-PCR Kit	ORF1ab or S	Unk	97.8 (94.4–99.1)n = 180US(3)	Nd
PCR	Hologic, SARS-CoV-2 Assay (Panther Fusion System)	ORF1ab	Unk	98.3 (96.8–99.1)n = 525FR, US(6)	Nd
PCR	KH Medical, RADI COVID-19 Detection Kit and RADI COVID-19 Triple Detection Kit	RdRP	Unk	96.8 (89.1–99.1) b,cn = 63CH, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	KH Medical, RADI COVID-19 Detection Kit and RADI COVID-19 Triple Detection Kit	S	Unk	98.4 (91.5–99.7) bn = 63CH, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	Primerdesign, genesig Real-Time PCR CoVID-19 kit	RdRP	Unk	95.3 (89.4–98.0) b,cn = 106CH, NL, PL	100.0 (98.8–100.0)n = 307CH, NL, UK
PCR	R-Biopharm, Ridagene SARS-CoV2	E	Unk	100.0 (94.3–100.0) bn = 63CH, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	Roche, COBAS SARS-CoV-2 test	ORF1ab or E	Unk	98.8 (97.9–99.3)n = 1,125AT, CH, DE, FR, SI, US(5)	100.0 (90.8–100.0) bn = 38CH, FR
PCR	Seegene, Allplex 2019-nCoV assay	E	Unk	85.0 (75.6–91.2) b,dn = 80CH, FR, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	Seegene, Allplex 2019-nCoV assay	RdRP	Unk	91.3 (83.0–95.7) b,cn = 80CH, FR, NL	100.0 (96.7–100.0)n = 112CH, NL
PCR	Tibmolbiol, SARS-CoV (COVID19) E-gene	E	Unk	100.0 (94.4–100.0) bn = 65CH, UK	100.0 (98.5–100.0)n = 250CH, UK

AT: Austria; AU: Australia; BE: Belgium; CH: Switzerland; COVID-19: coronavirus disease; CY: Cyprus; DE: Denmark; E: envelope gene; FI: Finland; FR: France; JP: Japan; N: nucleoprotein gene; Na: not applicable; Nd: not determined, either because there were no data or because there were data from only one country or study; NL: The Netherlands; PL: Poland; S: spike gene; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2; SE: Sweden; SI: Slovenia; UK: United Kingdom; Unk: unknown or unclearly defined; US: United States.

a Positive agreement and specificity values given as value (confidence interval), number of samples (n = X), list of countries (number of studies per country if > 1). Value in bold if both confidence interval width ≤ 5% and value ≥ 95% (for positive agreement) or ≥ 98% (for specificity).

b Confidence interval width > 5%.

c Moderate study heterogeneity (50.0 ≤ I2 < 75.0%).

d High study heterogeneity (I2 ≥ 75.0%).

Rows are sorted alphabetically by test, target and case population.

The correlation between independently assessed clinical performance results and manufacturer-reported results is shown in Figure 2. The manufacturer-reported documents are listed in Supplementary Table S2. Only independently assessed results with CI width ≤ 5% are included. A total of 11 of 32 sensitivity and four of 33 specificity results reported by the manufacturer were significantly larger (p < 0.05).

Figure 2

Independently assessed vs manufacturer-reported clinical sensitivity and specificity per SARS-CoV-2 test, up to 22 August 2020 (n = 55)

SARS-CoV-2: severe acute respiratory syndrome coronavirus 2.

Significantly different (p < 0.05) results are highlighted. Independently assessed results limited to those with 95% confidence interval width ≤ 5%. The inset expands the 95–100% region.

Discussion

This review presents a comprehensive independent overview of clinical performance of commercially available nucleic acid and antibody tests 5 months into the COVID-19 pandemic. A substantial amount of previously unpublished data from European countries are included as well. By August 2020, there are numerous commercial tests for which sufficient performance data are available to allow calculation of clinical sensitivity or positive agreement, and specificity with narrow confidence interval ranges. It is reassuring that the clinical performance of several nucleic acid and antibody tests exceeded the minimum performance criteria. As time progresses, the list of tests with sufficient available performance data is expected to grow.

At the same time, the available evidence for point-of-care nucleic acid and antigen tests remains scarce, even though these tests can have substantial practical advantages for e.g. screening. We therefore recommend more emphasis on the validation of these tests, including as part of a testing algorithm, whereby the sensitivity and specificity of taking two tests with a number of days in between is assessed, and which can for example be useful to reduce the duration of a quarantine period.

The comparison between the independently assessed clinical performance data and manufacturer-reported clinical performance revealed that in particular sensitivity is frequently (34.4% of the cases in this study) significantly overestimated by the manufacturer. At a minimum, this emphasises that such independent assessments are clearly necessary. In the longer term, an explicit and proactive regulatory mechanism in Europe to compare available independently generated evidence on these tests against the manufacturer-reported values, coupled with appropriate regulatory action, would be useful. This could also be rewarding towards those manufacturers that do provide robust estimates of their product’s performance. The new in vitro diagnostic medical devices Regulation (EU) 2017/746 (IVDR), which will enter into force in May 2022, will impose more stringent requirements on clinical performance studies done by manufacturers. In addition, the IVDR will also regulate the use of lab-developed tests such as the in-house PCR tests developed for COVID-19 [25]. Because of the COVID-19 pandemic, the European Commission has recently proposed to modify the roll-out [26].

Limitations of our article include that most of the included studies had a substantial risk of bias in the sample selection, especially for the sensitivity panel, as established also in the assessments performed in the systematic reviews that we used as a source. Results were mainly based on hospitalised cases or poorly defined populations, whereas the population of interest often consists of symptomatic cases in general, or even asymptomatic cases, and differences in performance may exist depending on disease severity. Performance also varies depending on the type of specimen used, and our study design allowed for the inclusion of multiple specimen types in accordance with the instructions for use. This reflected to some extent clinical practice, but is also a contributing factor to study heterogeneity that we did not address here. Similarly, the pre-analytical steps such as RNA extraction can have a substantial effect on performance. These are often not specified in detail or several processes may be allowed according to the instructions for use, which can have contributed to study heterogeneity. While this review addresses a pressing need for actionable clinical performance data, ideally, the clinical performance should be assessed through prospective studies or clinical trials with a guaranteed unbiased sample selection for a clearly defined target population and intended use of the test. Given the difficulty of assessing and extracting the data from individual studies in a coherent way, we recommend that the Standard for Reporting of Diagnostic Accuracy Studies (STARD) should also be followed when publishing the results [27].

In this context, the selection of the reference test is particularly important with respect to reference negative samples. As described in some of the assessed studies, it should be avoided that index test results are considered as false positives while the samples are from actual cases; for this reason we excluded nucleic acid-negative samples from suspected COVID-19 patients altogether. We therefore expect little bias in the specificity results, except potentially from under- or overrepresentation of confounders. This is especially relevant for seroprevalence studies where, in a low-prevalence situation, in particular the specificity of the test needs to be well defined and high. On the other hand, sensitivity results using a nucleic acid test as reference should be interpreted with caution because the positive samples may exclude some actual cases.

Possibilities to improve the reference test can include testing - potentially only the false positives - with a second reference nucleic acid test preferably targeting different genes, testing more than one sample from the same patient including for antibodies at a later time point, testing samples from both upper and lower respiratory tracts, and sequencing the sample. The handling of intermediate index test results is an issue that needs to be described in studies and in general, these should be considered as positive results rather than as negatives or excluding them from the validation, since in clinical practice they would normally require further follow-up to confirm the positivity of the sample. Finally, the quality of the execution of the tests is also an important factor. For non-point-of-care tests, external quality assessment exercises using well validated standard reference materials remain a critical tool to detect and address such issues.

Conclusion

Given the study limitations, the authors and organisations contributing to this study in no way recommend the use of the listed commercial tests over other not listed commercial or in-house tests. The clinical performance of tests may also change over time as the virus population evolves. We recommend, however, continuous monitoring of clinical performance both in Europe and globally, which is key for reliable monitoring of the pandemic and which will also support vaccine and antiviral development. These results should be shared publicly in a timely manner.