Evaluation of a SARS-CoV-2 antigen-detecting rapid diagnostic test as a self-test: Diagnostic performance and usability.

To control the spread of the coronavirus disease 2019 (COVID-19) epidemics, it is necessary to have easy-to-use, reliable diagnostic tests available. The nasopharyngeal sampling method being often uncomfortable, nasal sampling could prove to be a viable alternative to the reference sampling method. We performed a multicentre, prospective validation study of the COVID-VIRO® test, using a nasal swab sampling method, in a point-of-care setting. In addition, we performed a multicentre, prospective, and usability study to validate the use of the rapid antigen nasal diagnostic test by laypersons. In March 2021, 239 asymptomatic and symptomatic patients were included in the validation study. Compared with reverse-transcription polymerase chain reaction on nasopharyngeal samples, the sensitivity and specificity of the COVID-VIRO® Antigen test combined with a nasal sampling method were evaluated as 96.88% and 100%, respectively. A total of 101 individuals were included in the usability study. Among these, 99% of the participants rated the instructions material as good, 98% of the subjects executed the test procedure well, and 98% of the participants were able to correctly interpret the test results. This study validates the relevance of COVID-VIRO® as a diagnostic tool from nasal specimens as well as its usability in the general population. COVID-VIRO® diagnostic performances and ease of use make it suitable for widespread utilization.

INTRODUCTION

The severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) virus causes the infectious respiratory disease known as coronavirus disease 2019 (COVID‐19). Since its outbreak in Wuhan, China, in late 2019, the disease has spread to the entire global population and the World Health Organization (WHO) conferred pandemic status on March 11, 2020. SARS‐CoV‐2 infection mainly causes pneumonia and upper/lower respiratory tract infection. Symptoms of COVID‐19 infection appear after an average incubation period of about 5.2 days. The most common early symptoms of COVID‐19 disease are fever, cough, and fatigue, but other symptoms include headache, sore throat, and even acute respiratory distress syndrome, leading to respiratory failure.

In an effort to control the spread of the epidemic, it is crucial to have highly sensitive and specific tests available. Indeed these tests are key to identify and manage COVID‐19 cases and implement control measures. As of now, the gold standard for the detection of SARS‐CoV‐2 infection is the reverse‐transcription polymerase chain reaction (RT‐PCR) method based on the molecular detection of the virus genetic material from a nasopharyngeal sample. This detection is highly sensitive and reliable but requires very specific and somewhat expensive material and equipment. Moreover, nasopharyngeal sampling requires training to be performed in a safe and reliable manner and the results can sometimes only be obtained after several days, depending on the laboratory. Finally, although nasopharyngeal sampling is generally safe, this procedure is not risk‐free, especially when performed in a repetitive and intensive manner. Indeed, in some rare instances, cerebrospinal leak and meningitis have been documented. , , Saliva sampling has been recently approved and proved to be as reliable and simpler than nasopharyngeal sampling but still does not provide a quick result. To have rapid, reliable, and simple tests, lateral‐flow immunoassays have been recently developed to detect the presence of the virus antigens from nasopharyngeal samples. These SARS‐CoV‐2 specific antigen assays rapid antigen diagnostic test (RADT) are a simple and fast alternative to the RT‐PCR method and are available in pharmacies. They take around 15 min to provide a result but because of the sampling method, they are not convenient for an at‐home testing strategy. From a public health perspective, self‐tests can usefully complement point‐of‐care tests by allowing more globally scaled testing. If reliable, self‐test allows individuals to obtain a quick result, thereby supporting the early detection and isolation of COVID‐19 cases. These tools can be essential to large scale distribution of COVID‐19 diagnostic tests, and as such, the FDA has already approved several antigen home tests and the French health authority (Haute autorité de Santé [HAS]) has defined the minimum performance requirements for these tests with an emphasis on the necessity of conducting real‐life studies. , , The diagnostic performance of COVID‐VIRO® (AAZ, LMB), an antigen‐based rapid detection test, has already been assessed on nasopharyngeal samples performed by trained professionals. The objective of this study was to evaluate the diagnostic performance of COVID‐VIRO® using a nasal swab sampling method when compared with the reference method, as well as its usability as a self‐test adapted for the general population.

MATERIALS AND METHODS

The study was evaluated and approved by the French ethics committee (Comité de Protection des Personnes Nord‐Ouest IV) in October 2020 and was notified to the French data protection authority. This study was conducted in accordance with the Declaration of Helsinki. This implies that all participants provided written informed consent before undergoing any study‐specific procedure. Two different study settings were used, one for the performance study and one for the usability/practicability study:

The performance study was set in the two COVID units of the Centre Hospitalier Régional d'Orléans: La Madeleine Hospital and La Source Hospital. The inclusion criteria were the following: adults volunteers (>18 years old) with mild to moderate symptoms lasting less than 7 days and not requiring immediate hospitalization (headache, fatigue, fever, sore throat, aches and pains, loss of smell and taste, etc.). The noninclusion criteria were: hospitalized patients, symptomatic patients with symptom duration more than 7 days, asymptomatic patients, or asymptomatic contact with a known case.

Regarding the usability study, the adult volunteers who participated were patients of a medical analysis laboratory (Drouot laboratory) as well as volunteer patients consulting in our infectious diseases department or hospitalized in our COVID unit (Orléans Regional Hospital). No specific inclusion/noninclusion criteria were applied.

In vitro diagnostic device under investigation

COVID‐VIRO® (AAZ‐LMB) is a lateral flow immuno‐chromatographic test that uses highly sensitive monoclonal antibodies to detect SARS‐CoV‐2 core antigen in a nasal sample. The test uses monoclonal antibodies to the SARS‐CoV‐2 core protein attached to the test area (T) on a nitrocellulose strip (Figure 1). A monoclonal antibody to the SARS‐CoV‐2 core protein labeled with colloidal gold is used as a freeze‐dried conjugate.

Figure 1

Visual appearance of the COVID‐VIRO® test cassette and representation of the potential results

In the test, SARS‐CoV‐2 antigens in the sample interact with monoclonal anti‐SARS‐CoV‐2 antibodies to form a colored antibody‐antigen complex. This complex migrates by capillarity across the membrane to the test line (T, Figure 1) where it is captured by the membrane‐bound monoclonal anti‐SARS‐CoV‐2 antibodies. A colored test line appears in the results window (T) if SARS‐CoV‐2 antigens are present in the sample. The intensity of the colored test line will vary depending on the amount of SARS‐CoV‐2 antigen present in the sample. If no SARS‐CoV‐2 antigen is present in the sample, no color will appear on the test line (T). The control line is used as a procedural control and should always appear in the control area (C) if the test procedure is performed correctly. The visual interpretation of the result can be performed after 15 min.

Comparator

The RT‐PCR test for SARS‐CoV‐2 was performed in the virology unit of the CHR Orléans, France. Nucleic acid extraction was performed with an automated sample preparation system MGISP‐960 (MGI). Real‐time PCR detection of SARS‐CoV‐2 RNA targeting the ORF1ab, S, and N genes was performed with the TaqPath V2 COVID‐19 Multiplex RT‐PCR kit (Thermo Fisher Scientific). Amplification was performed on QuantStudio5 (Applied Biosystems). The results of the assay were performed according to the manufacturer's instructions. The assay includes an internal RNA extraction control and an amplification control. The samples were analyzed taking into account the new positivity criteria of the French Microbiology Society's expert committee (version 4 of January 14, 2021), in particular taking into account the specific characteristics of the Thermo Fisher Scientific Kit used for the RT‐PCR measurement.

Methodology

Performance study

Upon arrival at one of the two study centers, patients were registered for nasopharyngeal RT‐PCR testing. Eligible patients were informed about the study. After consent to participate, the trained nurse performed the COVID‐VIRO® nasal swab test and recorded the test result on the previously filled‐in collection form without communicating it to the patient. Then, a nasopharyngeal swab is taken by the nurse for the RT‐PCR test by the hospital laboratory. The RT‐PCR test was performed using the TaqPath V2 COVID‐19 Multiplex RT‐PCR from Thermo Fisher Scientific including a variant screening. The RT‐PCR result was then communicated to the patient within 24 h and recorded in the patient's file.

Usability study

Substudy 1: Comprehension of instructions and test execution

Each participant was asked to consult the instructions for use (written or video, French language only, available in Supplementary File 1 and in the following link: https://www.youtube.com/watch?v=lP8sPqMFJkA) in full before carrying out the self‐test. Each person was then asked to use the nasal swab from the kit, take a deep nasal swab in both nostrils, dip the swab into the diluent pad, close the pad and diluent, close the diluent dispenser with the dropper cap, and place four drops of sample into the well and obtain a valid result (Supplementary File 1). Each participant was asked to comment on the different steps of the self‐test on a questionnaire (Table 1). The person performing the test was supervised by an observer (laboratory staff, nurse, or doctor) who gave a posteriori assessment of the performance of the various steps by filling in an evaluation form for each participant (Table 1).

Table 1

COVID‐VIRO® usability and supervisor's questionnaires

Usability questionnaire
Question
1. What is your opinion regarding the written instructions for the COVID‐VIRO® test?	□ Poor		□ Good			□ Very good				□ Excellent
2. What is your opinion regarding the instructional video for the COVID‐VIRO® test?	□ Poor		□ Good			□ Very good				□ Excellent
3. What is your opinion regarding the execution of the nasal swab for the COVID‐VIRO® test?	□ Difficult			□ Easy					□ Very easy
4. What is your opinion regarding the execution of the procedures COVID‐VIRO® test?	□ Difficult			□ Easy					□ Very easy

Did the participant request the supervisor's assistance?

What is your opinion on the participant's performance of this test?

Substudy 2: Interpretation of test results

The usability study also included a test result interpretation exercise during which the observer instructed the participant to randomly select 1 of 5 self‐tests (1 negative, 2 positive, and 2 invalid, Figure 2), read it, and give his/her interpretation of the result. The participant interpretation as well as his/her opinion on the ease of interpretation was collected on a questionnaire (Table 2).

Figure 2

Sample tests were used for the interpretation study. Cassettes A is a negative test, B and C are invalid tests, and samples D and E are positive tests (D faint positive and E strong positive)

Table 2

Interpretation questionnaire

Question
3. What is the result of the test?	□ No band	□ 1 band	□ 2 bands
4. How did you interpret this result?	□ Negative	□ Positive	□ Inconclusive
5. How would you describe the reading and interpretation steps of the test?	□ Difficult	□ Easy	□ Very easy
6. How would you describe the ease of interpretation of the COVID‐VIRO® nasal self‐test?	□ Difficult	□ Easy	□ Very easy

Data analysis

Populations were described in terms of percentage, mean, standard deviation, range, and median values. The test data were analyzed in the Department of Infectious. True positive (TP) results were defined as positive individuals according to the reference method considered positive by the COVID‐VIRO® test, FP (false positive) results were negative individuals according to the reference method considered positive by COVID‐VIRO® test, FN (false negative) were positive individuals according to the reference method, considered negative by the COVID‐VIRO® test and VN (true negative) were defined as negative individuals according to the reference method considered as negative by the COVID‐VIRO® test. The specificity (Sp), sensitivity (Se), positive predictive value (PPV; the probability that subjects with a positive screening test truly have the disease), and negative predictive value (NPV, probability that subjects with a negative screening test truly don't have the disease) of the COVID‐VIRO® test compared with the reference test (RT‐PCR) were calculated according to the following formulas:

Sp (%) = 100 × [TN/(TN + FP)]

Se (%) = 100 × [TP/(TP + FN)]

PPV (%) = 100 × [TP/(TP + FP)]

NPV (%) = 100 × [TN/(TN + FN)]

Confidence intervals for sensitivity were obtained with the Wilson score method.

Populations were described in terms of absolute number and percentage.

RESULTS

Performance study

A total of 239 patients were recruited from the two COVID sites in the city of Orléans. These participants were distributed as follows: 94/239 (39%) from La Source Hospital and 145/239 (61%) from La Madeleine Hospital. Of these 239 patients, five were excluded as their RT‐PCRs were considered doubtful according to the classification criteria of the French Microbiology Society. Specifically, four of those five patients are RT‐PCR positive for the N gene but with Ct value above 32 (26, 34, 37, and 37, respectively) compared with the mean Ct value of 24. The last sample was RT‐PCR‐positive for the ORF Gene with a Ct value of 38 whereas the mean Ct value was 25. Those samples are positive from a laboratory standpoint but are excreting very low (Ct > 32 following the French guideline ) level of SARS‐CoV‐2 virus which means that they are no more contagious. Therefore, according to the French guideline, the lab can either consider them weak positive or negative. Consequently, they were removed from the analysis. The study population thus comprises 234 patients. The sex ratio of the study population was 0.70 (98 men and 141 women). The median age was 34 years (mean: 38 years, range: 24 years). Among this population, the median duration of symptoms before the sampling date was 2 days (mean: 2.56, range: 7). Two groups were constituted according to the RT‐PCR test results: 32 positive and 202 negative samples. Of the 32 positive samples, six were confirmed positive for the English variant (VOC 2020‐12/01) and one was suspected. The results are presented in Table 3.

Table 3

Performance of the COVID‐VIRO® antigenic rapid test in the overall population

		Total N = 234
True positive	N	234
	VP	31 (13.2%)
False positive	N	234
	FP	0 (0.0%)
False negative	N	234
	FN	1 (0.4%)
True negative	N	234
	VN	202 (86.3%)

In the overall population, the COVID‐VIRO® test was performed as follows:

Sensitivity: 96.88% (95% confidence interval [CI]: 83.78%–99.92%).

Specificity: 100% (95% CI: 98.19%–100.00%

Positive predictive value: 100%

Negative predictive value: 99.5% (95% CI: 96.70%–99.93%)

Concordant results between RT‐PCR and COVID‐VIRO® were observed for 233 (99.6%) patients. The only discrepant sample was RT‐PCR‐positive for the N gene with a Ct value of 31, below the exclusion threshold of 32.

Usability study

Substudy 1: Comprehension of instructions and test execution

A grand total of 101 subjects participated in the study. None of them was excluded, therefore the analysis was carried out on 101 participants. The repartition of the subjects is given in Table 4.

Table 4

Usability study population characteristics

Investigational site	N (%)
Drouot Laboratory	64 (63.4)
CHR Orléans infectious diseases unit	31 (30.7)
CHR Orléans COVID unit	6 (5.9)

Among these 101 participants, only one person did not obtain a valid result, 94 (93.1%) obtained a valid negative result, and 6 (5.9%) a valid positive result. These results show that 99.0% of the untrained individuals that used the COVID‐VIRO® self‐test were able to obtain a valid interpretable result (with one visible control band). The invalid test was the consequence of a bad execution (only three drops of the sample were deposited on the cassette instead of four as instructed).

Regarding the quality of the COVID‐VIRO® self‐test instructions, 84.16% of the participants found the quality of the written instructions to be very good or excellent and only one participant chose to respond “poor” to that question. Similarly, 78.22% found the instructional video very good or excellent. The negative answer was never given for that question. These results show that 99% and 100% of the participants (written instructions and video respectively) found the instructions good or better. Regarding the ease of execution of the COVID‐VIRO® sample collection (nasal swab), 100% of the participants found it easy or very easy (26.7% and 73.3%, respectively). Likewise, COVID‐VIRO® self‐test procedures were considered easy or very easy to perform by all the participants (33.7% easy and 66.3% very easy). During the execution of the test, every participant was supervised by a trained professional (physician, nurse, or laboratory personnel) and had the opportunity of requesting his/her assistance. Only 6/101 (5.9%) participants requested the supervisor's assistance. Then, the supervisor had to rate the execution quality of the test procedures by the participant. Only 2/101 (2%) of the subjects were considered as having poorly executed the test procedures whereas 99 (98%) were rated as good or very good (36/101—35.6% and 63/101—62.4%, respectively). The most frequent observation was that some test users shake the swab inside the dilution tube without pressing it against the tube wall.

Substudy 2: Interpretation study

All the subjects that participated in substudy 1 were included in substudy 2 which consisted of randomly choosing one test out of five control tests and in correctly reading and interpreting it. Among all the participants, 27 sorted a negative test, 28 an invalid test, and 46 a positive test (strong or faint). Overall 98% of the participants (99/101) interpreted the sorted test correctly while 2% (2/101) misinterpreted the test. However, these two subjects read the test correctly (identified the correct number of bands), but failed to interpret it. Of these two participants, one interpreted an invalid test as negative and one interpreted a positive test as invalid. It should be noted that the positive test that was interpreted as an invalid test showed a strong test band (sample E, Figure 2). The vast majority of the participants (98/101—97%) found that the COVID‐VIRO® reading and interpretation steps were easy or very easy (37/101—36.6% and 61/101—60.4% respectively) while only 3/101 (3%) rated these steps as difficult.

DISCUSSION

A previous prospective study of the COVID‐VIRO® rapid antigenic test (AAZ‐LMB) diagnostic performance in real‐life conditions on nasopharyngeal samples was conducted in October 2020. At the time, the performance of the test was very similar to the reference method as the specificity and sensitivity were 100% and 96.6%, respectively, which placed it above the requirements of the French National Authority for Health (HAS) (sensitivity ≥ 80% and specificity ≥ 99%)  and the World Health Organization (WHO). The current study was carried out using a swab specifically adapted to nasal sampling, to make the test practicable by a layperson. The results showed that this sampling method did not alter the test's performances as the specificity came out again at 100%, and the sensitivity was evaluated at 96.8%. Therefore, this study demonstrates the diagnostic performance of the COVID‐VIRO® test on a nasal sample. This result is in line with another study showing similar results with a rapid antigen test from a different manufacturer. Other studies have compared the performances of RADTs performed by untrained persons versus RADTs performed by healthcare professionals and found that although the performances of the tests were lower when performed by an untrained individual or in a home setting, they were still within an acceptable range and therefore the development of RADT self‐test needed to be encouraged and further investigated. ,

In our study, the prevalence of COVID‐19 cases was 14%, which reflects the situation in France at the time. In this context, the negative and positive predictive values were very high (99.5% and 100%, respectively). However, as the PPV of a test decreases with decreasing prevalence, it would be interesting to assess the performance of the test in a population with a lower COVID‐19 prevalence. The negative predictive value also decreases in a low prevalence setting but to a somewhat lesser extent. However, a low NPV could be detrimental to the epidemic containment efforts as individuals who tested negative with a RADT could participate in social mixing or showing careless behavior based on the fact that they have tested negative.

In addition to the performance assessment, we performed a usability study following the FDA recommendations. The participant was asked to read or watch the test instructions and perform all the procedures while being supervised, to assess whether the participant was able to correctly perform the test on his/her own and interpret it accurately. Due to the setting of this trial (private and public laboratories in two different cities), a large variety of persons of different ages, education levels, and socio‐economic backgrounds was included in the study. This resulted in a representative sampling of the French general population. The test instructions, either written or video, were favorably regarded as well written/made and comprehensible by the vast majority of the participants, showing that the legibility of the documents provided with the COVID‐VIRO® test is particularly high and should be accessible to a wide range of persons. The results showed that the COVID‐VIRO® test is very practicable as only a small fraction of the participants were not able to obtain valid and interpretable results. Moreover, almost all the participants were able to perform the test procedures without requesting the supervisor's assistance and their execution was highly rated by the supervisors. In terms of satisfaction, when asked their opinion about COVID‐VIRO®'s ease of use, all the participants declared that the sample collection or the subsequent testing procedures were easy or very easy to perform. Nevertheless, a very small fraction of the participants found the reading and the interpretation steps somewhat difficult. This, however, did not affect their capacity to correctly interpret the test results with one notable exception. Taken together, these results show that the COVID‐VIRO® test is highly adapted for use by a layperson.

To our knowledge, this is the first usability study of a nasal rapid antigen test combining an easy sampling method (nasal swab) with a highly accurate diagnostic test that should allow the health authorities to consider wider use of this test (children, iterative screening, …) by health professionals as well as its adaptation in self‐test version for use by laymen. Nasal tests are of particular interest because they can mitigate the risks and adverse effects of nasopharyngeal tests (discomfort, pain, deeper lesions, etc.) and are well suited for situations when no trained professional is available such as social gatherings, offices, and schools. However, it is important to consider that, as pointed out by the European Medicine Agency, the use of RADTs by the general population raises the concern of underreporting. Therefore, in the interest of the global effort to contain the spread of the virus, it could be necessary to include a set of instructions encouraging test users to report any positive results.

CONFLICT OF INTERESTS

The authors declare that there are no conflicts of interest. The authors alone are responsible for the content and the writing of the article.

AUTHOR CONTRIBUTIONS

Experimental strategy design: Thierry Prazuck, Raphael Serreau, and Nino G. Cassuto. Experiments: Anne Gravier, Mathilda Colin, Aurelie Thiellay, Daniela Pires‐Roteira, and Sandra Pallay. Data Curation, Thierry Prazuck. Manuscript writing: Thierry Prazuck, Laurent Hocqueloux, and Raphael Serreau. Manuscript editing: Thierry Prazuck and Nino G. Cassuto.

Supporting information.

Click here for additional data file.