Implementation of the cobas Liat influenza point-of-care test into an emergency department during a high-incidence season: a retrospective evaluation following real-world implementation.

The cobas Liat influenza A/B and respiratory syncytial virus (RSV) assay (Liat) was used in the adult emergency department of a large London hospital from 21st January 2018 to 14th April 2018. Influenza was detected in 308 of 1027 (30%) samples tested; influenza A in 157 (15.3%), influenza B in 149 (14.5%) and RSV in 28 (2.7%). When compared against Fast Track Diagnostics Respiratory Pathogens 21 multiplex polymerase chain reaction and Cepheid Xpert Xpress Flu/RSV assay, Liat performance for the detection of influenza A or B was: sensitivity 85% [95% confidence interval (CI) 76-92)], specificity 98% (95% CI 97-99), negative predictive value 94% (95% CI 92-96) and positive predictive value 95% (95% CI 91-97).

Introduction

Identifying influenza in the emergency department (ED) is important to inform decisions about patient management and infection prevention and control. Traditionally, patients have been isolated or treated on clinical suspicion until laboratory-based results become available 24–48 h later. Unfortunately, typical symptoms of influenza, such as rhinorrhoea and sore throat, are present in fewer than one-third of patients hospitalized with influenza, making clinical diagnosis challenging [1]. Conversely, overdiagnosis of unconfirmed influenza can also be harmful by prompting unnecessary use of single rooms during winter months when they are precious [2]. Rapid and reliable diagnostic tests are therefore essential.

Influenza rapid antigen tests have been used in outpatient settings, but their poor sensitivity (as low as 50–70%) makes them unsuitable for use in patients attending the ED [3]. Newer, point-of-care test (POCT) polymerase chain reaction (PCR)-based commercial tests are now available. One such test is the cobas Liat influenza A/B and respiratory syncytial virus (RSV) assay (Roche, Basel, Switzerland) (Liat), an automated multiplex PCR system with time to result of 20 min [4]. Previous studies have found its performance to be excellent, with sensitivity/specificity in the region of 100%/97.1–100% for influenza A and 97.8%–100/99.5–99.7% for influenza B when fresh prospectively collected samples are tested [3], [5]. However, when testing is performed outside laboratories in the ED, there is increased risk of errors such as testing the wrong sample because of transcription errors or poor use of the equipment by non-laboratory staff. Evaluations of real-world deployments outside laboratories are key to assessing the use of POCT in effective patient pathways.

Liat POCT testing was used in the ED of a large tertiary teaching hospital during a year of high incidence for influenza (9734 patients in England were hospitalized with confirmed influenza, compared with 1559 the year before) [6], [7], with a high proportion of influenza B demonstrated (49.0% vs 4.8%) [6], [7]. This provided a unique opportunity to evaluate the implementation of Liat in an ED during a year with high incidence of both influenza A and B. This paper evaluates the real-world performance of Liat. The impact of Liat on infection prevention and control and clinical outcomes is discussed in an accompanying paper [8].

Methods

Design and study population

This evaluation was conducted at St George's Hospital, a large tertiary teaching hospital in south-west London, during the 2017/18 influenza season. Liat POCT was operational in the adult ED from 21st January 2018 to 14th April 2018. Testing policy is provided as an appendix (see online supplementary material). All adults admitted with possible influenza received viral throat swab POCT to inform isolation policy and patient treatment, and assist with infection prevention and control. POCT was not used to assist with ED admission avoidance. The low prevalence of RSV in the tested population did not allow for meaningful evaluation of the RSV assay. Sample collection, testing and recording of results in an ED log book were performed by ED nursing staff who were trained and monitored by laboratory staff and the infection prevention and control team.

Negative samples were routinely retested in the diagnostic laboratory by the Fast Track Diagnostics Respiratory Pathogens 21 multiplex real-time PCR (Fast Track Diagnostics, Esch-sur-Alzette, Luxembourg) (rPCR) assay. Positive samples were not routinely retested but many patients had subsequent samples taken soon after admission. These were tested either by rPCR or the Cepheid Xpert Xpress Flu/RSV assay (Cepheid, Sunnyvale, CA, USA) (fPCR) where influenza was specifically queried or a rapid result was required.

Outcomes

Liat results were compared against rPCR/fPCR results from either the original sample or a confirmatory sample (taken within 24 h of Liat) to determine sensitivity and specificity after real-world implementation.

Sample testing

Both rPCR and fPCR testing were performed by trained diagnostic laboratory staff. rPCR testing was performed using the Roche Flow solution. This comprises a Hamilton primary sample handler and PCR set-up system (Hamilton Company, Reno, NV, USA) and Roche MagNA Pure 96 nucleic acid extraction system and Light Cycler 480 real time PCR system (Roche). The rPCR assay can detect 21 targets including influenza A, influenza B and RSV. fPCR is a cartridge-based molecular device capable of detecting influenza A, influenza B and RSV.

Data collection

A copy of the Liat result log book was entered into an Excel (Microsoft Corp., Redmond, WA, USA) spreadsheet. This was cross-referenced against rPCR/fPCR data extracted from the laboratory information management system. rPCR/fPCR results were only used if performed within one day of Liat.

Statistical analysis

Clopper–Pearson exact confidence intervals (CI) were applied to sensitivity and specificity values, and negative predictive values (NPV) and positive predictive values (PPV) were calculated with standard logit CI using the Medcalc online calculator [9]. NPV/PPV were estimated from the calculated sensitivity/specificity in the analysed population and known prevalence in the tested population. P-values were calculated using a Chi-squared test (all data were categorical).

By definition, a ‘false-negative’ Liat was where the Liat result was negative but the rPCR/fPCR result (taken within 24 h) was positive. A ‘false-positive’ Liat was where the Liat result was positive but the rPCR/fPCR result was negative. For the modified analysis, samples were excluded if a second rPCR/fPCR (taken within seven days for false-positive results and 24 h for false-negative results) validated the original Liat result. Samples were also excluded if documentation indicated that the record of the Liat result in the log book was erroneous. Cycle threshold (Ct) values for false-negative Liat results were examined.

Ethics

As service evaluation of a new service, solely using existing data collected during routine clinical practice, formal approval of the protocol by an ethics review board was not necessary.

Results

Over three months of operation, 1074 samples were tested by Liat in the ED. Of these, 47 samples were excluded from the analysis: 28 because the results were illegible/not recorded in the log book and 19 because of an invalid result. This left 1027 samples for inclusion in the main analysis (tested population). Invalid results were those where Liat displayed a result as ‘indeterminate’ or ‘invalid’, which occurred when the internal analyser determined that a PCR curve was abnormal or the internal positive control was not detected. The finding that 19 of 1074 (1.8%) results were invalid was the same as another recent study [3].

In 672 (65.4%) cases, an rPCR/fPCR was performed within one day to allow evaluation of the Liat result (analysed population): 40 vs fPCR and 632 vs rPCR. Inclusion in the analysed population was more likely for negative results [580/691 (83.9%)] than positive results [92/308 (29.9%); P < 0.0001], reflecting the policy to routinely retest negative samples by rPCR. This meant that the prevalence of influenza was lower in the analysed population than the tested population (13.5% vs 30%; P < 0.0001).

Influenza was detected in 308 (30%) of the tested population: influenza A in 157 (15.3%), influenza B in 149 (14.5%) and mixed influenza A and B in two (0.2%). RSV was detected in 28 (2.7%) and no virus was detected in 691 (67.2%). Results are provided in Table I .

Table I

Performance of the cobas Liat influenza A/B and respiratory syncytial virus (RSV) assay against Fast Track Diagnostics Respiratory Pathogens 21 multiplex real-time PCR (rPCR)/Cepheid Xpert Xpress Flu/RSV assay (fPCR)

	Sensitivity (%, 95% CI)	Specificity (%, 95% CI)	Estimateda negative predictive value% (95% CI)	Estimateda positive predictive value% (95% CI)
Modified
Influenza A or B	76/89 (85.4, 76.3–92.0)	565/576 (98.1, 96.6–99.0)	94.0 (91.8–95.7)	95.1 (90.8–97.3)
Influenza A	44/51 (86.3, 73.7–94.3)	452/456 (99.1, 97.8–99.8)	97.6 (96.1–98.5)	94.5 (88.1–97.7)
Influenza B	33/39 (84.6, 69.5–94.1)	623/630 (98.9, 97.7–99.6)	97.4 (96.0–98.4)	92.9 (86.0–97.8)
Unmodified
Influenza A or B	76/91 (83.5, 74.3–90.5)	565/581 (97.2, 95.6–98.4)	93.2 (91.0–95.0)	92.7 (88.3–95.7)
Influenza A	44/53 (83.0, 70.2–91.9)	452/458 (98.7, 97.2–99.5)	97.0 (95.2–98.2)	92.0 (83.0–96.0)
Influenza B	33/39 (84.6, 69.5–94.1)	623/633 (98.4, 97.1–99.2)	97.4 (95.9–98.4)	90.0 (82.6–94.6)
RSV	7/9 (77.8, 40.0–97.2)	658/663 (99.2, 98.2–99.8)	99.4 (98.6–99.7)	73.0 (53.0–87.4)

CI, confidence interval.

From calculated sensitivity/specificity and known prevalence in the tested population – i.e. influenza A or B (30%), influenza A (15.3%), influenza B (14.5%), RSV (2.7%).

For influenza A, there were six false-positive results (five vs rPCR, one vs fPCR) and nine false-negative results (nine vs rPCR). For influenza B, there were 10 false-positive results (eight vs rPCR, two vs fPCR) and six false-negative results (six vs rPCR). All false-negative results were detected by rPCR at high Ct values [median 33, interquartile range (IQR) 30–35]. All true-positive Liat results detected by rPCR (N = 69) had Ct values ≤ 35 (median 28, IQR 24–30).

For the modified analysis, two false-positive results and two false-negative results were removed because subsequent rPCR/fPCR testing validated the original Liat result. Another two false-positive results were removed because the Liat result was recorded as ‘POCT negative’ elsewhere, suggesting a transcription error in the log book. From the modified analysis, sensitivity for influenza A or B was 85.4% (95% CI 76.3–92.0) and specificity was 98.1% (95% CI 96.6–99.0) (Table I).

Of the 565 true-negative Liat results that were tested using rPCR, no virus was detected in 431 (76.3%). In the remaining 134, the following were detected in decreasing order of frequency: coronavirus, rhinovirus, metapneumovirus, adenovirus, parainfluenza types 3 and 4, RSV, enterovirus/paraechovirus, bocavirus and Mycoplasma pneumoniae.

Discussion

To the authors' knowledge, this is the first real-world evaluation of implementation of Liat into the routine work of an ED. Liat appeared to be less sensitive than in earlier studies [3], [5] for both influenza A and B (86.3% and 84.6%, respectively). Importantly, negative Liat patients were retested more often than positive Liat patients (83.9% vs 29.9%; P < 0.0001). This means that the positive Liat results analysed may disproportionally reflect those patients where the clinicians doubted the validity of the result and so requested a second sample. Perhaps also likely is that this represents the performance of the Liat assay outside the research setting. Errors in sample collection, testing and result recording are more likely when performed alongside normal clinical duties. Rather than a weakness, this is viewed as a strength of this study as it demonstrates the actual performance of Liat ‘in the field’.

Despite the lower estimates of sensitivity reported using this methodology, the performance of Liat at this prevalence of influenza was good. For influenza A/B, Liat demonstrated an NPV of 94.0% (95% CI 91.8–95.7) despite a population of high influenza prevalence (30%) (Table I). This provides greater confidence that during a high-incidence influenza season, a negative Liat result can be relied upon to withhold antiviral treatment and influenza infection prevention and control measures. Conversely, the PPV of 95.1% (95% CI 90.8–97.3) is sufficient to institute these measures.

The PPV for influenza B was 92.9% (95% CI 86.0–97.8) in the study population, with a relatively high prevalence (14.5%). This suggests that Liat should not be used on its own to cohort influenza B patients together in future years where there is a lower incidence of influenza B.

The fact that all false-negative results were detected by rPCR at high Ct values implies that Liat may be less accurate with samples containing lower viral loads. This is further evidenced by the fact that all true-positive Liat results detected by rPCR had Ct values ≤ 35. Older age and underlying co-morbidity are associated with lower viral loads [10]. The study policy was to test adults requiring hospital admission, which may mean that the cohort differs in these characteristics from published studies [3], including studies of children [5].

It is reasonable to use fPCR as the reference standard alongside rPCR because previous studies have demonstrated their comparable performance. For example, one study comparing fPCR against a real-time RT-PCR similar to rPCR found 100 % agreement between the two assays for detection of influenza A and influenza B, and 99.7 % agreement for negative results [11]. A subanalysis comparing Liat against either rPCR or rPCR alone is as follows: vs fPCR (40 samples), influenza A or B sensitivity was 8/8 (100%) (95% CI 63.1–100) and specificity was 30/32 (93.8%) (95% CI 79.2–99.2); vs rPCR (632 samples), influenza A or B sensitivity was 68/83 (81.9%) (95% CI 72.0–89.5) and specificity was 535/549 (97.5%) (95% CI 95.8–98.6).

The retrospective nature of this verification study means that, in many instances, the Liat result is being compared against a confirmatory sample (taken within 24 h of Liat), perhaps after initiation of antiviral treatment. It was not practical or cost-efficient to prospectively retest original samples in real time. This raises the possibility that samples may have become negative in the interim between the two tests. It should be noted, however, that if Liat gave a positive result and the confirmatory sample gave a negative result, this would represent a false-positive result in the analysis. False-positive results only impact specificity (not sensitivity), and this analysis found Liat to have excellent specificity. The key limitation of this approach is the potential for a selection bias stemming from the fact that only 65.4% of Liat results were retested, allowing analysis of the Liat result. Despite this limitation, which may lead to an underestimate of sensitivity, it is felt that the analysis provides useful insights into the performance of the test in a real-world setting, with fresh samples collected by staff prospectively during routine clinical practice being tested in real time during a high-incidence year for both influenza A and B.