Point-of-care diagnostic tests for influenza in the emergency department: A cost-effectiveness analysis in a high-risk population from a Canadian perspective.

BACKGROUND: Our objective was to assess the cost-effectiveness of novel rapid diagnostic tests: rapid influenza diagnostic tests (RIDT), digital immunoassays (DIA), rapid nucleic acid amplification tests (NAAT), and other treatment algorithms for influenza in high-risk patients presenting to hospital with influenza-like illness (ILI).
METHODS: We developed a decision-analytic model to assess the cost-effectiveness of diagnostic test strategies (RIDT, DIA, NAAT, clinical judgement, batch polymerase chain reaction) preceding treatment; no diagnostic testing and treating everyone; and not treating anyone. We modeled high-risk 65-year old patients from a health payer perspective and accrued outcomes over a patient's lifetime. We reported health outcomes, quality-adjusted life years (QALYs), healthcare costs, and net health benefit (NHB) to measure cost-effectiveness per cohort of 100,000 patients.
RESULTS: Treating everyone with no prior testing was the most cost-effective strategy, at a cost-effectiveness threshold of $50,000/QALY, in over 85% of simulations. This strategy yielded the highest NHB of 15.0344 QALYs, but inappropriately treats all patients without influenza. Of the novel rapid diagnostics, NAAT resulted in the highest NHB (15.0277 QALYs), and the least number of deaths (1,571 per 100,000). Sensitivity analyses determined that results were most impacted by the pretest probability of ILI being influenza, diagnostic test sensitivity, and treatment effectiveness.
CONCLUSIONS: Based on our model, treating high-risk patients presenting to hospital with influenza-like illness, without performing a novel rapid diagnostic test, resulted in the highest NHB and was most cost-effective. However, consideration of whether treatment is appropriate in the absence of diagnostic confirmation should be taken into account for decision-making by clinicians and policymakers.

Introduction

The influenza virus causes epidemics of acute respiratory illness, resulting in significant morbidity and mortality every year. Globally, annual epidemics contribute to approximately 3 to 5 million individuals developing severe illnesses, and 290,000 to 650,000 respiratory-related deaths [1]. Young children, older adults and patients with chronic or immunocompromising conditions are the groups at highest risk of infection, hospitalization, and severe outcomes (e.g., requiring critical care, or death) [2].

Even in high-income nations such as Canada, annual influenza results in approximately 12,000 (39 per 100,000) hospitalizations and 3,500 (11.3 per 100,000) deaths [3]. In individuals age 65 and over, the hospitalization rate increases to a six-year (2013 to 2019) average of 143 per 100,000 [4]. In a Cochrane review, oseltamivir, the most commonly administered antiviral neuraminidase inhibitor (NAI), was shown to reduce the time to symptom alleviation by 16.8 hours in adults and 29 hours in children when promptly administered, emphasizing the importance of rapid and accurate diagnosis of influenza [5]. Clinical judgement to diagnose influenza can be difficult due to the non-specific symptoms relative to other acute respiratory infections [6], making rapid diagnostic tests a valuable option to appropriately diagnose, with a high degree of accuracy, and start antiviral treatment [7, 8].

Until recently, the only method to confirm an influenza infection was the use of reverse transcriptase polymerase chain reaction (RT-PCR) [9]. However, RT-PCR tests are typically run in batches, resulting in turnaround times that can extend up to 24 hours and longer, thereby preventing health-care providers from using the diagnostic information to guide initial treatment. Several rapid diagnostic tests have been developed: traditional rapid influenza diagnostic tests (RIDT), digital immunoassays (DIA), and rapid nucleic acid amplification tests (NAAT). Each of these tests is relatively simple to administer and provides results within 30 minutes. In a recent meta-analysis conducted by Merckx and colleagues, all three categories of tests were associated with high overall (adults and children) specificities (>98%) for influenza A and B, with NAATs having the highest sensitivity (92% influenza A, 95% influenza B), followed by DIAs (80% influenza A, 77% influenza B) and RIDTs (54% influenza A, 53% influenza B) for the combined adult and children population [10].

While rapid diagnostic tests offer quicker results that could potentially inform treatment decisions, the cost-effectiveness of these tests in clinical environments (e.g. hospitals) is uncertain. The objective of this study was to assess the cost-effectiveness of these testing strategies for high-risk patients presenting to the emergency department (ED) with influenza-like-illness (ILI) using decision-analytic modeling from a health payer perspective. Evidence generated from this study can support seasonal influenza management and guide decisions about applying rapid testing for influenza in clinical practice. We did not consider the importance of distinguishing between influenza and COVID-19 or a future pandemic virus in this study.

Methods

Model structure

We developed a decision-analytic model to assess the cost-effectiveness of diagnostic testing strategies on the outcomes of 65 year-old patients presenting to the emergency department (ED) with symptoms or signs that could be suggestive of influenza, which we hereafter label as ILI (Fig 1). We modeled 65 year-old patients in the base case to approximate populations at high-risk for severe influenza illness, recognising that this is not the only high-risk population. We assessed five overall strategies: (1) no test, and do not treat patients (“Don’t Treat Anyone”), (2) no test and treat everyone (“Treat Everyone”), (3) rapidly test all patients with ILI and treat with NAI, (4) Batch PCR test, and treat until results become available (“Batch PCR–Treat”), and (5) Batch PCR test, but do not treat until results are available (“Batch PCR–Wait”). For strategy (3), four diagnostic methods were evaluated: (A) “RIDT”, (B) “DIA”, (C) “NAAT”, and (D) “Clinical Judgement”. All modelling and analyses were conducted using TreeAge Pro 2019 (TreeAge Software, Inc., Williamstown, MA).

Fig 1

Model schematic of decision-analytic model for high-risk patients presenting at ED.

The model simulated a disease history and care pathway for patients presenting to the ED with ILI, altering the probability of hospitalization, ICU admission and death based on the timing and appropriateness of treatment (Fig 1). Patients present to the ED with ILI and have a prior probability of influenza or another acute respiratory infection based on the known community prevalence of influenza. Cases of influenza were further defined as influenza A or B based on surveillance data. Patients testing positive (true or false positive) were assumed to all receive a regimen of NAI treatment, while patients testing negative did not. It was assumed that patients with a false negative result did not receive NAI therapy at any point during their hospitalization. While some individuals testing negative may still receive NAI therapy, we assumed that they did not in the base-case. However, we examined this assumption in a scenario analysis where 50 percent of individuals testing negative for influenza still received NAI therapy. From the ED, patients could discharged home, or be admitted to the general ward or ICU, with cases admitted to the ICU having a higher probability of mortality. True positives treated with NAI were assigned a lower risk of mortality (see Table 1), as well as a decrease in the duration of symptoms based on data from recent meta-analyses [5, 8].

Table 1

Key parameters for base-case.

Variable	Base-case value	Range	Source
Diagnostic Tests
Influenza A
Sensitivity. Adults
RIDT	0.426	0.348–0.509	Merckx 2017 [10]
DIA	0.754	0.666–0.826	Merckx 2017 [10]
NAAT	0.874	0.711–0.956	Merckx 2017 [10]
Clinical Judgement	0.36	0.22–0.52	Dugas 2015 [16]
Batch PCR	0.95	0.75–1†	Assumption (Merckx 2017) [10]
Specificity, Adults
RIDT	0.995	0.986–0.998	Merckx 2017 [10]
DIA	0.967	0.947–0.98	Merckx 2017 [10]
NAAT	0.98	0.932–0.995	Merckx 2017 [10]
Clinical Judgement	0.78	0.72–0.83	Dugas 2015 [16]
Batch PCR	0.95	0.75–1†	Assumption (Merckx 2017) [10]
Influenza B
Sensitivity, Adults
RIDT	0.332	0.199–0.507	Merckx 2017 [10]
DIA	0.57	0.395–0.716	Merckx 2017 [10]
NAAT	0.757	0.518–0.907	Merckx 2017 [10]
Clinical Judgement	0.36	0.22–0.52	Dugas 2015 [16]
Batch PCR	0.95	0.75–1†	Assumption [10]
Specificity. Adults
RIDT	0.999	0.994–1	Merckx 2017 [10]
DIA	0.988	0.975–0.995	Merckx 2017 [10]
NAAT	0.993	0.978–0.998	Merckx 2017 [10]
Clinical Judgement	0.78	0.72–0.83	Dugas 2015 [16]
Batch PCR	0.95	0.75–1†	Assumption [10]
ILI and Influenza-Related Probabilities
Pre-test probability of influenza	0.144	0–1	Seasonal assumptions [17]
Influenza A (Influenza B‡)	0.873	0–1	Seasonal assumptions [17]
Hospitalization	0.116	0.09–0.15†	Ng 2018 [18]
ICU Hospitalization, ≥ 65y	0.134	0.1–0.17†	CIRN (FluWatch) [19]
Tx within 48 hrs of symptom onset	0.481	0.36–0.6†	Muthuri 2014 [8]
Adverse events, Tx	0.075	0.056–0.094†	Santesso 2019 (Unpublished)
Adverse events, no Tx	0.027	0.02–0.034†	Santesso 2019 (Unpublished)
Mortality (ICU admitted, Early Tx)	0.276	0.21–0.35†	Muthuri 2014 [8]
Mortality (ICU-admitted, Late Tx)	0.3198	0.24–0.4†	Muthuri 2014 [8]
Mortality (ICU-admitted, No Tx)	0.5344	0.4–0.67†	Muthuri 2014 [8]
Mortality (Non-ICU, Early Tx)	0.0809	0.06–0.1†	Muthuri 2014 [8]
Mortality (Non-ICU, Late Tx)	0.1218	0.09–0.15†	Muthuri 2014 [8]
Mortality (Non-ICU, No Tx)	0.1218	0.09–0.15†	Muthuri 2014 [8]
Utilities (QALYs)
Population, age dependent	0.88–0.94	0.8722–0.9426	Mittmann 1999 [20]
QALYs lost for ILI (disutility), ≥ 65y	0.0293	0.0233–0.0349	Sander 2009 [21]
QALY improvement for symptom alleviation from treatment, > 18y	0.00166	0.0012–0.0021†	Assumption (Jefferson 2015) [5]
Adverse event (disutility)	0.0113	0.008–0.014†	Greiner 2006 [22]
Costs
RIDT, per test	20	20–26	Merckx 2017 [10]
DIA, per test	20	20–26	Merckx 2017 [10]
NAAT, per test	40	40–130	Merckx 2017 [10]
Batch PCR, per test	58	28–88	Soto 2016 [23]
Emergency department visit	468	351–585†	Ng 2018 [18]
Hospitalization	7,977	5,983–9,971†	Ng 2018 [18]
ICU Hospitalization	11,875	8,906–14,844†	Ng 2018 [18]
Oseltamivir treatment	42	34–42	Ontario Drug Benefit [24]

† Uncertainty of key parameter was not reported and ± 25% was used to create a plausible range.

‡ Probability of influenza B was complementary to probability of influenza A

CIRN, Canadian Immunization Research Network; DIA, digital immunoassay; ICU, intensive care unit; ILI, influenza-like-illness; NAAT, nucleic acid amplification test; PCR, polymerase chain reaction; QALY, quality-adjusted life year; RIDT, rapid influenza diagnostic test; Tx, treatment; y, years of age

Our model used a single healthcare payer perspective (applicable to each province in Canada) and lifetime time horizon to capture the potential benefits of averted mortality through optimal therapy for patients. Our model reported health outcomes (proportion of patients treated appropriately, adverse events, and mortality), quality-adjusted life years (QALYs), total healthcare costs, and net health benefit (NHB) to measure cost-effectiveness. Costs and QALYs were discounted at an annual rate of 1.5% as recommended [11].

Net health benefit

In cost-effectiveness analysis, there are various units of measure used to present cost-effectiveness. Common outcomes include incremental cost-effectiveness ratios (ICERs) that take a cost-utility approach and express value in a $/QALY gained, net monetary benefit (NMB) that expresses value in terms of costs, and net health benefits (NHB) that expresses value in terms of the health outcome chosen (i.e., QALYs in this study).

As the number of strategies being compared increases, the ratio statistics of the ICER become more difficult to calculate, interpret and compare among each other. An ICER cannot be interpreted without also knowing the quadrant of the cost-effectiveness plane in which the strategy lies, as ratio statistics will yield a positive ICER when there are: 1) cost savings and a reduction in QALYs, and 2) more costs but also QALYs gained. During analysis of multiple strategies, they are ranked by increasing effectiveness to calculate ICERs in reference to the less effective strategy. However, some strategies will need to be ruled out if they are extendedly dominated (i.e., the strategy has an ICER greater than a more effective alternative), and so the ICERs for the more effective alternative would need to be re-calculated each time once the extendedly dominated strategy is removed. The decision rule to identify the most cost-effective strategy is unintuitive; it is not possible to rank strategies from most to least cost-effective using the ICER as the ratio statistics compares to one reference strategy at a time. In these situations, the NHB outcome can be used to present the cost-effectiveness of multiple strategies. The NHB approach does not use ratio statistics and has a natural unit measure of QALYs. This approach allows us to rank the strategies by their cost-effectiveness compared to each other, based on the highest number of QALYS (NHB) provided at a pre-specified cost-effectiveness threshold [12].

In this paper, we used the NHB approach and express all cost-effectiveness outcomes in NHB, which is expressed in units of QALYs. As such, we do not calculate or report ICERs in this study. The NHB framework has been commonly used to simplify cost-effectiveness results for decision-makers [13, 14]. The NHB of strategy n is defined as:

Where healthn refers to the total amount of health resulting from strategy n (units: QALY), costn refers to the total cost of strategy n (unit: $), and CET represents the cost-effectiveness threshold (unit: $/QALY). A positive NHB value represents a cost-effective intervention (i.e. effective trade-off between costs and health benefits for that strategy) at the chosen CET, and higher NHB values represent better value-for-money (i.e. more economically desirable strategies). In this analysis, we calculated NHB at commonly used CET of $50,000/QALY [15].

Parameters and key assumptions

Table 1 outlines the base-case values and data sources for the parameter used in the model, which were obtained from published surveillance data and the literature.

Diagnostic tests

The diagnostic test properties evaluated in this analysis were based on a recent meta-analysis [10]. We also considered a more recent meta-analysis [25], but did not incorporate its estimates because influenza A and B were not considered separately. All three types of rapid POC tests were associated with high specificities for adults (>96%) for influenza A and B. However, NAATs had the highest sensitivity for adults (87.4% for influenza A, 75.7% for influenza B), followed by DIAs (75.4% influenza A, 57% influenza B), and RIDTs (42.6% influenza A, 33.2% influenza B). Based on this meta-analysis, the sensitivity and specificity of Batch PCR were 100%, and results were available in 24 hours. However, for our base-case analysis, we assumed that Batch PCR sensitivity and specificity were slightly less than perfect at 95%. For the “Batch PCR–Wait” strategy, we assumed that the base-case probability of patients being treated within 48 hours of symptom onset (48%) [8], was reduced by half (24%) to account for the delayed results and potential start of treatment. For “Clinician Judgement”, we used estimates from a study by Dugas and colleagues who assessed sensitivity (36%) and specificity (78%) in a high-risk population similar to our modeled population [16].

Probabilities

Influenza epidemiology (e.g. prevalence, distribution of virus strains) was extracted from Canadian data sources for the base-case analysis. In the 2016–2017 surveillance season, the prevalence of influenza among patients presenting with ILI peaked at 14.4%. During this period, influenza A represented 87.3% of all laboratory-confirmed influenza cases [17]. We estimated hospitalization rates from an health administrative data study on influenza for influenza-confirmed patients in Canada [18]. We used hospitalization rates resulting from influenza only, and assumed that patients with ILI and not severely ill were not hospitalized (i.e., discharged home).

Treatment

Given that oseltamivir represents the most commonly prescribed medication for influenza, we assumed that all patients received oseltamivir as NAI treatment. Patients infected with influenza who were treated with NAI in hospital had a reduced time to symptom alleviation and lower risk of mortality relative to untreated patients [5, 8]. A recent meta-analysis of individual patient data was used to determine the proportion of patients treated within 48 hours of symptom onset (48%), and the reduction in mortality risk; the magnitude of reduction was stratified by the timing (early vs. late) of treatment [8]. The level of detail and generalizability offered by this meta-analysis made it suitable for use in the base-case analysis. Meta-analyses of randomized controlled trials have suggested no evidence of mortality benefit from oseltamivir[5, 26]. However, since these review includes RCTs of low-risk patients or mixed populations (i.e., the large majority of enrolled patients did not have severe influenza infection), the evidence is highly indirect for our study’s target population. As such, we use the meta-analysis of observational studies, which is more direct to our study’s target population, in the base-case and we conducted scenario analysis in which oseltamivir treatment has no mortality or hospitalization benefit. The probability of adverse events associated with treatment was estimated in a meta-analysis to be 7.5% (Nancy Santesso and colleagues, personal communication).

QALYs and utilities

QALYs are calculated as the product of a utility and the number of life years gained. A utility is a numeric measure of the preference for a specific health state, and ranges between 0 (death) to 1 (perfect health), capturing the quality-of-life associated with the number of life years in a specific health state. We extracted QALY decrements for influenza from an economic evaluation in the United States, and assumed all patients with ILI or influenza received the same QALY decrement of 0.0146 to 0.0293 depending on their age (0.0293 for the base-case patient 65 years of age) [21, 27]. We assumed this decrement was constant over the episode of influenza, and that differential severity or length of stay would not significantly change the decrement. Benefit from oseltamivir treatment was estimated to be 0.00166 QALYs, based on time to symptom alleviation from the Cochrane review [5]. In the “Batch PCR–Treat”, we assumed that one day of NAI treatment regimen prior to test results becoming available do not provide QALY benefit to patients testing negative. Accrued lifetime QALYs were calculated using utilities from a community-dwelling population between 0.88 and 0.94 [20].

Costs

All direct costs were extracted from the literature and inflated to 2017 Canadian dollars. For diagnostic tests costs, we used the lower limit of range estimates from Merckx et al. for RIDT ($20), DIA ($20), and NAATs ($40) [10]. We assumed the hospital setting invested in start-up and capital costs for all diagnostic tests and rapid diagnostic tests did not require lab technician time. All healthcare utilization costs related to influenza were extracted from an administrative data study from the Canadian Immunisation Research Network [18]. A complete course of oseltamivir treatment was $42, which was calculated using Ontario Drug Benefit list prices and current recommended treatment algorithms [9, 24].

Analysis

The base-case analysis was conducted for adult patients 65 years of age presenting to the ED with ILI, with a seasonal pre-test probability of influenza of 14.4% and a seasonal influenza A probability of 87.3%. The probability of being hospitalized was 11.6% and the probability of being treated with oseltamivir within 48 hours of symptom onset was 48%. We assumed all patients testing positive for influenza were given oseltamivir based on current treatment recommendations, and that adverse events did not extend length of stay or increase healthcare utilization.

We assessed cost-effectiveness in multiple scenario analyses: best-case (e.g. upper limit of test characteristics) and worst-case (e.g. lower limit) scenarios for all diagnostic tests, a scenario where cost of adverse events were equivalent to the cost of an ED visit, a scenario where treatment with oseltamivir does not reduce risk of hospitalization or mortality, a scenario where 50% of individuals who test negative for influenza are still given treatment, and use of diagnostics in children (5 years of age) with the appropriate data. We conducted extensive deterministic sensitivity analysis to assess parameter uncertainty (e.g., pre-test probability for influenza since epidemiology varies seasonally and regionally). We assigned beta distributions for probabilities and utilities, and gamma distributions for costs to perform a probabilistic sensitivity analysis using 100,000 Monte Carlo simulations. We reported results following the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Guidelines (S1 File) [28].

Sources of funding

This work was partially supported by the World Health Organization. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. S. M. is supported by a CIHR Frederick Banting and Charles Best Canada Graduate Scholarship Doctoral Award GSD-159274. B.S. is supported by a Canada Research Chair in Economics of Infectious Diseases (CRC-950-232429). There was no additional external funding received for this study.

Results

Base-case analysis

The preferred strategies by increasing NHB (lowest to highest) are: “Don’t Treat Anyone”, “Clinical Judgement”, “RIDT”, “Batch PCR–Wait”, “DIA”, “NAAT”, “Batch PCR–Treat” and “Treat Everyone”. “Treat Everyone” resulted in the highest number of expected QALYs per patient at 15.0477 at an expected cost of $630.01. At a CET of $50,000/QALY, “Treat Everyone” resulted in the highest NHB (15.0344 QALYs), and was considered the most cost-effective strategy in the high-risk older population. The NHB for “Batch PCR—Treat” and “NAAT” strategies were 15.0318, and 15.0277 QALYs, respectively. All results are summarized in Table 2. “Batch PCR–Treat” and “NAAT” resulted in reduced NHBs due to reduced health benefits (e.g. increased hospitalization, mortality) compared to “Treat Everyone”, and increased diagnostic costs. However, “Batch PCR–Treat” resulted in 5,099 fewer adverse events (nausea/vomiting) per 100,000 persons when compared to “Treat Everyone” since the latter strategy does not confirm influenza diagnosis. “Treat Everyone” appropriately treated 100% of patients with influenza but inappropriately treated 100% of patients with ILI only (i.e., no influenza). Appropriateness of treatment outcomes are summarized in Fig 2.

Table 2

Base-case results.

	Health Outcomes (Proportions) †				Health Outcomes (per 100,000)			Cost-effectiveness
Strategy	Patients with influenza		Patients without influenza		Adverse events	Hospitalisations	Mortality	QALYs	Costs (CAD)	NHB at $50,000 CET (QALYs)
	Appropriate (Tx-Flu)	Inappropriate (No Tx-Flu)	Appropriate (No Tx-No Flu)	Inappropriate (Tx–No Flu)
“Don't Treat Anyone”	0.00	1.00	1.00	0.00	404	1,680	1,836	14.9961	608.19	14.9839
“Clinical Judgement”	0.35	0.65	0.78	0.22	2,037	1,626	1,735	15.0145	611.02	15.0023
“RIDT”	0.41	0.59	1.00	0.00	712	1,619	1,715	15.0175	622.52	15.005
“Batch PCR–Wait”	0.95	0.05	0.95	0.05	1,362	1,595	1,659	15.0241	661.30	15.0109
“DIA”	0.73	0.27	0.97	0.03	1,113	1,569	1,604	15.0338	618.99	15.0214
“NAAT”	0.85	0.15	0.98	0.02	1,117	1,546	1,571	15.0404	636.75	15.0277
“Batch PCR–Treat”	0.95	0.05	0.00	1.00	2,348	1,522	1,537	15.0450	661.19	15.0318
“Treat Everyone”	1.00	0.00	0.00	1.00	7,447	1,518	1,533	15.0470	630.01	15.0344

† Calculations are described in S2 File.

CAD, Canadian dollars; CET, cost-effectiveness threshold; DIA, digital immunoassay tests; Hosp., Hospitalization; NAAT, nucleic acid amplification test; NHB, Net health benefit; PCR, polymerase chain reaction; QALYs, quality-adjusted life years; RIDT, rapid influenza diagnostic tests; Tx, treatment

Fig 2

Treatment appropriateness results for all strategies.

Although “Don’t Treat Anyone” was the least costly strategy ($608.91 per patient), it resulted in the highest number of hospitalizations (1,680 per 100,000), and deaths (1,836 per 100,000) which contributed to this strategy having the lowest NHB of 14.9839 QALYs. This strategy also resulted in the highest proportion of untreated influenza cases.

Of the three rapid diagnostic tests, using NAAT to inform NAI treatment (“NAAT”) was the most cost-effective. This strategy resulted in the greatest health benefit, NHB and lowest number of deaths (1,571 deaths per 100,000) compared to “DIA” (1,604 deaths per 100,000) and “RIDT” (1,715 deaths per 100,000). “Clinical Judgement” was the least preferred method of diagnosis in terms of NHB when compared to RIDTs, DIAs, NAATs, and Batch PCR. Costs and effectiveness of all strategies are plotted on a cost-effectiveness plane in Fig 3.

Fig 3

Cost-effectiveness plane.

Sensitivity analysis

At a pretest probability of 0%, “Don’t Treat Anyone” was the most cost-effective strategy based on the highest number of NHBs at 15.2689 QALYs (S1 Table). As the pre-test probability for influenza increased to 1%, the most cost-effective strategy changed to “Treat Everyone” followed by “Batch PCR–Treat”. This order of preferred strategies remained constant as the pretest probability increased to 100%. Our model results were robust to the following variables within the ranges listed in Table 1: QALY improvement from NAI, disutility of adverse events, probability of treatment within 48 hours of symptom onset, probability of death (ICU or non-ICU) after early treatment, probability of adverse events, cost of oseltamivir, and cost of batch PCR test.

Probabilistic sensitivity analysis determined that “Treat Everyone” strategy for the high-risk population was likely to be the most cost-effective strategy in over 85% of 100,000 simulations. A cost-effectiveness acceptability curve is included in S1 Fig.

Scenario analysis

When the cost of AEs are assumed to be equivalent to the cost of an ED visit, the most cost-effective strategy remained “Treat Everyone”, despite the average cost per patient increasing from $630 to $665, it afforded the highest NHB of 15.0337 QALYs. In children (corresponding data inputs in S2 Table), the preferred top three cost-effective strategies remain unchanged. In the scenario where early treatment (i.e., treatment within 48 hours of symptom onset) provided similar mortality benefits as late treatment (i.e., ≥ 48 hours post-symptom onset), “Treat Everyone” was still most cost-effective, followed by “Batch PCR–Treat”. However, the NHB was considerably lower at 14.9916 QALYs compared to 15.0344 QALYs in the base-case for the most cost-effective strategy. In a subsequent scenario where oseltamivir treatment was assumed to provide no mortality, hospitalization or quality-of-life benefit, “Don’t Treat Anyone” was the most cost-effective with a NHB of 14.9840 QALYs, followed by the “Clinical Judgement” with NHB of 14.9836 QALYs. In this scenario, “Treat Everyone” and “Batch PCR–Treat” have the lowest NHB.

When modeling diagnostic tests at their lowest sensitivity and specificity limits (i.e. worst case), the order of strategies’ cost-effectiveness was unchanged from the base-case analysis. At the upper limits of sensitivity and specificity (i.e. best case), “Batch PCR–Treat” provided an incremental gain of 0.0014 QALYs over “Treat Everyone” at an incremental cost of $29.28. At a CET of $50,000/QALY, “Batch PCR–Treat” was equally as cost-effective as “Treat Everyone” with both strategies having a NHB of 15.0344 QALYs at a CET of $50,000/QALY. The cost-effectiveness ranking of strategies utilizing diagnostic tests by NHB was: “Batch PCR–Treat”, “NAAT”, “DIA”, “Batch PCR–Wait”, “Clinical Judgement”, and “RIDT”. All scenario analysis results are summarized in S3 Table.

Discussion

Based on our analysis, the preferred strategy in terms of health impact (QALYs) and cost-effectiveness (NHB) for high-risk older patients admitted to the ED presenting with ILI was “Treat Everyone”, while the preferred diagnostic strategy to confirm influenza was “Batch PCR–Treat”. While the “Treat Everyone” strategy is most cost-effective in terms of NHB, it precludes test results that are required to confirm influenza or to eliminate it as a diagnosis. “Batch PCR–Treat” was the second most cost-effective strategy. Similar to “Treat Everyone”, this strategy starts high-risk severe patients on NAI therapy while awaiting test results, but continuation of treatment depends on the returned diagnostic result. This strategy reduces the number of individuals who are inappropriately treated and the number of adverse events. Of the three rapid diagnostic tests, “NAAT” resulted in the most optimal health outcomes (most QALYs, lowest number of deaths and inappropriate testing) and was cost-effective when compared to “DIA” and “RIDT”. This was expected given that diagnostic test sensitivity was critical in identifying true influenza cases accurately for a quick start of antiviral NAI treatment benefit, and “NAAT” had the highest sensitivity of all rapid diagnostics at 0.87. While the NHB allowed us to determine the most cost-effective strategies, the differences in the average costs, QALYs, and NHB per patient between strategies are considered small in magnitude. For example, “Treat Everyone” resulted in a gain of 0.0066 QALYs compared to “NAAT”, which is roughly equivalent to a gain of 2.4 days.

Sensitivity and scenario analysis suggested that while costs of treatment and diagnostics are important to consider in influenza management, they had little impact on the cost-effectiveness when compared to diagnostic test parameters, treatment benefits and seasonal prevalence of influenza. In a scenario analysis where the upper limit of sensitivity and specificity were used for all tests (i.e. the best-case scenario), “Batch PCR–Treat” was most preferred. These results suggested that the sensitivity and specificity are influential parameters to this model. A lower estimate of these test characteristics from the meta-analysis by Merckx and colleagues could have underestimated the strategies’ cost-effectiveness. In this scenario, the sensitivity and specificity of Batch PCR for both influenza A and B were 1.00, which was the assumption used by Merckx and colleagues in their meta-analysis of rapid diagnostic tests [10].

In the literature, there have been several cost-effectiveness analyses of diagnostic testing for influenza, within the ED or hospital [23, 29–32]. Our results are comparable to a study by Dugas and colleagues in the United States, who assessed the cost-effectiveness of PCR-based rapid influenza testing and treatment using a decision-analytic model [29]. Dugas and colleagues concluded similar order of preferred strategies: treating all patients was most cost-effective and treating no patients with antivirals was the least. Dugas and colleagues used a QALY improvement of symptoms of 0.006, a pretest probability for influenza of 0.20 and evaluated high-risk patients who were 65 years of age. We used a similar but more conservative approach for QALY improvement due to NAI treatment (0.0017), pretest probability for influenza (0.15), and diagnostic sensitivity and specificity for batch PCR and clinical judgement.

In a Canadian study, Nshimyumukiza and colleagues estimated the cost-effectiveness of POC rapid tests versus clinical judgement in incremental costs per life-year saved for one seasonal influenza season, concluding that POC rapid tests were dominant compared to clinical judgement in Quebec, Canada [31]. We determined that “NAAT”, “DIA” and “RIDT” were considered more cost-effective than “Clinical Judgement” based on NHB using a CET of $50,000/QALY in Canada. Our study differs in that we report additional health outcomes, and cost-effectiveness using QALYs instead of life-years gained, and considered various testing strategies to guide treatment using updated diagnostic test characteristics.

Our analysis was subject to several limitations. These results should apply only to high-risk elderly patients presenting in the ED setting, and should not be extrapolated to lower risk populations or to other settings such as primary care, where risk of hospitalization, risk of mortality and cost of care may be lower. We did not incorporate resistance to antivirals or influenza transmission in our model, which are potential indirect consequences of the “Treat Everyone”, and “Don’t Treat Anyone” strategies, respectively. However, we estimated the proportion of high-risk older patients that would be appropriately and inappropriately treated with NAI in these strategies. While "Treat Everyone" resulted in higher NHB (i.e., cost-effectiveness), it inappropriately treats a large number of patients without influenza, which should be taken into consideration when comparing this strategy to “Batch PCR–Treat” during decision-making by both clinicians and policy-makers. As discussed previously, “Treat Everyone” may create antiviral resistance and lead to an unnecessary number of serious adverse events that may increase healthcare utilization. In addition, the incorrect use of the NAI therapy may be considered an opportunity cost where this volume of treatment could be used appropriately in other individuals presenting early with influenza. Clinicians value testing because it establishes a diagnosis and forces a focus on other possibilities if the test result is negative; this scenario was not modelled, which could understate the benefits of diagnostic testing prior to treating patients. However, severely ill patients with suspected influenza are typically treated with broad spectrum antibiotics and oseltamivir, pending results of multiple diagnostic tests. We assumed that healthcare facilities offering these diagnostic tests had negligible infrastructure and equipment costs, which may have underestimated the costs of newer diagnostics (e.g. NAAT and DIA).

Our model did not consider re-admissions or hospitalizations for other medical concerns. We assumed that high-risk older patients with ILI or influenza experience a similar disutility in quality-of-life (i.e., reduction in QALYs) and that influenza severity and hospitalization did not significantly alter the QALY decrement experienced by admitted patients. Since Muthuri and colleagues’ meta-analysis on NAI treatment benefit on mortality was based on pandemic data, we conducted scenario analyses to examine the cost-effectiveness of strategies where early treatment did not confer additional mortality benefit than late treatment (which still provided some mortality benefit over no treatment). In this scenario, while the NHB between the strategies converged, the most cost-effective strategies remained the “Treat Everyone” followed by the “Batch PCR–Treat”. In another scenario analysis where treatment with oseltamivir did not confer any mortality benefit, the “Don’t Treat Anyone” strategy provided the most health outcomes and NHB (i.e., was most cost-effective) which is expected given treatment may result in adverse outcomes but provides no mortality, hospitalization or quality-of-life benefit. This finding along with the previous scenario where early treatment has no early benefit suggests that mortality benefit from treatment, regardless of early timing, drives the cost-effectiveness between the strategies.

Despite these limitations, this analysis comprehensively assessed the cost-effectiveness and impact of influenza point-of-care diagnostic tests on health outcomes in high-risk elderly patients admitted to the ED presenting with ILI. We reported results in QALYs, costs, and other health outcomes that are generalizable to other interventions and diagnostics for system level comparisons by decision-makers. Our analysis incorporated strong meta-analysis evidence on recently developed rapid diagnostic tests for influenza that have not been previously compared. Cost-effectiveness is an important consideration when implementing newer, more costly, diagnostic technologies. These results are transferable in jurisdictions with similar influenza epidemiology, healthcare system (i.e. single payer system), and population health status.

Conclusion

Treating high-risk older patients without performing a novel rapid diagnostic test resulted in the highest NHB and was the most cost-effective strategy. This strategy was less costly and reduced mortality through quicker and increased uptake of NAI. However, it inappropriately treats 100% of patients without influenza and does not provide diagnostic confirmation that can be attained by Batch PCR. Our analysis provides evidence on the impact of rapid diagnostic tests for influenza in the emergency department in terms of QALYs and cost-effectiveness that can be used by health policy decision-makers.

CHEERS checklist.

(PDF)

Click here for additional data file.

Calculations used for treatment appropriateness.

(PDF)

Click here for additional data file.

Sensitivity analyses results.

(PDF)

Click here for additional data file.

Parameters for children population.

(PDF)

Click here for additional data file.

Scenario analysis NHB results at a cost-effectiveness threshold of $50,000/QALY.

(PDF)

Click here for additional data file.

Cost-effectiveness acceptability curve.

(PDF)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

5 Aug 2020

PONE-D-20-15378

Point-of-Care Diagnostic Tests for Influenza in the Emergency Department: A Cost-Effectiveness Analysis in a High-Risk Population from a Canadian Perspective

PLOS ONE

Dear Dr. Mac,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 04 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ruslan Kalendar, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your methods, please include a section describing the data used for this study including source, citation, description of the population represented by the data, and what categories of data were extracted.

3. In your ethics statement in the Methods section and in the online submission form, please provide additional information about the data used in your retrospective study. Specifically, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

4.Thank you for stating in your Funding Statement:

[Sources of Funding: This work was partially supported by the World Health Organization.  The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.].

Please provide an amended statement that declares *all* the funding or sources of support (whether external or internal to your organization) received during this study, as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now.  Please also include the statement “There was no additional external funding received for this study.” in your updated Funding Statement.

Please include your amended Funding Statement within your cover letter. We will change the online submission form on your behalf.

5.Thank you for stating the following in the Competing Interests section:

[I have read the journal's policy and the authors of this manuscript have the following competing interests: NKJA co-chaired the WHO Guideline Development Group – Clinical Management of Severe Influenza Infections.].

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

6. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: No

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Reviewer #1: General comments

There are two important problems with this manuscript. First, they base their assumption of mortality benefit on a very optimistic assessment from an observational study, and not on the best evidence from RCTs. Second, while their analysis is focused on a 65 year old presenting to the ED, that is not clearly emphasized throughout the abstract and discussion, giving readers the impression that it could apply to all patients in all settings.

Specific comments

Abstract

What is the $/QALY gained? The results are presented in such a way that the casual reader could infer that treating everyone results in 15 QALYs for a cost of only $600.

You state: “most favourable outcomes per cohort (1,571 deaths per 100,000).” Of course, death is not a favourable outcome, reducing deaths would be.

It should be clear that this analysi only applies to high risk adults presenting to the ED, and not to average risk adults presenting in primary care which is of course a far more common scenario. It is important that these results not be extrapolated, something the abstract as currently written wrould encourage as it says nothing about ED or high risk.

Introduction

Lines 33-35: But, no RCT evidence of mortality benefit (see Cochrane review by Jefferson and Ebell MH, Williamson M, Schofill, J. Effectiveness of oseltamivir in adults: a meta-analysis of published and unpublished clinical trials. Fam Pract 2013; Apr;30(2):125-33.

Line 40: Would add “with a high degree of accuracy” to this sentence. We have long had rapid tests, they just lacked sensitivity.

Line 43: “In recent years” would only apply to the point of care molecular tests, not RIDT.

Line 53-54

Methods

Lines 82-82: In the real world, unfortunately, many patients with negative tests still get a NAI.

Lines 85-86: Which meta-analyses? They should be cited. The two cited above found no difference in mortality based on RCT data. The one study (never published) by Roche of patients 65 and older found absolutely no symptom benefit. Your model hinges on a faulty assumption.

Lines 96-105: Please provide an example of how to ingerpret this. It is quite opaque to me, I’m used to seeing $/QALY or even $/quality adjusted life day for short-term events. No idea how to interpret NHB.

Table 1:

The sensitivities cited here are lower than those in the introduction (53% for RIDT) and also lower than those in other systematic reviews (~60%). This would bias against a testing strategy.

The probability of influenza of 0.144 is based, apparently, on seasonal prevalence in the community. Instead, it should be based on the prevalence of older patients presenting with ILI who have influenza. I suspect that is higher than 14%

Your mortality benefit is based on the IPD meta-analysis by Muthuri (Lancet Respir Med

. 2014 May;2(5):395-404). This was an observational study, not a SR of randomized trials. Of course it provides an optimistic estimate of benefit. The best available evidence from systematic reviews of RCTs (above) does not support a significant mortality benefit and at best only a small reduction in pneumonia.

Lines 92-93: This statement to the reader who is not a health economist (like me) makes it sound like there is an outlandishly large benefit at a very low cost (15 more QALY for $630). I know that isn’t true, but it is hard for the general reader to understand what is meant: “yielding an expected 15.0477 QALYs per patient, 193 at an expected cost of $630.01, and the greatest NHB of 15.0344 QALYs”. We are more accustomed to seeing ICERs.

Table 2.

Here I can see the ICER is about 0.05 additional QALYs at an additional cost of $22 for the treat everyone vs don’t treat anyone strategies.

Page 13, Discussion, para 1: You fail to say “high risk older patients” here.

Page 16, first para: “we conducted scenario analysis where the probability of death due to early treatment was equivalent to late treatment, showing that the order of test and treat strategies did not substantially change based on estimates of NAI effectiveness.” So, are you saying you modeled no mortality benefit and no reduction in hospitalizations, and it didn’t affect your results?

Page 16, 2nd para: In ‘high risk elderly patients”, not all patients coming to ED. You are over-generalizing your results. Obviously the high hospitalization risk of 12% means this ia very very different population from the general population with ILI. In the oseltamivir trials, the hospitalization rate was only 1.4%.

There is no funding declaration.

Reviewer #2:

This paper describes excellent comparison of cost effectiveness of different treatment strategies used in cases of influenza like illness presented to the emergency department. The authors have very well compared these strategies and discussed well the impact of the top most strategies. This provides an useful information on cost efficacy of different treatment approaches in such situation.

In discussion, the side effects of approach "Treat everyone" should be discussed in more detail with other published findings. In my view, that is an important piece to discuss the side effects of treatment that could happen in patients without the illness if that approach is recommended.

The comparison of "Batch PCR-Treat" and "Treat Everyone" could be highlighted more.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Reviewer #1: Yes: Mark H. Ebell MD, MS, Professor of Epidemiology, University of Georgia, Athens, USA

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

22 Sep 2020

Stephen Mac

Institute of Health Policy, Management, and Evaluation

University of Toronto

155 College St. Suite 425

Toronto, ON M5T3M6

Email: sm.mac@mail.utoronto.ca

To: Ruslan Kalendar

Academic Editor

PLOS ONE

September 16, 2020

RE: PONE-D-20-15378 “Point-of-Care Diagnostic Tests for Influenza in the Emergency Department: A Cost-Effectiveness Analysis in a High-Risk Population from a Canadian Perspective”

Dear Dr. Kalendar and Reviewers,

Thank you for taking the time to review our manuscript and providing insightful comments and suggestions. We have taken into consideration all comments and suggestions highlighted by the reviewers and revised our manuscript accordingly for resubmission to PLOS One. Please find enclosed our revised manuscript entitled “Point-of-Care Diagnostic Tests for Influenza in the Emergency Department: A Cost-Effectiveness Analysis in a High-Risk Population from a Canadian Perspective”.

Below we summarize the key issues that were raised, our explanations (if applicable), how we addressed them, and where the revisions were made in the revised manuscript with tracked changes.

1. Treatment effect. There was concern from Reviewer 1 that our model’s use of the meta-analysis from Muthuri et al. was faulty since other reviews of randomized clinical trials have shown no mortality and hospitalization benefits.

Explanation: We decided to use the meta-analysis of observational studies (Muthuri et al.) since the reviews of randomised controlled trials (RCTs) do not only include high-risk patients (i.e., the large majority of enrolled patients did not have severe influenza infection), and therefore the evidence from them is highly indirect for our study’s target population (high-risk older population presenting to the ED); so that the direct evidence from observational studies is more appropriate to use, even though it is downgraded for risk of bias using the appropriate GRADE assessment for study design. This was the position taken at the time of study conceptualization by the Guideline Development Group (GDG) members for the World Health Organization (WHO) guideline. We do acknowledge that this is a debatable point but believe that our rationale is sufficient to support the use of the meta-analysis of observational studies for mortality benefit in our study.

Revisions: We have added this rationale and acknowledged that there are high-quality studies with a mixed population, that suggest no mortality benefit, into the Methods section (p.10-11, Ln 176-182) of the revised manuscript. Furthermore, we have simulated a scenario analysis, which is outlined in the Methods of the revised manuscript (p.12, Ln 219-221), where we assume that the NAI has no mortality, hospitalization benefit (i.e., citing the RCT mentioned in the comments) or quality-of-life benefit. The results suggest that “Treat Everyone” is not cost-effective and that “Don’t Treat Anyone” (i.e., not testing and not treating) is the most cost-effective in this scenario. We have included the results of this additional analysis in the Results (p.16, Ln 292-296), summarized in S6 Table, and elaborated on the implications of these findings in the Discussion (p.20, Ln 381-387) of the revised manuscript.

2. The use of net health benefit (NHB) instead of ICERs. There was confusion as to what the NHB actually represents in this paper and the omission of ICERs, which the reviewer is most familiar with cost-effectiveness analyses.

Explanation: We understand that typically cost-effectiveness analyses (CEA) report the value of interventions or technologies in terms of incremental cost-effectiveness ratios (ICER) but in our study where we compare eight strategies, we decided to report cost-effectiveness using the net health benefit approach (NHB).

In CEA, there are various units of measure to present cost-effectiveness: using incremental cost-effectiveness ratios (ICER) taking a cost-utility approach which would express value in a $/QALY gained, net monetary benefit (NMB) which would express value in terms of costs, and net health benefits (NHB) which expresses value in terms of the health outcome chosen (i.e., in quality-adjusted life years in this study). (Paulden, M., PharmacoEconomics (2020) 38:781–784).

While most conventional CEA in the literature report value in terms of ICERs of cost per QALY gained, it is typically preferred when comparing 2-3 strategies. As the number of strategies being compared increases, the ratio statistics of the ICER become more difficult to calculate, interpret and compare amongst each other. An ICER cannot be interpreted without also knowing the quadrant of the cost-effectiveness plane in which the strategy lies, as ratio statistics will yield a positive ICER when there are: 1) cost savings and a reduction in QALYs, and 2) more costs but also QALYs gained. Some strategies will need to be ruled out if they are extendedly dominated, and the ICERs would need to be re-calculated depending on the reference strategy. The decision rule to identify the most cost-effective strategy is unintuitive; it is not possible to rank strategies from most to least cost-effective using the ICER as the ratio statistics compares to one reference strategy at a time. (Paulden, M., PharmacoEconomics (2020) 38:781–784).

In these situations, the NHB outcome can be used to present the cost-effectiveness of multiple strategies. The NHB approach does not use ratio statistics and has a natural unit measure of QALYs. This approach allows us to rank the strategies by their cost-effectiveness compared to each other, based on the highest number of QALYS (NHB) provided at a pre-specified cost-effectiveness threshold.

Revisions: We recognize that it may be confusing throughout the study by reporting the costs and then the NHB, which has a natural unit of QALYs. Therefore, we have taken the necessary steps below to ensure the reader understands the use of the NHB approach. Throughout the revised manuscript, we have:

• Removed any mention of the ICER (i.e., we do not report the $/QALY gained anywhere as this approach was not used).

• Reported the strategies’ outcomes more consistently and distinctly in terms of costs, QALYs, and NHB

• Further elaborated on the NHB approach in the Methods section (p.6-7, Ln103-135) and the rationale for using this as opposed to the ICER approach.

Following this letter, we included a table outlining our detailed response to all comments for your review. Thank you for your time and consideration. We look forward to your decision.

Sincerely,

Stephen Mac

On behalf of all authors below

Stephen Mac PhD(c), MBiotech

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Ryan O’Reilly MD(c), PhD(c)

Department of Medicine, McMaster University, Hamilton, Canada

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Neill Adhikari MDCM, MSc

Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Canada

Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Canada

Rob Fowler MDCM, FRCPC, MSc

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Canada

Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Canada

Beate Sander PhD

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Public Health Ontario, Toronto, Canada

ICES, Toronto, Canada

Reviewer 1

Abstract

What is the $/QALY gained? The results are presented in such a way that the casual reader could infer that treating everyone results in 15 QALYs for a cost of only $600.

Response: We understand that typically cost-effectiveness analyses (CEA) report the value of interventions or technologies in terms of incremental cost-effectiveness ratios (ICER) but in our study where we compare eight strategies, we decided to report cost-effectiveness using the net health benefit approach (NHB).

In CEA, there are various units of measure to present cost-effectiveness: using incremental cost-effectiveness ratios (ICER) taking a cost-utility approach which would express value in a $/QALY gained, net monetary benefit (NMB) which would express value in terms of costs, and net health benefits (NHB) which expresses value in terms of the health outcome chosen (i.e., in quality-adjusted life years in this study). (Paulden, M., PharmacoEconomics (2020) 38:781–784).

While most conventional CEA in the literature report value in terms of ICERs of cost per QALY gained, it is typically preferred when comparing 2-3 strategies. As the number of strategies being compared increases, the ratio statistics of the ICER become more difficult to calculate, interpret and compare amongst each other. An ICER cannot be interpreted without also knowing the quadrant of the cost-effectiveness plane in which the strategy lies, as ratio statistics will yield a positive ICER when there are: 1) cost savings and a reduction in QALYs, and 2) more costs but also QALYs gained. Some strategies will need to be ruled out if they are extendedly dominated, and the ICERs would need to be re-calculated depending on the reference strategy. The decision rule to identify the most cost-effective strategy is unintuitive; it is not possible to rank strategies from most to least cost-effective using the ICER as the ratio statistics compares to one reference strategy at a time. (Paulden, M., PharmacoEconomics (2020) 38:781–784).

In these situations, the NHB outcome can be used to present the cost-effectiveness of multiple strategies. The NHB approach does not use ratio statistics and has a natural unit measure of QALYs. This approach allows us to rank the strategies by their cost-effectiveness compared to each other, based on the highest number of QALYS (NHB) provided at a pre-specified cost-effectiveness threshold.

We recognize that it may be confusing throughout the study by reporting the costs and then the NHB, which has a natural unit of QALYs. Therefore, we have taken the necessary steps below to ensure the reader understands the use of the NHB approach. Throughout the revised manuscript, we have:

• Removed any mention of the ICER (i.e., we do not report the $/QALY gained anywhere as this approach was not used).

• Reported the strategies’ outcomes more consistently and distinctly in terms of costs, QALYs, and NHB

• Further elaborated on the NHB approach in the Methods section (p.6-7, Ln103-135) and the rationale for using this as opposed to the ICER approach.

You state: “most favourable outcomes per cohort (1,571 deaths per 100,000).” Of course, death is not a favourable outcome, reducing deaths would be.

Response: We have re-worded this to read: “…the least number of deaths (1,571 per 100,000)” in the revised manuscript (p. 1, Ln 16-17).

It should be clear that this analysis only applies to high risk adults presenting to the ED, and not to average risk adults presenting in primary care which is of course a far more common scenario. It is important that these results not be extrapolated, something the abstract as currently written would encourage as it says nothing about ED or high risk.

Response: Thank you for pointing this out. We have added the target population (high-risk adults) into the Abstract in the revised manuscript (p.1, Ln 8, 20).

Introduction

Lines 33-35: But no RCT evidence of mortality benefit (see Cochrane review by Jefferson and Ebell MH, Williamson M, Schofill, J. Effectiveness of oseltamivir in adults: a meta-analysis of published and unpublished clinical trials. Fam Pract 2013; Apr;30(2):125-33.

Response: Thank you for this comment and let us explain our rationale. We decided to use the meta-analysis of observational studies (Muthuri et al.) since the reviews of randomised controlled trials (RCTs) do not only include high-risk patients (i.e., the large majority of enrolled patients did not have severe influenza infection), and therefore the evidence from them is highly indirect for our study’s target population (high-risk older population presenting to the ED); so that the direct evidence from observational studies is more appropriate to use, even though it is downgraded for risk of bias using the appropriate GRADE assessment for study design. This was the position taken at the time of study conceptualization by the Guideline Development Group (GDG) members for the World Health Organization (WHO) guideline. We do acknowledge that this is a debatable point but believe that our rationale is sufficient to support the use of the meta-analysis of observational studies for mortality benefit in our study.

We have added this rationale and acknowledged that there are high-quality studies with a mixed population, that suggest no mortality benefit, into the Methods section (p.10-11, Ln 176-182) of the revised manuscript. Furthermore, we have simulated a scenario analysis, which is outlined in the Methods of the revised manuscript (p.12, Ln 219-221), where we assume that the NAI has no mortality, hospitalization benefit (i.e., citing the RCT mentioned in the comments) or quality-of-life benefit. The results suggest that “Treat Everyone” is not cost-effective and that “Don’t Treat Anyone” (i.e., not testing and not treating) is the most cost-effective in this scenario. We have included the results of this additional analysis in the Results (p.16, Ln 292-296), summarized in S6 Table, and elaborated on the implications of these findings in the Discussion (p.20, Ln 381-387) of the revised manuscript.

Line 40: Would add “with a high degree of accuracy” to this sentence. We have long had rapid tests, they just lacked sensitivity. Added as suggested in the revised manuscript (p.2, Ln 41).

Line 43: “In recent years” would only apply to the point of care molecular tests, not RIDT.

Response: Removed words as suggested in the revised manuscript (p.2, Ln 46).

Methods

Lines 82-82: In the real world, unfortunately, many patients with negative tests still get a NAI.

Response: We acknowledge that possibility and have assumed for the base-case results that these patients with negative tests do not get a NAI. We have revised in the Methods (p.5, Ln 87-90; p.12, Ln220-221). We also conducted scenario analysis where 50% of patients with a negative test still received a NAI. Results are summarized in S6 Table.

Lines 85-86: Which meta-analyses? They should be cited. The two cited above found no difference in mortality based on RCT data. The one study (never published) by Roche of patients 65 and older found absolutely no symptom benefit. Your model hinges on a faulty assumption.

Response: We have cited the meta-analyses used in the revised manuscript (p.5, Ln 93). Regarding our model assumption, please see the comments above for our rationale to use this meta-analysis for mortality benefit, and the other for calculation of QALY benefit. We do acknowledge that this is a debatable point but believe that our rationale is sufficient to support the use of the meta-analysis of observational studies for mortality benefit in our study.

Lines 96-105: Please provide an example of how to interpret this. It is quite opaque to me; I’m used to seeing $/QALY or even $/quality adjusted life day for short-term events. No idea how to interpret NHB.

Response: As discussed above, we have further elaborated on the rationale of using the NHB approach, how to calculate the outcomes (QALYs), and interpret it as a cost-effectiveness measure in the Methods section. We hope that this will help readers understand how to use and interpret the NHB to understand the cost-effectiveness of each of the strategies.

Table 1: The sensitivities cited here are lower than those in the introduction (53% for RIDT) and also lower than those in other systematic reviews (~60%). This would bias against a testing strategy.

Response: Thank you for pointing this out. We based all sensitivity and specificity parameters from the Merckx et al. systematic review and meta-analysis. In Table 1, the test parameters are specifically for adults whereas the parameters in the Introduction are in total over all populations. We have revised in Table 1 (p.8-9) to include “adult”, and the Introduction (p.4, Ln 53) to clarify that these parameters are for the entire meta-analysis population: “adults and children”. We acknowledge that there are other systematic reviews and studies that suggest higher or lower test parameters. However, the Merckx et al study was considered a high-quality study. We conducted a scenario analysis to address the uncertainty of test parameters in the Results (p.16-17, Ln 298-306), elaborated further on the importance of sensitivity and specificity in the Discussion (p.17, Ln 327-330), and summarized all results in S6 Table.

The probability of influenza of 0.144 is based, apparently, on seasonal prevalence in the community. Instead, it should be based on the prevalence of older patients presenting with ILI who have influenza. I suspect that is higher than 14%

Response: We agree that the probability of influenza is a seasonal prevalence estimate in the community that can fluctuate from season-to-season and depending on the population of interest. To address this, we conducted sensitivity analysis to explore the impact of this uncertainty on the study results. This is reported in the Results (p.15, Ln 270-274), and S3 Table.

Your mortality benefit is based on the IPD meta-analysis by Muthuri (Lancet Respir Med. 2014 May;2(5):395-404). This was an observational study, not a SR of randomized trials. Of course, it provides an optimistic estimate of benefit. The best available evidence from systematic reviews of RCTs (above) does not support a significant mortality benefit and at best only a small reduction in pneumonia.

Response: We do acknowledge that this is a debatable point but believe that our rationale is sufficient to support the use of the meta-analysis of observational studies for mortality benefit in our study. Please see comments above.

Lines 92-93: This statement to the reader who is not a health economist (like me) makes it sound like there is an outlandishly large benefit at a very low cost (15 more QALY for $630). I know that isn’t true, but it is hard for the general reader to understand what is meant: “yielding an expected 15.0477 QALYs per patient, 193 at an expected cost of $630.01, and the greatest NHB of 15.0344 QALYs”. We are more accustomed to seeing ICERs.

Response: Thank you for pointing this out. As discussed above, we have revised the manuscript throughout to describe all strategies’ outcomes more consistently and distinctly in terms of costs, QALYs, and NHB to avoid confusion for audience who are not health economists. We will only report NHB outcomes in this study to avoid going back and forth and confusion to the readers.

Table 2. Here I can see the ICER is about 0.05 additional QALYs at an additional cost of $22 for the treat everyone vs doesn’t treat anyone strategies.

Response: As discussed above, we will only report NHB outcomes in this study to avoid going back and forth and confusing the readers.

Discussion

Page 13, Discussion, para 1: You fail to say “high risk older patients” here.

Response: Added as suggested in the revised manuscript (p.17, Ln 310).

Page 16, first para: “we conducted scenario analysis where the probability of death due to early treatment was equivalent to late treatment, showing that the order of test and treat strategies did not substantially change based on estimates of NAI effectiveness.” So, are you saying you modeled no mortality benefit and no reduction in hospitalizations, and it didn’t affect your results?

Response: Thank you for this comment. This sentence as it is currently written can be misinterpreted. In this scenario, we assume that the mortality benefit of early treatment is equal to the mortality benefit of late treatment (i.e., early treatment does not provide additional mortality benefit if within 48 hours of symptom onset). However, there is still some mortality benefit with late treatment vs. no treatment. While the ordering of the strategies in term of cost-effectiveness did not change, the NHB of each strategy was reduced. We have made these results and discussion/explanation clearer in the Discussion (p.20, Ln 376-387).

Page 16, 2nd para: In ‘high risk elderly patients”, not all patients coming to ED. You are over-generalizing your results. Obviously, the high hospitalization risk of 12% means this is a very different population from the general population with ILI. In the oseltamivir trials, the hospitalization rate was only 1.4%.

Response: Thank you for pointing this out. We have added the target population (high-risk elderly patients) so that we are not overgeneralizing, and readers understand that this is for a specific population with higher hospitalization rates. The changes are in the revised manuscript (p.20, Ln 391).

There is no funding declaration.

Response: This was stated at the end of the Abstract. We have now added the funding declaration into the Methods as suggested in the revised manuscript (p.13, Ln 229-231).

Reviewer 2

In discussion, the side effects of approach "Treat everyone" should be discussed in more detail with other published findings. In my view, that is an important piece to discuss the side effects of treatment that could happen in patients without the illness if that approach is recommended.

Response: We agree with this comment and have tried to outline some of the possible adverse events and consequences of the “Treat Everyone” strategy in the discussion. We have further elaborated on this in the revised manuscript (p.19, Ln 359-364).

The comparison of "Batch PCR-Treat" and "Treat Everyone" could be highlighted more.

Response: Thank you for this comment. We have further compared the “Batch PCR – Treat” and “Treat Everyone” strategies in the discussion of the revised manuscript (p.17, Ln 311-317).

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

26 Oct 2020

PONE-D-20-15378R1

Point-of-care diagnostic tests for influenza in the emergency department: A cost-effectiveness analysis in a high-risk population from a Canadian perspective

PLOS ONE

Dear Dr. Mac,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Authors need to prepare a manuscript in accordance with the reviewer's comment.

Please submit your revised manuscript by Dec 10 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ruslan Kalendar, PhD

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

**********

6. Review Comments to the Author

Reviewer #1: The authors have done a good job of responding to comments. I have a few final recommendations:

1. The authors write:

Other meta-analysis in the literature have suggested no evidence of mortality benefit from oseltamivir[26]. However, since this review includes RCTs of low-risk patients or mixed populations (i.e., the large majority of enrolled patients did not have severe influenza infection), the evidence is highly indirect for our study’s target population.

I suggest revising this as follows and adding reference to the Cochrane Review by Jefferson and colleagues.

Meta-analyses of randomized controlled trials have suggested no evidence of mortality benefit from oseltamivir[26,]. However, since this review includes RCTs of low-risk patients or mixed populations (i.e., the large majority of enrolled patients did not have severe influenza infection), the evidence is highly indirect for our study’s target population.

2. There should be a clear statement to the effect that these results apply only to high risk elderly patients being evaluated in the ED setting and should not be extrapolated to lower risk populations or to other settings such as primary care, where risk of mortality and hospitalization and cost of care are all much lower, but the adverse events related to the drug and cost of the drug remain the same.

3. There should also be a discussion of the magnitude of the difference in QALYs, costs, and net health benefit between strategies. For example, in Table 2 the QALYs for NAAT strategy were 15.0404 and for treat everyone were 15.0470. This is a gain of only 0.0066 QALYs or 2.4 days on average, which is quite small.

**********

Submitted filename: PONE-D-20-15378 R1 Ebell final comments.docx

Click here for additional data file.

28 Oct 2020

Stephen Mac

Institute of Health Policy, Management, and Evaluation

University of Toronto

155 College St. Suite 425

Toronto, ON M5T3M6

Email: sm.mac@mail.utoronto.ca

To: Ruslan Kalendar

Academic Editor

PLOS ONE

October 28, 2020

RE: PONE-D-20-15378-R1 “Point-of-Care Diagnostic Tests for Influenza in the Emergency Department: A Cost-Effectiveness Analysis in a High-Risk Population from a Canadian Perspective”

Dear Dr. Kalendar,

Thank you for taking the time to review our manuscript and providing additional comments. We have addressed all comments and suggestions and revised our manuscript accordingly for resubmission to PLOS One. Please find enclosed our revised manuscript entitled “Point-of-Care Diagnostic Tests for Influenza in the Emergency Department: A Cost-Effectiveness Analysis in a High-Risk Population from a Canadian Perspective”.

Following this letter, we included a table outlining our detailed response to all comments for your review. Thank you for your time and consideration. We look forward to your final decision.

Sincerely,

Stephen Mac

On behalf of all authors below

Stephen Mac PhD(c), MBiotech

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Ryan O’Reilly MD(c), PhD(c)

Department of Medicine, McMaster University, Hamilton, Canada

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Neill Adhikari MDCM, MSc

Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Canada

Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Canada

Rob Fowler MDCM, FRCPC, MSc

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Canada

Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, Canada

Beate Sander PhD

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

Toronto Health Economics and Technology Assessment (THETA) Collaborative, University Health Network, Toronto, Canada

Public Health Ontario, Toronto, Canada

ICES, Toronto, Canada

Reviewer [Response]

The authors write: Other meta-analysis in the literature have suggested no evidence of mortality benefit from oseltamivir[26]. However, since this review includes RCTs of low-risk patients or mixed populations (i.e., the large majority of enrolled patients did not have severe influenza infection), the evidence is highly indirect for our study’s target population.

I suggest revising this as follows and adding reference to the Cochrane Review by Jefferson and colleagues.

Meta-analyses of randomized controlled trials have suggested no evidence of mortality benefit from oseltamivir[26,]. However, since this review includes RCTs of low-risk patients or mixed populations (i.e., the large majority of enrolled patients did not have severe influenza infection), the evidence is highly indirect for our study’s target population.

RESPONSE: [We have revised as suggested on page 9 (line 170 to 172).]

There should be a clear statement to the effect that these results apply only to high risk elderly patients being evaluated in the ED setting and should not be extrapolated to lower risk populations or to other settings such as primary care, where risk of mortality and hospitalization and cost of care are all much lower, but the adverse events related to the drug and cost of the drug remain the same.

RESPONSE: [We have revised as suggested in the limitations of the Discussion on page 19 (line 346 to 349).]

There should also be a discussion of the magnitude of the difference in QALYs, costs, and net health benefit between strategies. For example, in Table 2 the QALYs for NAAT strategy were 15.0404 and for treat everyone were 15.0470. This is a gain of only 0.0066 QALYs or 2.4 days on average, which is quite small.

RESPONSE: [We have revised as suggested in the first paragraph of the Discussion on page 17 (line 311 to 314).]

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

30 Oct 2020

Point-of-care diagnostic tests for influenza in the emergency department: A cost-effectiveness analysis in a high-risk population from a Canadian perspective

PONE-D-20-15378R2

Dear Dr. Mac,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ruslan Kalendar, PhD

Academic Editor

PLOS ONE

6 Nov 2020

PONE-D-20-15378R2

Point-of-care diagnostic tests for influenza in the emergency department: A cost-effectiveness analysis in a high-risk population from a Canadian perspective

Dear Dr. Mac:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Ruslan Kalendar

Academic Editor

PLOS ONE