Value of Information Analysis Informing Adoption and Research Decisions in a Portfolio of Health Care Interventions.

Background: Value of information (VOI) analysis quantifies the value of additional research in reducing decision uncertainty. It addresses adoption and research decisions simultaneously by comparing the expected benefits and costs of research studies. Nevertheless, the application of this approach in practice remains limited.
Objectives: To apply VOI analysis in health care interventions to guide adoption decisions, optimize trial design, and prioritize research.
Methods: The analysis was from the perspective of Queensland Health, Australia. It included four interventions: clinically indicated catheter replacement, tissue adhesive for securing catheters, negative pressure wound therapy (NPWT) in caesarean sections, and nutritional support for preventing pressure ulcers. For each intervention, cost-effectiveness analysis was performed, decision uncertainty characterized, and VOI calculated using Monte Carlo simulations. The benefits and costs of additional research were considered together with the costs and consequences of acting now versus waiting for more information. All values are reported in 2014 Australian dollars (AU$).
Results: All interventions were cost-effective, but with various levels of decision uncertainty. The current evidence is sufficient to support the adoption of clinically indicated catheter replacement. For the tissue adhesive, an additional study before adoption is worthwhile with a four-arm trial of 220 patients per arm. Additional research on NPWT before adoption is worthwhile with a two-arm trial of 200 patients per arm. Nutritional support should be adopted with a two-arm trial of 1200 patients per arm. Based on the expected net monetary benefits, the studies were ranked as follows: 1) NPWT (AU$1.2 million), 2) tissue adhesive (AU$0.3 milliion), and 3) nutritional support (AU$0.1 million). Conclusions: VOI analysis is a useful and practical approach to inform adoption and research decisions. Efforts should be focused on facilitating its integration into decision making frameworks.

Decisions on adopting new health care interventions would ideally be based on the best available evidence on their safety, efficacy, and cost-effectiveness.[1] Economic evaluations are commonly performed to compare the costs and effects of alternative interventions; however, the true values of cost and effect estimates are never certain, and thus, any decision based on that evaluation inherits risks of wrong decisions, which may have costly consequences to the health care system. Although uncertainty in decision making can be reduced with more information, conducting additional research may not be worthwhile given the high costs of research studies. Furthermore, there is an opportunity cost for delaying the implementation of a beneficial intervention awaiting the results of the additional research.[2,3] Therefore, it is reasonable to assess the need and value of additional research before making decisions.[4] Value of information (VOI) analysis quantifies the expected value of research in reducing decision uncertainty.[4,5] It is based on the principle that additional information reduces the probability of making a suboptimal decision, thus reducing the opportunity loss of that decision.[3,5]

The VOI framework comprises a range of measures that can be systematically applied to inform adoption and research decisions simultaneously. The most common measure of VOI is the expected value of perfect information (EVPI), which is the value of additional information to resolve uncertainty in all parameters.[3] This measure informs if additional research is required (i.e., potentially worthwhile). Another measure is the expected value of perfect parameter information (EVPPI), which represents the value of resolving uncertainty in a parameter or a subset of parameters.[3] EVPPI guides the focus of additional research (i.e., which parameters to study) and, subsequently, the type of that research (e.g., randomized trial). Both EVPI and EVPPI measure the maximum (i.e., upper bound) value of additional research, but this is not sufficient to inform decisions. To establish a sufficient condition, we need to consider the marginal benefits and marginal costs of a specific research study design.[2,3] The value of reducing, rather than resolving, uncertainty through collecting data in a research study is the expected value of sample information (EVSI).[6] The difference between the EVSI and the expected costs of the study is the expected net benefit of sampling (ENBS). The study is worthwhile when its expected benefits exceed its expected costs (i.e., ENBS is positive).[6] Furthermore, the optimal trial design is the one that maximizes ENBS, and the research study with the highest ENBS should be prioritized.[2,7,8]

There has been an increase in the reporting of EVPI in cost-effectiveness analyses; however, the reporting of other VOI measures, particularly EVSI and ENBS, in applied studies remains limited.[9,10] Recent systematic reviews of economic evaluations published until 2013 showed that only four studies had reported EVSI and ENBS in their results.[10,11] Moreover, the applications were mostly within single-intervention evaluations. Although two published studies reported the use of VOI in prioritizing research in a group of interventions, these studies ranked research trials based on EVPI or EVPPI values and not ENBS estimates.[7,12] This underreporting may be attributed to the perceived complex EVSI calculation, particularly when two-level Monte Carlo simulation is required.[7,10,12] Other possible reasons may include the lack of awareness about the value of this approach and/or the skepticism regarding its usefulness. While there has been great progress in simplifying VOI computation, with the introduction of novel efficient and flexible methods,[13-18] there is still a need to demonstrate the value and practicality of this approach in real-world cases where a range of interventions is being evaluated.

The National Centre of Research Excellence in Nursing (NCREN) in Queensland is Australia′s first center of research excellence in nursing funded by the National Health and Medical Research Council. The purpose of the NCREN is to provide evidence for clinicians and policy makers to improve patient care. A research focus of the NCREN is interventions promoting skin integrity in hospitalized patients. Our team has been evaluating the cost-effectiveness of these interventions and has applied VOI methods to inform the value and optimal design of future research studies.

The aim of this article is to report the application of VOI methods in a portfolio of NCREN interventions to inform adoption decisions, enhance trial design, and prioritize research studies by answering the following related questions[8,19]:

Is the intervention cost-effective?

Is additional research required? And if yes,

What type of research?

Do the expected benefits of sampling exceed the costs?

What is the optimal research study design?

What priority should this research study take?

Methods

The general approach for each of the included interventions was to 1) conduct cost-effectiveness analysis and characterize decision uncertainty; 2) estimate relevant VOI measures; and 3) consider research expected benefits and costs. The detailed analyses for the included interventions can be found elsewhere.[20-23]

The Interventions

The selected interventions met the following criteria: target a wide population of patients, the evidence to support their adoption is limited, and there were NCREN pilot studies or systematic reviews on their effectiveness. The study included four interventions. The first two interventions are related to intravascular devices: the clinically indicated replacement of intravenous catheters and tissue adhesive in securing arterial catheters. Catheters are the most commonly used devices in hospitals, and therefore, extending the dwelling time (e.g., until complications occur) of the inserted catheters can reduce health care resources in terms of equipment and staff time.[24] The current practice is to change peripheral venous catheters regularly every 72 to 96 hours, regardless of the presence of complications (e.g., phlebitis).[25] For catheter securement, there are various devices to keep the catheters inserted in their place including conventional dressings (e.g., standard polyurethane); however, a novel tissue adhesive material has been proposed as an alternative.[26]

The third intervention is negative pressure wound therapy (NPWT) in women undergoing caesarean section. The standard of care is to apply hydrocolloid dressing on the site of incision following surgery; however, high-risk women (e.g., obese) undergoing caesarean sections may still develop postsurgery complications (e.g., infection). Therefore, there is a growing interest in using NPWT to prevent surgical site infection in this group of patients.[27] The fourth intervention is nutritional support for preventing pressure ulcers in high-risk patients. Older patients with restricted mobility have a high risk of pressure injury leading to prolonged hospitalization.[28] Nutritional support, mainly in the form of high-protein supplements, can reduce the incidence of pressure ulcers in these patients.[29-32]

Cost-Effectiveness Analysis

The cost-effectiveness analyses were from the perspective of Queensland Health, Australia. Costs and monetary benefits were expressed in Australian dollar (AU$) using 2014 prices. The net monetary benefit (NB) was calculated, which is the effect multiplied by the willingness-to-pay (WTP) threshold, minus the cost. A positive incremental net benefit (INB) indicates that the new intervention is the preferred option.[33]

For the clinically indicated catheter replacement intervention, the comparator was routine (i.e., every third day) replacement of catheters. We performed the cost-effectiveness analysis using patient-level data from a randomized controlled trial (RCT) of 3200 patients.[22] The primary outcome of the trial was phlebitis rate during catheterization. Furthermore, resource use data (e.g., staff time, equipment) were collected and valued during the trial.[22] The time horizon of the analysis was 1 month because of the acute nature of both the intervention and the associated adverse event of phlebitis. We set the WTP threshold at AU$0.0 per phlebitis case avoided since the treatment of phlebitis typically consisted only of removal and replacement of the affected intravenous catheter.[22] Uncertainty in the results was characterized using 1000 bootstrap replications.

For the tissue adhesive intervention, our analysis was using parametric modelling of patient-level data from a pilot trial (n = 123) that randomized patients to standard polyurethane dressing, tissue adhesive, bordered polyurethane dressing, or a sutureless securement device.[23] The primary endpoint was catheter failure rate. Data on resources used were collected alongside the trial including the equipment and staff time required for insertion and removal of arterial catheters. Equipment costs were based on negotiated hospital prices, and staff time was valued according to the fixed industrial award wages in Australia. The time horizon for this analysis was also 1 month. We set the WTP threshold at AU$100 per arterial catheter success. Probability distributions were assigned to cost and effectiveness parameters whereby catheter success was assigned a beta distribution and cost data assigned a lognormal distribution.[23] Decision uncertainty was characterized using probabilistic sensitivity analysis (PSA) with 10,000 Monte Carlo simulations.[23]

We conducted cost-utility analyses for the remaining two interventions using two decision models constructed in TreeAge Pro (Version 2014 R1); a decision tree for NPWT compared with hydrocolloid dressing in preventing surgical site infections following caesarean sections, and a six-health-state Markov cohort model for nutritional support compared with standard hospital diet in preventing pressure ulcers.[20,21] To capture the occurrence of the complications and treatment outcomes (e.g., healing), the time horizons of the NPWT and nutritional support interventions were 6 months and 12 months, respectively. The WTP threshold for the two analyses was AU$50,000 per quality-adjusted life-year (QALY) gained.[34] We systematically searched and identified relevant evidence on the cost, utility scores, and efficacy from various sources to populate the models. Due to the scarcity of evidence on NPWT, we combined the data on the relative effectiveness of the device from a pilot study (n = 92) on obese women undergoing caesarean sections with the data from an RCT (n = 81) on NPWT in high-risk patients.[21] We performed a meta-analysis of five RCTs (n = 1381) to estimate the relative effectiveness of nutritional support in preventing pressure ulcers compared with standard hospital diet.[20] Input parameters in the two models were assigned probability distributions; in general, beta distributions for probabilities and utilities, gamma distributions for costs and disutilities, and lognormal distributions for relative risks (RR).[20,21] A PSA of 10,000 Monte Carlo simulation was performed to characterize decision uncertainty in each model.[20,21]

Value of Information Analysis

Monte Carlo simulation was used to calculate VOI measures as described in detail in the literature,[6,35,36] and summarized in Appendix 1 (online). We started the analysis by estimating the EVPI by calculating the difference between the expected NB of a decision with perfect information and the decision made based on current information. However, because decisions are taken at the population level, population-EVPI was determined via multiplying the per-patient estimate by the total number of patients who will benefit from additional information over the expected lifetime of the intervention.[4,19] Based on expert advice, the lifetime for both clinically indicated catheter replacement and tissue adhesive interventions was 5 years, whereas the lifetime for both NPWT and nutritional support was 10 years. All future population values were discounted at 5% annual rate. Furthermore, sensitivity analyses were performed to test the impact of varying technology lifetimes and WTP threshold on VOI estimates.

When the population EVPI appears to be too small compared with the expected research costs, additional research would not be required, and accordingly, the decision would be to adopt or reject the intervention based on the current evidence. On the other hand, when the population EVPI is likely to exceed the costs of additional research, then further research is potentially worthwhile. In this case, the EVPPI would be calculated to know the focus and type of additional research. The next step was to calculate population-EVSI to estimate the value of the future research study (with specific sample size) in reducing decision uncertainty.

Considering Research Benefits and Costs

To establish the sufficient condition for decision making, we compared the expected research benefit (i.e., population-EVSI) with its expected total cost. The total cost has two components, one financial (i.e., direct costs) and the other reflecting opportunity loss.[37] Direct trial costs comprise the fixed costs of setting up a trial and variable costs, which is the cost per patient. Direct costs of research were obtained from the estimated research costs for relevant grant applications submitted by the NCREN researchers. The opportunity cost component stems from the fact that patients allocated to the standard intervention arm will not benefit from the new intervention; similarly, the eligible population that is not included in the trial will also incur opportunity costs awaiting the results and intervention implementation.[38,39] The ENBS is the difference between the population-EVSI and the total cost for a future research study.[4,8]

When the expected direct research costs exceed the expected benefits, the research would not be cost-effective and the decision should be to adopt or reject based on the results of the cost-effectiveness analysis. When the expected research benefits exceed direct research costs, we need to assess whether we should delay the decision and wait for the results of the future research (i.e., only in research [OIR]), allowing for opportunity costs of delay, or approve the intervention and undertake the research study (i.e., approve with research [AWR]), allowing for irrecoverable costs.[2,40] Irrecoverable costs are the sunk costs that cannot be recovered when a decision is reversed.[19,41] These include costs associated with intervention implementation (e.g., purchasing equipment, training) and an opportunity cost when the benefits of additional research are forgone because the research is less likely to take place when the intervention is implemented.[2,19,40] In this study, and assuming that research would be possible with approval, OIR would be decided if the intervention is not cost-effective but additional research is worthwhile, or if the intervention is cost-effective and additional research is worthwhile but there is significant irrecoverable costs. AWR would be decided if the intervention is cost-effective, additional research is worthwhile, and the irrecoverable costs are lower than the opportunity cost of delay (Figure 1).[40,41]

Figure 1

Flowchart of decision pathways in the study.

The most efficient design of a future trial would be the one that maximizes the ENBS. Trial design dimensions would include sample size, number of comparators, and follow-up duration. Additionally, we calculated and compared the return on investment (ROI) estimates, which is the ENBS to cost ratio, for the sample sizes determined using VOI and those determined using the traditional approach of hypothesis testing. Finally, the future research studies were ranked according to their expected monetary benefits from highest to lowest.

Results

Is the Intervention Cost-Effective?

The INB of the clinically indicated catheter replacement intervention was AU$7.60, suggesting that it was the preferred option compared with routine catheter replacement.[22] The probability of this intervention being cost-effective was almost 100%. For the tissue adhesive for securing arterial catheters, the intervention had the highest net monetary benefit at AU$14.10 compared with other devices, indicating that it was the preferred device.[23] The probability of tissue adhesive being cost-effective was 35%. NPWT was cost-effective compared with standard dressing with an INB of AU$70.00.[21] The probability of NPWT being cost-effective was 65% at a WTP of AU$50,000 per QALY gained. Nutritional support was cost-effective compared with standard hospital diet with an INB of AU$675.00.[20] The probability of this intervention being cost-effectives was 87% (Table 1).

Table 1

Cost-Effectiveness and Value of Information Analyses Results

Intervention	INB (AU$)	Probability Cost-Effective	EVPI (AU$)	Comp. Time	EVPPI[a] (AU$)	Comp. Time	EVSI[b] (AU$)	Comp. Time	Total Cost (AU$)	ENBS (AU$)
Clinically indicated catheter replacement	7.60	100%	0	Seconds	—	—	—	—	—	—
Tissue adhesive	14.10	35%	850,000	Seconds	—	—	573,324	1 min	250,000	325,324
NPWT in CS	70.00	65%	2,700,000	Seconds	2,600,000	1 min	1,200,000	1 min	900,000	1,200,000
Nutritional support in PU	675.00	87%	5,500,000	Seconds	2,500,000	4 h	970,000	8 h	870,000	100,000

Note: AU$ = Australian dollar; INB = incremental net benefit; EVPI = expected value of perfect information; EVPPI = expected value of perfect parameter information; EVSI = expected value of sample information; ENBS = expected net benefit of sampling; NPWT = negative pressure wound therapy; CS = caesarean section; PU = pressure ulcer; Comp. Time = approximate computation time.

For the parameter with the highest value (i.e., relative risk).

For a sample size of 220 patients in the tissue adhesive, 200 patients in the NPWT, and 1200 patients in the nutritional support.

Is Additional Research Required?

For the clinically indicated catheter replacement, the EVPI was approximately zero, indicating that additional research is not worthwhile, and therefore, the decision should be to adopt the intervention based on the available evidence. Conversely, the population EVPIs for other interventions were AU$850,000 for tissue adhesive, AU$2.7 million for NPWT, and AU$5.5 million for the nutritional support intervention, suggesting that additional research is likely worthwhile. The calculated EVPI values were sensitive to the probability of an intervention being cost-effective, the WTP threshold, and the population expected to benefit from the intervention. The sensitivity analyses confirmed that EVPI was negligible for the clinically indicated catheter replacement and remained above AU$300,000 for the rest of the interventions.

What Type of Research?

For the tissue adhesive intervention, VOI measures were calculated for both cost and effect parameters from the pilot study; thus, the future study should collect data on both costs and effects. For the NPWT, the parameter with the highest population EVPPI was the RR of surgical site infection at AU$2.6 million.[21] This suggested that the additional research is warranted in the form of an RCT comparing the relative effectiveness of NPWT with standard dressing in preventing surgical site infections in high-risk caesarean section patients. For the nutritional support intervention, the parameter with the highest EVPPI was the RR of pressure ulcer at AU$2.5 million.[20] This indicated that a future RCT should study the relative effectiveness of nutritional support in preventing pressure ulcers compared with standard hospital diet in high-risk patients.

Do the Expected Benefits of Sampling Exceed the Costs?

For the tissue adhesive intervention, the ENBS was positive for a future study with a sample size ranging from 50 to 980 patients per arm for all possible designs (i.e., two- to four-arm trial designs).[23] There was insignificant opportunity cost for delaying adoption, but there are potential irrecoverable costs for the introduction of the adhesive material into the health care system, and therefore, additional research before adoption is worthwhile and the decision in this case would be OIR. In the NPWT intervention, the ENBS remained positive, even with the opportunity cost of delaying adoption, across a range of possible sample sizes from 50 to 800 patients in each arm.[21] This suggested that the expected benefits of additional research would exceed the expected costs of delay. Moreover, there would be a high irrecoverable cost from purchasing and stocking NPWT devices because it is less likely for these to be used in other indications. Thus, additional research before adoption is worthwhile and the decision should be OIR. On the other hand, withholding the adoption of nutritional support intervention until the results of the future study are known would result in a negative ENBS. Nevertheless, adopting the intervention concurrently with research would result in a positive ENBS for a range of sample sizes between 900 and 2000 patients per arm.[20] There is minimal irrecoverable cost associated with the possible reversal of implementation decision based on the findings of the future study because there are no adoption expenses (e.g., training or equipment).[20] Furthermore, the intervention is intended to support standard hospital diet, and therefore, there are no benefits forgone by replacing the standard intervention.[20] Thus, the decision should be AWR. Figure 2 illustrates the EVSI and ENBS for the future research studies.

Figure 2

Expected value of sample information and expected net benefit of sampling curves for tissue adhesive (A), negative pressure wound therapy (B), and nutritional support intervention (C).

What Is the Optimal Research Study Design?

For a future trial collecting data on the effects and costs of tissue adhesive compared with other catheter securement devices, the ENBS would be maximized at AU$325,000 in the four-arm design with 220 patients in each arm, for 1 year of follow-up at a total cost of AU$250,000.[23] This would provide an ROI of 130%.[23] The ENBS from the initially calculated sample size of 388 patients per arm was AU$280,000, providing an ROI of 79%.[23] In a sensitivity analysis, the optimal design remained with four arms and a sample size of 150 to 250 per arm when the lifetime of the technology was increased to 10 years and the WTP threshold varied between AU$50 and AU$400 per catheter success.

For the future trial on NPWT, the optimal design would have a sample size of 200 patients receiving NPWT and 200 receiving usual care with a follow-up duration of 11/2 years. This design would give a maximum ENBS of AU$1.2 million at a total cost of AU$900,000, resulting in a 133% ROI.[21] The initial design with 400 patients per arm would provide an ROI of 66%.[21] The optimal sample size range was 100 to 300 patients per arm when the WTP threshold varied between AU$25,000 and AU$75,000 per QALY gained and the technology lifetime altered between 5 and 15 years.

In the future trial studying the relative effectiveness of nutritional support in preventing pressure ulcers, the ENBS would be maximized at AU$100,000 with 1200 patients receiving nutritional support and 1200 in the hospital diet arm, providing an 11% ROI.[20] The sensitivity analysis showed that the sample size ranged from zero (i.e., research is not cost-effective) to 1400 patients per arm when the WTP threshold varied between AU$25,000 and AU$75,000 per QALY gained and the lifetime of the technology ranged between 5 and 15 years.

What Priority Should This Research Study Take?

The future research studies can be ranked based on their ENBS estimates. Among the three interventions where future research is worthwhile, the ENBS would be the highest for the NPWT at AU$1.2 million, followed by tissue adhesive with AU$325,000, and nutritional support with AU$100,000. The same ranking would be kept if the future studies were ranked according to their ROI with 133% for the NPWT, 130% for tissue adhesive, and 11% for the nutritional support intervention. A sensitivity analysis showed that the ENBS for the tissue adhesive intervention would range from AU$300,000 to AU$600,000 when the lifetime of the technology was increased to 10 years and the WTP threshold varied between AU$50 to AU$400 per catheter success. Varying the WTP threshold from AU$25,000 to AU$75,000 per QALY gained and the intervention lifetime between 5 and 15 years resulted in an ENBS range between AU$400,000 and AU$3 million for the NPWT intervention and an ENBS range between negative (i.e., research is not worthwhile) and AU$500,000 for the nutritional support intervention.

Discussion

This is the first study to apply VOI in a portfolio of real-world interventions under a center for research excellence to inform the decision on further research, improve study design, and set research priority using the sufficient condition of ENBS estimation. The evaluated interventions appeared to be cost-effective, but there was uncertainty in the adoption decision and this uncertainty varied across the interventions. VOI analysis guided the decision on whether to adopt an intervention based on current evidence or if additional research to reduce uncertainty is likely worthwhile. When additional research was required, VOI analysis informed the research aspects that should be investigated and the most efficient designs of future studies. Furthermore, the prioritization of the future studies was set based on their ENBS values.

The evaluated interventions had different levels of evidence to support their adoption. Ideally, all relevant current evidence to inform cost-effectiveness analysis should be sought from various sources. Nevertheless, economic evaluation alongside a single trial remains attractive due to its high internal validity, particularly when that trial is the only evidence to answer the decision problem, which was the case in the two catheter-related interventions.[42] For the NPWT and nutritional support interventions, the economic evaluations were based on analytic modelling with parameter information gathered systematically from various sources.

Monte Carlo simulation is the standard approach for VOI measures calculation and it was the main method used in this study. This approach is flexible and can be used in economic models and alongside clinical trials. One of the perceived challenges for a wider VOI application is the computation burden of VOI measures; however, the calculation of VOI measures was not the biggest challenge in this study. In fact, the most demanding tasks were performing the cost-effectiveness analyses and characterizing decision uncertainty. EVPI calculation was straightforward from uncertainty characterization (e.g., PSA sample). One-level Monte Carlo simulation to calculate EVSI and EVPPI in the linear models (the decision tree) took seconds. Even in the Markov model for the nutritional support the calculation took around 4 hours for each EVPPI estimate and around 8 hours for each EVSI sample size estimate. This faster than expected computation in this study can be attributed to a number of factors: first, the models used were relatively simple; however, each model was a good representation of the decision problem and were validated by expert clinicians. Second, EVSI calculation assumed conjugacy between prior parameter distributions and proposed data likelihoods; thus, the parameters for posterior distributions could be readily calculated using closed forms. Third, the calculation was using modern computers that tend to perform faster than the older computers used a decade or two ago when VOI was first introduced. Recently, efficient methods have been developed and applied for calculating multiparameter EVPPI and EVSI directly from PSA samples.[13,14,16] It would be useful to see how the new methods compare with Monte Carlo simulation in real-world scenarios of models of various complexities.[13]

This study highlighted the value of making simultaneous adoption and research decisions by considering the tradeoff between the benefits and costs of additional research (i.e., VOI analysis) as well as the costs and consequences of acting now against waiting for more information (i.e., options analysis). Keeping options open to change a decision is a reasonable approach for risky decisions with irreversible costs.[40] For instance, there is no value in additional research on clinically indicated catheter replacement intervention and this strategy should be adopted, and this recommendation has been included in recent clinical guidelines on intravenous catheters.[43] For the tissue adhesive and NPWT interventions, two large trials are now recruiting.

Regarding trial designs, VOI analysis provides an alternative and economic approach to the traditional method of sample size calculation based on type I and II error and the smallest clinically significant difference.[4,5] Sample sizes calculated using VOI analyses were more economical (i.e., smaller) than those calculated using the traditional approach. Furthermore, the study demonstrated how VOI can inform broader aspects of research design such as the number of trial arms and follow-up duration.[39,44] The NCREN research team has used the results of this study to guide the design of larger trials (e.g., Securing All intra Venous devices Effectively Trial) and to demonstrate the value of research in grant applications.

An important addition in this article is the use of ENBS to establish research priorities. The future research studies would be prioritized based on their ENBS as 1) NPWT (ENBS = AU$1.2 million); 2) tissue adhesive (ENBS = AU$325,324); 3) nutritional support (ENBS = AU$100,000). Importantly, prioritizing these studies based on their EVPI or EVPPI estimates, as suggested by previous studies,[7,12] might lead to a suboptimal ranking. For instance, if the future studies in this study were ranked according to their EVPI′s, the ranking would be 1) nutritional support (EVPI = AU$5.5 million); 2) NPWT (EVPI = AU$2.7 million); and 3) tissue adhesive (EVPI = AU$850,000). Giving priority to a trial with lower ENBS than other competing studies would lead to inefficient utilization of limited research resources. There will be an opportunity cost when a more beneficial research project is abandoned or delayed as resources are directed to less beneficial research proposals.

The estimated VOI measures in this study may be affected by the assumptions made in the analyses such as the model structure, WTP thresholds, and the lifetime of the technology. At the single intervention level, we performed sensitivity analyses to explore the effect of varying WTP and technology lifetimes on VOI results. While the sensitivity analyses suggested that the results were sensitive to the assumptions made, the overall conclusions would not change. Nevertheless, an important point to consider here is that the ENBS′s used to rank studies were estimated for different interventions using different approaches and assumptions for cost-effectiveness analysis (e.g., trial based v Markov models), uncertainty handling (e.g., inputs parametrization), and VOI calculation. Although a sensitivity analysis to generate possible ENBS ranges for the different interventions may be useful to explore how sensitive the ranking is to certain assumptions (e.g., technology lifetime), it remains challenging to control for the heterogeneity in the compared evaluations. Developing VOI guidelines to standardize VOI calculation procedures may allow better comparison of the VOI measures estimated by different analysts for various interventions.[45] Furthermore, there is a possibility that the VOI measures calculated were overestimated because we assumed perfect implementation of the interventions.[45,46] In practice, perfect implementation is impossible due to uptake barriers (e.g., resistance to change). Moreover, it is likely that implementation will differ between the different interventions, and thus, not adjusting for implementation may bias study rankings. Considering the VOI simultaneously with the value of implementation in decision making was addressed in the works of Fenwick and Hoomans,[47,48] which have been recently extended by Andronis and Barton who proposed an approach to adjust EVSI for the level of implementation.[49]

This study demonstrated the usefulness and practicality of applying VOI analysis in health care interventions. VOI analysis simultaneously informed implementation and research decisions in a group of real-world interventions by considering the marginal costs and marginal benefits of additional research. It also emphasized that methodological and computational issues that are often perceived with this approach should not be the biggest challenge for its wider application. With the recent introduction of efficient VOI methods, efforts should be focused on encouraging the use of VOI approach and facilitating its incorporation into decision making frameworks. This may be achieved by effective communication with stakeholders (i.e., decision makers, researchers) to enhance their understanding of the approach and its value. It is also essential to know the needs, expectations, and concerns of the different stakeholders and to study the facilitators and the barriers for VOI methods uptake.