Use of Value of Information in Healthcare Decision Making: Exploring Multiple Perspectives.

BACKGROUND: Value of information (VOI) is a tool that can be used to inform decisions concerning additional research in healthcare. VOI estimates the value of obtaining additional information and indicates the optimal design for additional research. Although it is recognized as good practice in handling uncertainty, it is still hardly used in decision making in the Netherlands.
OBJECTIVE: This paper aims to examine the potential value of VOI, barriers and facilitators and the way forward with the use of VOI in the decision-making process for reimbursement of pharmaceuticals in the Netherlands.
METHODS: Three focus group interviews were conducted with researchers, policy makers, and representatives of pharmaceutical companies.
RESULTS: The results revealed that although all stakeholders recognize the relevance of VOI, it is hardly used and many barriers to the performance and use of VOI were identified. One of these barriers is that not all uncertainties are easily incorporated in VOI, and the results may be biased if structural uncertainties are ignored. Furthermore, not all research designs indicated by VOI may be feasible in practice.
CONCLUSIONS: To fully embed VOI into current decision-making processes, a threshold incremental cost-effectiveness ratio and guidelines that clarify when and how VOI should be performed are needed. In addition, it should be clear to all stakeholders how the results of VOI are used in decision making.

Key Points for Decision Makers

Introduction

Decision making in healthcare is often informed by economic evaluations, and—in many of those decisions—uncertainty exists concerning the effectiveness and costs of the technology under study [1]. Value of information (VOI) is a tool that can be used to study the uncertainty associated with a coverage decision and its implications. VOI estimates the value of collecting additional data to reduce decision uncertainty and indicates the optimal design for additional research to obtain these data by combining the probability and monetary consequences of an incorrect decision [2]. To assess the value of reducing all decision uncertainty, the expected value of perfect information (EVPI) can be calculated. The expected value of partial perfect information (EVPPI) is the EVPI focused on one parameter or a subset of parameters. More extensive VOI analyses can be used to examine the value of a certain sample size—the expected value of sample information (EVSI)—and to trade off this value against the costs of additional research by calculating the expected net benefit of sampling (ENBS) [3]. Finally, methods like real options analysis (ROA) can assist in deciding whether to adopt a health technology for reimbursement now (with the risk of investing in a suboptimal therapy) or to wait for more evidence (with the risk of withholding from patients the optimal treatment: opportunity costs). This trade-off becomes particularly relevant when the decision to adopt a health technology is difficult to reverse due to high costs or other factors [4].

In the Netherlands, a coverage with evidence development (CED) policy for expensive intramural drugs was implemented in the last decade. Intramural drugs are part of the treatment carried out by or under the responsibility of a medical specialist. This policy implied that intramural drugs with added therapeutic value and an annual budget impact exceeding €2.5 million should be reassessed after 4 years to decide upon their definitive adoption [5]. During these 4 years, additional data are collected to inform the adoption decision, mainly through the use of registries. Within CED policy, companies were obliged to submit an initial assessment dossier to the National Health Care Institute in the Netherlands [Zorginstituut Nederland (ZIN)]. It is argued that VOI can provide a useful contribution in the process between initial assessment and re-assessment, e.g. deciding upon the value of applying the CED policy in the first place, and directing further research [6].

Although VOI is described as best practice for handling decision uncertainty [7], its application remains limited. Document analysis of five CED cases published between 2013 and 2014 revealed that no VOI analyses were performed in these five cases (see the “Appendix”). These five cases represent all cases published by 2014 for the Netherlands, excluding those drugs that were used for multiple indications. If a drug was used for multiple indications, only the most recent case was included in the document analyses. Information concerning the perspectives of stakeholders on the use of VOI is lacking [8]. The aim of this paper is to examine (1) the potential value of VOI within a CED policy, (2) the barriers and facilitators of using VOI and (3) the way forward with using VOI in the reimbursement of health technologies perceived by different stakeholders (policy makers, researchers and pharmaceutical companies) in the Netherlands.

Methods

To examine the potential value, barriers, facilitators and the way forward with VOI, three focus groups were conducted with stakeholders and analysed in accordance with the COREQ (Consolidated Criteria for Reporting Qualitative Research) guidelines [9].

Focus Groups

Participants

Representatives from the pharmaceutical companies, policy makers and researchers all currently involved in reimbursement decisions in the Netherlands were invited to participate in homogenous focus groups in 2014. In-depth and rich insights can be gathered, and different views concerning VOI can be explored, via the interaction among participants in a focus group [10]. The following participants were included: managers and market-access managers from pharmaceutical companies who had submitted an assessment or re-assessment dossier; academic and non-academic researchers in the field of cost-effectiveness modelling; and policy makers from ZIN, the Netherlands Organization for Health Research and Development (ZonMW), the Health Council of the Netherlands, and the National Institute for Public Health and the Environment (RIVM). Since this study is a collaborative effort between different research groups in the Netherlands and ZIN, relevant stakeholders could be identified through the professional networks of the project team and approached via email. Three focus groups were organized in which 13 researchers, five policy makers and seven representatives from the pharmaceutical industry participated. Familiarity with VOI was not an inclusion criteria; therefore, the senior researcher introduced VOI during a short presentation at the start of the focus group.

Data Collection and Analysis

All three focus groups were facilitated by the first and second authors (JB and BR). Each focus group was moderated by one senior researcher (MA, TF, MP) and started with a short presentation on the methodology of VOI. Subsequently, the following topics were discussed: experience with VOI, the current use of VOI, the value of VOI, barriers and facilitators in using VOI, way forward with VOI and its place in the decision-making process. The members were categorized as having ‘no experience’, ‘moderate experience’ or ‘much experience’ to indicate the level of experience with VOI. Members who had not used the results of or performed a VOI themselves were classed as having no experience, members who had used the results of or performed a VOI only a few times were classed as having moderate experience and members who had performed extensive VOI analyses and used VOI repeatedly were classed as having much experience. The focus groups lasted 2 h and were tape recorded and transcribed verbatim. Content analysis was used to analyse the transcripts of the focus group [11]. First, two researchers (JB and BR) allocated open codes to text fragments in the focus group transcript; based on these codes, a report of the meetings was developed and sent back to the participants for validation [10, 12]. The reactions of the participants were integrated by adjusting the codes. A data matrix was designed to visualize the codes, the related citations from the transcripts and the categories of the three focus groups and was discussed with all project members.

Results

Perspectives of Stakeholders on Value of Information (VOI)

Although some researchers and participants from the pharmaceutical industry had experience in performing VOI, most of the participants expressed having limited experience with VOI. The main results are summarized in Table 1.

Table 1

Main results from focus groups

ICER incremental cost-effectiveness ratio, ROA real options analysis, VOI value of information

Value of VOI

Respondents agreed that VOI can be a relevant tool to decide upon the need for and type of additional research, although some respondents questioned the additional value of VOI compared with one-way sensitivity analyses and/or probabilistic sensitivity analysis (PSA). Respondents stated that VOI results could be used to decide which research designs would be necessary to reduce uncertainty. Besides directing additional research, VOI results could also be helpful in supporting the decision that further research is unnecessary. Focus group participants from the pharmaceutical industry mentioned that VOI in combination with a ROA could give more information concerning the required timeframe for additional research. Furthermore, policy makers and researchers mentioned that VOI can be helpful in prioritizing research subjects.

Barriers and Facilitators

Participants in all three focus groups stated that not all optimal research designs, indicated by VOI, are feasible. This could be due to small patient populations or because other drugs are used for the same indication. A randomized controlled trial was perceived as unfeasible if the whole patient population was using the drug under study and was considered unethical if the necessity of the new drug was very high. These aspects were experienced as barriers to VOI in situations in which it is clear in advance that specific research designs are impossible or not accepted by policy makers as evidence. Researchers and participants from the pharmaceutical industry therefore argued that international data collection should be considered (especially in diseases with a low prevalence). Participants further explained that it is not always feasible to collect real-world data, since clinicians are often responsible for the data collection and it can be difficult to adjust or expand existing registries. Representatives of the pharmaceutical industry experienced a gap between the evidence that is required by policy makers and the evidence that can be generated by clinicians in real-world settings.

Various participants mentioned that VOI does not incorporate all uncertainties. The policy makers highlighted that other uncertainties, such as that related to the prescription behaviour of professionals, should also be considered in decisions concerning additional research. Researchers also mentioned that VOI results can provide a distorted view of all existing uncertainties, since VOI often disregards the structural uncertainties of a model. The pharmaceutical industry participants further questioned which perspective should be taken into account in VOI and who should pay for the additional research.

Methodological and practical barriers to performing VOI were also identified in the focus groups. Representatives from pharmaceutical companies indicated that performing VOI may be complex and that policy makers have limited knowledge in interpreting the results. The feasibility of performing VOI may be hindered by lack of expertise and the fact that some VOI analyses take a long computational time. In the focus group with the researchers, the different types of VOI analyses were discussed; one researcher mentioned that performing an EVPI alone gives insufficient information to policy makers. Further analyses, including the EVPPI and EVSI, are necessary to indicate the group of parameters for which it would be worthwhile to collect additional data and which sample size is required.

Way Forward with VOI

In the focus group with the pharmaceutical industry, participants mentioned a situation in which VOI was performed and presented to the decision makers, after which the VOI results were ignored in the decision concerning additional research. An overall impression of researchers and the pharmaceutical company representatives is that VOI is pointless unless the VOI results are used in decision making concerning coverage and additional research. This implies that policy makers should accept the results, including if VOI indicates that additional research is not valuable.

Although all participants agreed that VOI is not always necessary, some policy makers expressed the fear that if VOI is an optional extra, it will not be used in practice. The participants of all three focus groups agreed that it is necessary to develop criteria as to when VOI should be performed. There was no consensus among the participants about those criteria, apart from the general idea that VOI should be used when uncertainty exists concerning cost effectiveness. For example, VOI has no value in cases where the incremental cost-effectiveness ratio (ICER) is extremely high or extremely low and it is already clear at the initial assessment that no substantial changes in the ICER are expected. Individual participants mentioned that VOI should be used for drugs with a high budget impact, for expensive drugs and in combination with coverage and evidence-development schemes. All participants agreed that deciding on these criteria should be a collaborative effort between policy makers, researchers and the pharmaceutical industry. Researchers further indicated that it is important to create guidelines on how VOI should be performed. Researchers and pharmaceutical industry representatives pointed out that a threshold ICER is necessary to interpret the results of VOI. The pharmaceutical industry participants stated that the maximum budget that can be spent on additional research by each funding party (government, pharmaceutical industry) should be agreed upon and taken into account when deciding upon additional research. Furthermore, some policy makers suggested that VOI could also be useful for directing effectiveness research using a threshold for minimum clinical relevant difference.

Discussion

The current use of VOI in practice remains limited, although stakeholders agree that VOI potentially has value to inform research decisions. The participants of the focus groups perceived that the optimal research designs indicated by VOI may not be feasible in practice. Some practical challenges for using VOI could be identified, such as the long computational time and a lack of knowledge concerning VOI. Furthermore, not all uncertainties that should be regarded in health policy decision making are easily incorporated in VOI. Guidelines that clarify how VOI should be performed are needed. Moreover, a threshold ICER and clarity on how the VOI results will be used in decision making regarding adoption and additional research are needed to fully imbed VOI in current decision making.

VOI can support the adoption decision, because it goes beyond reflecting the probability of making a wrong decision by showing the consequences of such a decision [7, 13]. In that respect, VOI can also be used in negotiations for risk-sharing agreements and price arrangements. This use of VOI appeared to be less obvious to the stakeholders, as some respondents questioned the added value of VOI compared with other uncertainty analyses like PSA. Other stakeholders stated that EVPI alone was regarded as insufficient. This does not imply that all VOI analyses are always necessary; to decide if further research is necessary, an EVPPI may be sufficient to prioritize groups of parameters for further research projects [14, 15]. In general, more clarity is needed on which VOI analyses are to be performed in which situation.

The fact that randomized controlled trials are often unfeasible was perceived as a barrier to perform VOI. Particularly for less prevalent diseases, it may be desirable to initiate international research projects, and evidence from (ongoing) international studies could be included in the VOI analyses. This may enable more informative registries, especially for rare diseases. However, many authorities also require local information that is not easily obtained from international studies (e.g. local resource use or quality of life).

A CED policy decision should also take into consideration which research designs are practically and ethically feasible. Hence, VOI analyses should be performed for the study designs that are considered feasible for a specific case. In addition, the long computational time required to perform certain analyses was mentioned as a barrier to VOI. This point has also been raised by other researchers [8, 14]. Participants described that VOI can be complex and that knowledge on how to perform and interpret VOI may be lacking. However, in the literature, new tools have been described that simplify VOI calculations and reduce computational challenges [16-18]. The focus group with researchers did emphasize the importance of incorporating structural uncertainty to obtain robust and credible VOI assessments. Incorporating uncertainties like the uncertainty concerning the choice of comparators and the inclusion of specific events in the modelling process is important for decision-making purposes [19, 20]. The policy makers stressed the importance of uncertainties unrelated to the cost-effectiveness assessment. These uncertainties (e.g. ethical dimensions of a decision) should be considered separately by decision makers and could be addressed more formally in a multi-criteria decision analysis [21].

The focus groups results show that VOI is only relevant if it is embedded in the policy window of a jurisdiction regarding adoption decisions. The respondents stressed that decision makers should incorporate the VOI results in the decision-making process if these analyses are requested. VOI outcomes can guide further research, even though there is no fixed threshold ICER, if VOI does not vary within the range of plausible threshold ICERs. However, if VOI results are highly variable within a plausible range of threshold ICERs, a fixed threshold is needed to interpret the results of VOI and to use VOI in decision making [22]. In the Netherlands, as well as in many other countries, this threshold has not been formally defined [22, 23]. Different unofficial thresholds are described in the literature, e.g. a threshold of €20,000 per quality-adjusted life-year for preventive interventions [24]. The focus group revealed that guidelines are needed that clarify the decision problems for which VOI should be performed, since the respondents agreed that VOI is not necessary in all cases. VOI could be useful if uncertainty concerning the cost effectiveness of expensive drugs or drugs with a high budget impact is substantial.

The focus groups were performed in the Netherlands; however, the insights gathered concerning the use of and way forward with VOI might be useful for other jurisdictions that apply a CED policy and a threshold ICER. However, by examining effectiveness only (e.g. using a threshold for minimum clinically relevant difference), VOI could also be used in countries that do not use cost-effectiveness information for coverage decisions (e.g. Germany and the USA) [25].

To fully embed VOI in healthcare, a threshold ICER and guidelines that clarify how VOI should be used within research and reimbursement decisions are needed.

Value of information (VOI) is considered valuable to guide decisions on additional research and adoption concerning pharmaceuticals.

VOI is currently not common practice in research and decisions concerning pharmaceuticals.

Practical guidelines are needed that indicate how to perform VOI.

A threshold incremental cost-effectiveness ratio (ICER) should be defined to enable VOI results to be interpreted.

A policy framework is needed that includes criteria of when to perform VOI, agreements on how the results are being used and on who pays what amount for additional research.

Table 2

Document analysis

	Agalsidase alfa (Replagal®) and agalsidase beta (Fabrazym®)	Alglucosidase alfa (Myozyme®)	Infliximab (Remicade®)	Omalizumab (Xolair®)	Ranibizumab (Lucentis®)
Timings of initial and re-assessment dossier	May 2007–Jun 2011	Jul 2006–Feb 2011	Oct 2006–Feb 2011	May 2006–Mar 2011	Apr 2007–Jun 2011
Type of economic analysis	Cost effectiveness	Cost utility	Cost effectiveness	Cost effectiveness	Cost effectiveness
Comparator	BSC (current care/natural history of the disease without enzyme-replacement therapy)	Usual supportive care	Cyclosporine; colectomy with IPAA	Usual medical care	BSC and photodynamic therapy
Population	All adult pts in the Dutch Fabry study cohort	Children and adults with late-onset Pompe disease	Adult pts with moderate to severe UC who responded inadequately to or are intolerant to conventional therapy or to whom such therapy is contraindicated	Adults and adolescents (>12 years) with severe persistent allergic asthma	Pts with AMD
ICER	€3,282,252	Children: €232,699Adults: €2,700,000	Infliximab vs. cyclosporine: €23,585 (NE quadrant); infliximab vs. colectomy with IPAA: €15,057 (NE quadrant)	€39,215 NE quadrant (in initial assessment dossier); €35,257 NE quadrant (in re-assessment dossier)	Lucentis was dominant in all cases except for the MARINA trial-based comparison with BSC (€40,397/QALY)An additional ICER was estimated (using published evidence) vs. bevacizumab: €53,453/QALY, i.e. using bevacizumab instead of lucentis would lead to savings of €3,658 and a QALY loss of 0.09
Was PSA undertaken?	Yes (in re-assessment dossier)	Yes (in re-assessment dossier	Yes (in re-assessment dossier)	Yes (in initial assessment and re-assessment dossier)	Yes (in re-assessment dossier)
Was VOI undertaken?	No	No	No	No	No
Additional research performed	Single-centre observational study	National prospective observational study with historic controls and data from the IPA survey were used	Retrospective chart analyses of UC pts treated with infliximab since 2003 for which two Dutch medical centres (one university hospital and one general hospital) were included	A European multi-centre pt outcomes registry (experience) was conducted for the observational collection of data. Retrospective data on the medical history and resource use over the year prior to the start of omalizumab treatment were also collected	Prospective pt registry

AMD age-related macular degeneration, BSC best supportive care, ICER incremental cost-effectiveness ratio, IPA International Pompe Association, IPAA ileo-pouch-anal anastomosis, NE north-east, PSA probabilistic sensitivity analysis, pt(s) patient(s), QALY quality-adjusted life-year, UC ulcerative colitis, VOI value of information