Weighing Clinical Evidence Using Patient Preferences: An Application of Probabilistic Multi-Criteria Decision Analysis.

The need for patient engagement has been recognized by regulatory agencies, but there is no consensus about how to operationalize this. One approach is the formal elicitation and use of patient preferences for weighing clinical outcomes. The aim of this study was to demonstrate how patient preferences can be used to weigh clinical outcomes when both preferences and clinical outcomes are uncertain by applying a probabilistic value-based multi-criteria decision analysis (MCDA) method. Probability distributions were used to model random variation and parameter uncertainty in preferences, and parameter uncertainty in clinical outcomes. The posterior value distributions and rank probabilities for each treatment were obtained using Monte-Carlo simulations. The probability of achieving the first rank is the probability that a treatment represents the highest value to patients. We illustrated our methodology for a simplified case on six HIV treatments. Preferences were modeled with normal distributions and clinical outcomes were modeled with beta distributions. The treatment value distributions showed the rank order of treatments according to patients and illustrate the remaining decision uncertainty. This study demonstrated how patient preference data can be used to weigh clinical evidence using MCDA. The model takes into account uncertainty in preferences and clinical outcomes. The model can support decision makers during the aggregation step of the MCDA process and provides a first step toward preference-based personalized medicine, yet requires further testing regarding its appropriate use in real-world settings.

Key Points for Decision Makers

Introduction

Assessing the relative effectiveness of treatments is an essential component in regulatory and reimbursement decisions and is substantiated by strong methodology and clinical evidence development guidelines. However, interpretation of clinical evidence is a largely subjective and opinion-based process and such judgments are rarely formally included in decision-making processes [1]. Nevertheless, subjective value judgments of stakeholders (in particular those of patients [1-4]) are an essential part of what finally determines treatment value [5]. For instance, in decisions considering two or more equally effective drugs, the (subjective) relative severity of the associated adverse events may dominate the final decision. Currently, subjective value judgments such as the relative severity of adverse events are mostly considered implicitly in healthcare policy decisions. Patient engagement is increasingly promoted [5-7], and several mechanisms of patient engagement have been used, including patient panels and patient-reported outcomes. Whereas decision makers can use such implicit viewpoints, these can still be inaccurate or biased [8, 9]. A potentially more representative and transparent approach is the use of results from survey-based stated preference studies [10]. These studies yield numeric estimates of the relative importance that respondents place on attributes of medical services such as clinical outcomes or other treatment characteristics [1, 2].

Although elicited preferences can be used as a piece of information in the deliberative process of assessing the relative treatment value, a more formal approach would actually use such preferences to weigh clinical endpoints and, thus, prioritize interventions by explicitly mapping the benefits and risks. One method that could be used to structure and analyze decision problems and to explicitly include patient preferences is multi-criteria decision analysis (MCDA). The application of MCDA allows decision makers to structure their decision problem and work on it in a transparent and consistent way [11, 12]. Importantly, MCDA is flexible in that it allows multiple stakeholder groups, including patients, to assign preference weights to clinical outcomes (which are referred to as criteria in MCDA). An MCDA process typically comprises several steps, including (1) definition of the decision problem and alternatives; (2) identification of the decision criteria; (3) weighting of the criteria; (4) identification of the performance of the alternatives; (5) scoring the performance; (6) aggregating the results and dealing with uncertainty; and (7) reporting the results [13].

The use of MCDA in healthcare has grown over the last few years [14, 15], which has led to the International Society For Pharmacoeconomics and Outcomes Research (ISPOR) MCDA Emerging Good Practices Taskforce publishing guidance on the basic concepts and implementation of MCDA in healthcare recently [13, 16].

The most commonly used method for aggregating the criterion weights and performance scores in MCDA is the linear additive value function. By combining the criterion weights derived from a group of patients and the actual clinical performance scores of the therapeutic alternatives, one can estimate patient-weighted treatment values [17]. This is a relatively straightforward approach but it only reflects the mean clinical treatment performance for an average patient and uncertainty in neither the criteria weights nor performance scores is evaluated.

There are, however, several solutions to include uncertainty in the MCDA model. For instance, Wen et al. [18] and Chim et al. [19] use probability distributions for clinical evidence and point estimates for preferences. Kaltoft et al. [20] estimate preference uncertainty by defining preference subgroups and coupling these to point estimates for clinical performance. Lynd et al. [21] used point estimates from a discrete choice study to assign utilities to events in a (probabilistic) discrete event simulation. In two other methods, probability distributions for clinical evidence is combined with uniform distributions for criterion weights as a non-informative prior [22, 23]. One common problem of all these methods is that they do not simultaneously combine parameter uncertainty and random variation in both patient preferences and clinical evidence. Here, parameter uncertainty is uncertainty around an estimated quantity (such as a group mean) which can be reduced with more measurements [24, 25]. Random preference variation is the systematic variation of preferences across the population. This variation can, by definition, not be reduced by repeated measurements but can only be better characterized [24, 25]. The importance of including preference variation is recently recognized in the development of personalized medicine based on (variation in) preferences [26, 27].

The aim of this study was to demonstrate an application of probabilistic MCDA that would allow a joint analysis of patient preferences and clinical evidence for treatments, taking into account uncertainty in both preferences and clinical evidence. The proposed model can be applied during the aggregation step of a value-based MCDA and is able to handle three sources of uncertainty: random preference variation, parameter uncertainty in preferences, and parameter uncertainty in clinical evidence. The model is designed to yield value distributions that enable explicit probabilistic statements about which treatments are preferred. The model is illustrated using a simplified case study of highly active antiretroviral therapies (HAART) for HIV patients.

A Proposed Multi-Criteria Decision Analysis (MCDA) Method for Capturing the Value of Treatments Under Uncertainty

Building the Value Function and Defining Sources of Uncertainty

We adopted a value-based MCDA method. The value of a particular treatment , denoted , was assumed to be a linear additive function of criteria (Eq. 1):where denotes the preference weight of criterion , and is the performance of treatment on criterion . We assumed that patients prefer treatments with a higher value to treatments with a lower value, that preference weights and the clinical performances were not correlated, and that the clinical performances were measured on an interval scale.

To introduce random preference variation, suppose each patient has his/her own preference weight for each criterion (Eq. 2):

The term in this equation is composed of population mean preference weight and a patient-specific random effect (Eq. 3):with (Eq. 4):

Parameter Estimators

There are different methods for obtaining preference weights, such as swing weighting. However, in this study the required parameter estimators and were obtained by analyzing patient-level data from stated preference studies. Parameter uncertainty in these estimators across all criteria is reflected in the covariance matrices and . The estimator for clinical performance is denoted and can be obtained from clinical trial reports.

Sampling Framework

We assumed that the vectors and were both distributed according to multivariate normal distributions. We denoted the value distribution of treatment across the population with , assuming that the individual values were identically distributed for patients . In a probabilistic model all uncertain parameters are varied at the same time, and this implies that the probability distribution is a complex and analytically challenging combination of the distributions for , and . We therefore approximated with Monte-Carlo simulations (Fig. 1). The Monte-Carlo simulations were programmed as follows. In each simulation run , we first sampled a population mean preference weight from and a standard deviation from . From that we obtained a respondent using Eqs. 3 and 4 (Eq. 5):with a diagonal covariance matrix with on the diagonal. This means that in every simulation run a hypothetical patient with a particular vector of preferences was obtained. The probability distribution of was denoted with and chosen for each criterion depending on what best modeling practices recommend for that type of clinical performance [24]. From we sampled in each simulation run. The preference sample was combined with samples from the probability distributions for each criterion to calculate the value for each treatment in the simulation run with (Eq. 6):

Fig. 1

Overview of the Monte-Carlo simulation method used in the model. the treatment, and the Monte Carlo simulation run

The Monte-Carlo simulation process was repeated a large number of times and was programmed in R [28].

Model Outcomes

From the Monte-Carlo simulations the mean value of each treatment in the population was obtained, calculated as the posterior mean, . This metric can be interpreted as the mean perceived value of the treatment’s clinical performance according to patients. Since we assumed that patients prefer treatments with a higher value to treatments with a lower value, the treatment value should only be interpreted relative to the value of other treatments. The degree to which the value distributions of two treatments overlap indicates how uncertain we are about selecting the treatment with the highest value. More concretely, to make probabilistic statements about the degree to which we are sure about which treatment has the highest value in each simulation, treatments were ranked from the first (highest value) rank to the last (lowest value) rank based on their respective . The rank achieved by treatment in simulation run was denoted with . We denoted the rank of treatment with . The probability that this rank equals was estimated with the percentage of simulations treatment had rank (Eqs. 7 and 8):where

Since we assumed patients prefer treatments with a higher value to treatments with a lower value, the treatment with the highest estimated probability of being ranked first was considered the most preferred treatment. One minus the probability that the most preferred treatment is ranked first is the rank reversal probability for the first rank. The rank reversal probability for the first rank is used as a measure of decision uncertainty, because it is the probability that the most preferred treatment (according to our model) turned out not to be the most valuable treatment.

An Illustration of the Proposed Approach on a Simplified HIV Case

To illustrate the proposed model, it is applied to the case of HIV treatments. The MCDA was designed according to the guidelines as proposed by the ISPOR taskforce [13]. Although the data used comes from real studies, the main purpose of this paper is to illustrate the modeling approach. The simulation results presented in this paper can therefore not be used to inform clinical HIV decisions. All data used (with references) as well as the R script can be found in the Electronic Supplementary Material to replicate our results.

Identification of Treatment Alternatives and Decision Criteria

The example is about the comparison of the relative value of HAARTs for HIV-positive patients from a regulatory perspective. Treatments under consideration are those recommended by the US National Institutes of Health (NIH) for treatment-naïve patients [29]. A HAART consists of an active drug component and one of two backbones (abacavir/lamivudine [AL] or tenofovir/emtricitabine [TE]). The treatments included in the case study were four combination treatments (dolutegravir + AL/TE,1 efavirenz + AL, raltegravir + TE, atazanavir/ritonavir + TE, elvitegravir/cobicistat + TE, and darunavir/ritonavir + TE) plus two backbone-only treatments (AL and TE). For simplicity we denote combination treatments only by their active drug component (e.g., dolutegravir instead of dolutegravir + AL/TE, etc.) in the remainder of this paper.

In the MCDA model we used the same criteria as identified in an earlier preference study [30], namely probability of virologic failure (), probability of allergic reaction (), probability of bone damage (), and probability of kidney damage () [30]. All probabilities were all defined over a 52-week time horizon.

Measuring Performance of the Alternatives

The clinical performance of the treatments on the included criteria was obtained from clinical trial reports (for an overview of the included trials see the NIH guideline [29]). All criteria were represented as probabilities. For each treatment we therefore retrieved from the clinical trial reports the number of events and number of non-events ( per criterion . The performance estimates for each criterion were then calculated with . For studies that reported time horizons other than 52 weeks, we used linear extrapolation to obtain performance estimates for a 52-week time horizon. We used a linear partial value function for each criterion. No rescaling of the performances was needed for the partial value functions since probabilities are already naturally constrained between 0 and 1. For criteria measured on other scales, such a rescaling to the range using a predefined lower and upper level would be required [31].

Plasma HIV RNA of more than 50 copies/mL 52 weeks after treatment start was considered a virologic failure event. Reported incidence of rash was used as a surrogate measure for the TE-induced allergic reaction event. Reported fractures in the clinical trials were considered to constitute bone damage events. However, this was not reported for TE; therefore, decreases in bone mineral density of more than 6% were used as a surrogate endpoint. Because of the diagnostic and treatment options available, we limited our model to include only treatable bone damage [29, 32]. Reported cases of renal failure were considered to be kidney damage events, and we therefore limited our model to include only non-treatable kidney damage [33].

Input for Criteria Weights: Results from a Previously Published Discrete Choice Experiment

The present methodological study did not elicit criteria weights itself, but used the results from a previous stated preference study to inform the preference weights. In that study by Hauber et al. [30], 147 treatment-naïve HIV-positive African Americans gave their preferences for four criteria relevant for HIV treatment. The study employed a discrete choice study design with 24 choice tasks. In the choice tasks the criteria were defined as the probability of the event in the next 52 weeks. An overview of the preference data is given in Table 1.

Table 1

Preference data used from Hauber et al. [30]. All are per percentage point probability of the event occurring in the next 52 weeks, i.e., the partial value of a 2% probability of allergic reaction in the next 52 weeks is −0.12. Note that the covariance and are not presented here for brevity but can be found in the Electronic Supplementary Material. Both and are assumed to be distributed with a multivariate normal distribution

	Levels used in DCE (%) [21]	β^	SE(β^)	σ^_k	SEσ^_k
Virological failure prevented	96.0, 85.0, 79.0	−0.05	0.01	0.05	0.02
Allergic reaction	0.0, 1.0, 8.0, 12.0	−0.06	0.01	0.06	0.02
Bone damage (treatable)	0.0, 1.0, 5.0, 10.0	−0.01	0.02	0.06	0.06
Kidney damage (not treatable)	0.0, 1.0, 5.0, 10.0	−0.22	0.04	0.21	0.05

DCE discrete choice experiment

Aggregating Scores and Performance

In the base-case analysis we assumed an additive value function. The value of treatment for patient (Eq. 2) could therefore be specified as Eq. 10:

The mean population preferences and the random preference variations were assumed to be normally distributed. The parameter estimates for these distributions were obtained from a (mixed logit [34]) analysis of the used preference study (Table 1). The performances of each treatment are probabilities and were therefore modeled with beta distributions [24]. Beta distributions require two input parameters ( and ) and these were estimated from the clinical trials cited in the NIH guideline. A complete overview of the clinical trials used in this study can be found in the Electronic Supplementary Material. We used the number of reported events as and the number of non-events as . In total, 100,000 Monte-Carlo simulations were performed.

Handling Uncertainty: Scenario Analyses

A scenario analysis was employed to illustrate the effect of the different sources of uncertainty on the treatment value distributions. The base-case scenario described previously considered all types of uncertainty simultaneously, i.e., parameter uncertainty in preferences, random preference variation, and parameter uncertainty in clinical evidence. Each of the other scenarios tested for one specific source of uncertainty. This means that each scenario included one source of uncertainty while the other two sources were assumed to have no uncertainty and were thus fixed at a particular value. In the first scenario, only parameter uncertainty in preferences (as parameterized by the covariance matrix ) was taken into account. Therefore, was set to zero for all criteria and individuals, and all clinical performances were set to the mean clinical performance of treatments as found in Table 2. In the second scenario, only random preference variation was included. This means that the was set to be a zero matrix, all clinical performances were set to the mean clinical performance of treatments as found in Table 2, and was distributed as in the base case. In the third scenario, only parameter uncertainty in clinical performance was included. This means that was set to zero for all criteria and individuals, was set to be a zero matrix, and performances were distributed with as in the base case.

Table 2

Clinical evidence used. For references to all included clinical trials, see the National Institutes of Health guideline [29] and the Electronic Supplementary Material. All performances were defined over a 52-week time horizon and assumed to be distributed with beta distributions

HAART regimen	Probability of virological failure (95% CI)	Probability of allergic reaction (95% CI)	Probability of bone damage (95% CI)	Probability of kidney damage (95% CI)
Dolutegravir + TE/AL	7.61% (6.10–9.25)	0.24% (0.01–0.90)		0.09% (0.00–0.34)
Atazanavir/ritonavir + TE	5.25% (3.94–6.77)	1.72% (0.74–3.10)	1.29% (0.48–2.5%)	0.43% (0.05–1.21)
Elvitegravir/cobicistat + TE	13.52% (10.17–17.34)		2.02% (0.83–3.77%)	0.87% (0.18–2.12)
Efavirenz + AL	13.67% (12.32–15.13)	1.86% (1.21–2.66)	2.12% (1.41–2.98%)	0.29% (0.11–0.56)
AL (backbone)	16.3% (13.74–18.99)	2.32% (1.01–4.18)	4.57% (3.02–6.38%)	2.50% (1.47–3.75)
TE (backbone)	20.41% (17.58–23.39)	1.01% (0.28–2.19)	2.19% (1.18–3.52%)	3.84% (2.56–5.36)

AL abacavir/lamivudine, CI confidence interval, HAART highly active antiretroviral therapy, TE tenofovir/emtricitabine

Examination and Discussion of Findings

Outcomes of Case Study

The model outcome was a value distribution for each included HAART (Fig. 2). In the base case, dolutegravir has the highest patient-weighted estimated mean value of −0.39 with an empirical 95% confidence interval (CI) running from −1.25 to 0.48. The backbone-only treatments AL and TE had the lowest mean values (−1.49 and −1.86, respectively). In all scenario analyses, mean treatment values were similar to the base-case results (Fig. 2; Table 3) and the most likely rank order of treatments did not change. The width of the CIs did vary between the base case and the scenarios. When only random preference variation was considered in the second scenario, the CIs were only slightly narrower than those of the base case. In the two other scenarios that considered parameter uncertainty the CIs were substantially narrower.

Fig. 2

Barplots of the regimen values with 95% confidence intervals across the four analysis scenarios. Purple dolutegravir, dark blue elvitegravir/cobicistat, light blue atazanavir/ritonavir, green efavirenz, yellow abacavir/lamivudine, red tenofovir/emtricitabine

Table 3

Values (with 95% confidence intervals) for the included highly active antiretroviral therapy regimens across the four analyses

HAART regimen	Base case	Scenario analyses
	All three types of uncertainty	Only parameter uncertainty in preferences	Only patient-specific preference variation	Only parameter uncertainty in performances
Dolutegravir + TE/AL	−0.39 (−1.25 to 0.48)	−0.38 (−0.57 to −0.19)	−0.39 (−1.24 to 0.47)	−0.38 (−0.48 to −0.30)
Atazanavir/ritonavir + TE	−0.46 (−1.2 to 0.24)	−0.45 (−0.62 to −0.28)	−0.45 (−1.14 to 0.23)	−0.45 (−0.64 to −0.32)
Elvitegravir/cobicistat + TE	−0.83 (−2.52 to 0.8)	−0.82 (−1.22 to −0.42)	−0.83 (−2.41 to 0.74)	−0.83 (−1.13 to −0.59)
Efavirenz + AL	−0.83 (−2.4 to 0.77)	−0.82 (−1.19 to −0.44)	−0.83 (−2.41 to 0.76)	−0.82 (−0.92 to −0.73)
AL backbone	−1.49 (−3.77 to 0.83)	−1.47 (−2.06 to −0.89)	−1.48 (−3.7 to 0.75)	−1.48 (−1.79 to −1.20)
TE backbone	−1.86 (−4.68 to 1.07)	−1.85 (−2.54 to −1.17)	−1.85 (−4.64 to 0.94)	−1.86 (−2.21 to −1.55)

AL abacavir/lamivudine, HAART highly active antiretroviral therapy, TE tenofovir/emtricitabine

In 49.1% of the simulations, dolutegravir was ranked first (Table 4). This implies that in 50.9% of simulations, another treatment was preferred. Atazanavir/ritonavir had the highest probability of being ranked second (40.0%) and efavirenz had the highest probability of being ranked third (40.3%). The narrower value distributions in the scenarios are reflected in the rank probabilities. The rank probabilities in the scenario that considered only parameter uncertainty in preferences were higher: the probability of dolutegravir attaining first rank and atazanavir/ritonavir attaining second rank were both more than 90%. This means that we were confident about the ranking of treatments for the first and second rank. Similarly, the first and second rank probabilities for dolutegravir and atazanavir/ritonavir are more than 75% in the scenario that considers only parameter uncertainty in clinical performances. From the fourth until last rank, there is slightly more decision uncertainty in the scenario that considers parameter uncertainty in preferences than there is in the scenario that considers parameter uncertainty in clinical performances.

Table 4

Ranking probabilities for all included regimens across the four analyses

	Dolutegravir + TE/AL (%)	Atazanavir/ritonavir + TE (%)	Elvitegravir/cobicistat + TE (%)	Efavirenz + AL (%)	AL backbone (%)	TE backbone (%)
Base case
All three types of uncertainty
1	49.07	34.21	4.95	4.28	1.92	5.57
2	36.12	39.98	10.53	7.54	4.08	1.75
3	5.56	8.69	40.25	40.31	3.27	1.92
4	3.46	8.47	39.96	39.00	5.69	3.42
5	4.60	3.52	3.76	4.96	69.52	13.64
6	1.19	5.13	0.55	3.91	15.52	73.70
Scenario analyses
1: Only parameter uncertainty in preferences
1	90.42	9.58	0.00	0.00	0.00	0.00
2	9.58	90.26	0.14	0.02	0.00	0.00
3	0.00	0.11	46.26	53.63	0.00	0.00
4	0.00	0.05	53.60	46.35	0.00	0.00
5	0.00	0.00	0.00	0.00	100.00	0.00
6	0.00	0.00	0.00	0.00	0.00	100.00
2: Only patient-specific preference variation
1	53.28	32.25	2.34	5.32	1.59	5.22
2	33.77	46.18	8.50	6.43	3.93	1.19
3	4.68	5.81	42.71	42.69	2.07	2.04
4	2.76	8.52	44.4	37.84	3.69	2.79
5	4.41	2.46	1.90	3.49	79.51	8.23
6	1.10	4.78	0.15	4.23	9.21	80.53
3: Only parameter uncertainty in performances
1	77.2	22.79	0.01	0.00	0.00	0.00
2	22.8	76.41	0.68	0.11	0.00	0.00
3	0.00	0.74	51.00	48.26	0.00	0.00
4	0.00	0.06	48.19	51.63	0.12	0.00
5	0.00	0.00	0.12	0.00	95.2	4.68
6	0.00	0.00	0.00	0.00	4.68	95.32

AL abacavir/lamivudine, TE tenofovir/emtricitabine, TE/AL either tenofovir/emtricitabine or abacavir/lamivudine

Implications of the Modeling Framework for Personalized Medicine

The main objective of this study was to develop a novel methodology. The HIV case was used to demonstrate the concepts and cannot be used for guidance on HIV treatment decisions. However, the modelling framework does allow an exploration of its usefulness for other applications: assessing the impact of uncertain parameters (clinical evidence and preferences) on decision uncertainty and personalizing treatment based on preferences. The assessment of the impact of the uncertainty in model parameters on decision uncertainty is important for identifying the value of additional research: it is most worthwhile to further investigate model parameters where more information would most likely reduce decision uncertainty. In the presented case, there is a clear difference between the impact of the sources of uncertainty. When only parameter uncertainty in either preferences or performances was considered, one treatment was clearly the most valuable treatment. However, when random preference variation was considered, there was considerable overall decision uncertainty. The value of additional clinical research may therefore not be high but decision aids (that help patients think about their preferences, reducing their individual preference uncertainty) may be valuable.

This relates to the second application for which our work may be used: personalized medicine. Although most research in that field has focused on personalizing treatment based on clinically measurable patient characteristics such as genetic differences, there is an increasing interest in personalizing treatment based on patient preferences [26]. Our model could help decision makers to make the first steps toward such a personalization by showing to what extent (uncertainty in) patient preferences influences the choice of treatment. A next step to formalize the assessment of the extent to which differences in preferences are relevant would be to calculate metrics such as the value of heterogeneity [35]. This metric shows the marginal population-wide value gained from having patients choose between more than one treatment. This value will be low if there is a clear most valuable treatment for all patients, but it will be high if there is clinical equipoise and/or much patient-specific preference variation. An important outstanding research issue here is how values derived from an MCDA can be contrasted with financial costs [36]. Further research could also be directed toward the integration of patient-specific clinical outcome measures; that is, introducing a patient-specific performance distribution that yields estimates of the performance of a specific treatment for a specific patient. For the HIV case this could, for example, be operationalized by estimating the treatability of kidney damage based on respondent characteristics such as level of kidney functioning at treatment start. Such a holistic view of patient-specific variation in both preferences and clinical outcomes would be a step toward combining the two current viewpoints on personalized medicine [26].

Strengths, Weaknesses, and a Comparison to the Existing Literature

The first strength of the current study is that it has demonstrated a methodological approach for combining preference data and clinical data into one value metric, which may contribute to the ongoing attempts to integrate patient preference research in health technology assessment and market approval. A second strength of the study is that the developed model allows for the simultaneous consideration of the impact of three sources of uncertainty, i.e., random preference variation and parameter uncertainty in both clinical and preference estimates.

The developed model uses results from stated preference studies to inform value judgements in policy decisions from one of the most important stakeholders in healthcare: the patient. By including stated preference studies in the model, the patient preferences are incorporated in an explicit, structured and representative manner. Many different types of preference elicitation methods exist. In this study we used results from a discrete choice study, but in theory our model is able to handle preference weights obtained from a wide range of preference elicitation methods as long as a value function can be constructed and probability distributions can be assigned to weights. Finally, the possibility of including preference studies with large sample sizes allows for the investigation of variation and uncertainty in preferences. The impact of these was investigated by assigning informative probability distributions, which sets our study apart from earlier studies that have used point estimates [18-21] or non-informative (uniform) distributions [22, 23].

The decision to use a probabilistic approach in the present study follows from a recent review identifying five approaches to deal with uncertainty in MCDA [25]. The approach adopted in this study seems most advantageous for our aims for a number of reasons. It is the approach that is best suited for dealing with the preferences of a group of stakeholders and is most able to consider multiple uncertain parameters [25]. Another advantage is that it is possible to implement Monte-Carlo simulations as a flexible method that can combine all types of parametric and non-parametric probability distributions [24]. For these reasons, decision makers could apply the method during the aggregation and uncertainty steps of the MCDA process described in the recent ISPOR taskforce report [13]. This would be especially advantageous in policy decisions where the patient perspective is considered explicitly and where various sources of uncertainty are relevant to consider.

Even though the model may be appropriate given our aims, a limitation of all studies that use results from stated preference research is that a person’s stated preference may not be the same as their revealed preference [37]. A limitation specific to the illustrative case study was that treatment value is assumed to be linearly related to the clinical performance. This assumption implies that we assumed we could extrapolate linearly beyond the performance levels originally included in the preference study to calculate partial values. The linearity specification could not be rejected in the preference study but this could also have been due to the sample size [30]. A related limitation is that we assumed that we could extrapolate performance measures to conform to the 52-week time horizon used in the preference study. Furthermore, another limitation is that random preference variation was assumed to be normally distributed. Although this is practical and a commonly made assumption in patient preference research [38], in our study it resulted in a small percentage of Monte-Carlo simulations having sign reversals for the preference weights, mainly for the virologic failure criterion. In a larger patient sample it may have been possible to estimate a more specific functional form for the value function [39]. Performance samples for virologic failure all fell outside of the range for which preferences were elicited, which may have biased the value estimates.

Conclusion

In an attempt to explore new approaches for increasing patient engagement in healthcare policy decisions, the current paper presents a probabilistic MCDA model in which treatment values were estimated by weighting clinical trial evidence with results from a patient preference study. The model outcomes were patient-weighted probability distributions of relative treatment value and the respective rank probabilities. The developed model was illustrated using a simplified case study. The adopted probabilistic approach integrates random preference variation and parameter uncertainty in patient preferences with parameter uncertainty in clinical evidence using a Monte-Carlo simulation method. Further research about the use of the modelling approach for non-simplified cases and the match to decision maker needs is required.

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 3067 kb)

Supplementary material 2 (TXT 4 kb)

Supplementary material 3 (DOCX 27 kb)

Supplementary material 4 (TXT 0 kb)

Healthcare decisions require an assessment of the value treatments provide for patients. Such assessments are made under uncertainty and there is no consensus about how to account for patient preferences in making these assessments.

This study applies a multi-criteria decision analysis model where clinical evidence is weighted with patient preferences. In this way, patient-weighted treatment values can be estimated in a representative manner while building on the existing the clinical evidence.

The probabilistic approach adopted in the model allows for the simultaneous modelling of measurement uncertainty and patient-specific preference variation. Scenario analyses show that the impact of these different types of uncertainty on decision uncertainty is substantially different in a simplified case on HIV treatments.