A New Approach for Sampling Ordered Parameters in Probabilistic Sensitivity Analysis.

BACKGROUND: Probabilistic sensitivity analysis (PSA) in cost-effectiveness analysis involves sampling a large number of realisations of an economic model. For some parameters, we may be uncertain around the true mean values of the variables, but the ordering of the values is known. Typical sampling approaches lack either statistical or clinical validity. For example, sampling using a common number generator results in extreme dependence, and independent sampling can lead to realisations with incorrect ordering.
METHODS: We propose a new sampling approach for ordered parameters, the difference method (DM) approach, which samples the parameters of interest via a difference parameter. If the parameters of interest are bounded, it involves transforming the variables so that they are unbounded and then sampling via the difference parameter. We have provided a Microsoft Excel workbook to implement the method. The proposed approach is illustrated with an example sampling ordered parameters for utility and cost.
RESULTS: The DM approach has a number of advantages when comparing with the typical approaches used in practice. It generates PSA samples that have similar summary statistics as the given values in our examples, while maintaining the constraint that one value was greater than another. The method also implies plausible positive correlation between the two ordered variables.
CONCLUSIONS: Both clinical and statistical validity should be checked when producing PSA samples. The DM approach should be considered as a solution to potential problems in generating PSA samples for ordered parameters.

Key Points for Decision Makers

Introduction

In health technology assessment, probabilistic sensitivity analysis (PSA) represents the generally accepted approach for characterising the uncertainty in parameters included in an economic model and for producing accurate results in non-linear models [1, 2]. This involves generating a large number of realisations of the economic model, each time sampling values from the distributions applied to each uncertain parameter included in the model.

Sometimes there is an absolute belief that the value of one variable is greater than the value of another. There may be uncertainty around the true mean values of the variables, but the ordering of the values is known. For example, if an individual rates his/her general health as ‘good’, then later as ‘fair’, we might be uncertain about how to map the ‘good’ and ‘fair’ health evaluations onto a numeric scale, but can assume the ‘good’ general health score will be higher than the ‘fair’ general health score. Health-related quality of life (HRQoL) for two different severity levels of a disease may also be related in this manner.

Failure to account for constraints between samples of ordered parameters may result in PSA values that do not accurately characterise the uncertainty present in a decision problem. In theory, this could result in decisions made on the allocation of scarce healthcare resources being suboptimal, although the direction of bias would depend on the specific model. Other outputs and analyses that are reliant on the PSA, such as cost-effectiveness acceptability curves (CEACs) and frontiers (CEAFs), and value of information analyses, are likely to also be inaccurate if the constraint that the value of one variable is greater than another is not accounted for appropriately.

Based on the authors’ experience, as National Institute for Health and Care Excellence (NICE) committee members, Evidence Review Group members dealing with a large number of NICE submissions every year, and peer reviewers, we have identified multiple examples where sampling of ordered parameters has not been handled correctly. Independent sampling from the distributions of the ordered parameters could result in a PSA that lacks clinical validity as in some realisations the logical constraint may be violated, with the sampled value of parameters potentially implying that having a disease makes people healthier. We refer to this method as the independent sampling approach. For example, in an NICE Single Technology Appraisal [3] the use of the independent sampling approach results in the utility in the progressed state being higher than in the stable state. Another sampling method used in practice used a common random number to draw from both distributions [4], which induces extreme dependence: one variable is a deterministic function of another. In addition to the extreme dependence induced by this method, the sampled realisations of PSA could still violate the known ordering if the cumulative density functions of the two parameters cross. We refer to this method as the common random number generator approach. A flawed alternative observed in papers sent to the authors for peer review is a method whereby samples are excluded when the ordering assumption is violated. We refer to this method as the modified independent sampling approach. This approach results in the summary statistics of the sampled realisations not equalling that of the source data, and provides biased sampled realisations.

Typical approaches described above lack either statistical or clinical validity. To have clinical validity when sampling parameters where it is known that one value is greater than another, all PSA realisations should exhibit the logical order of the given parameters. To have statistical validity, we have deemed that the following criteria should be met: (1) if the parameters of interest are bounded then all PSA realisations should be in these bounded ranges; (2) the summary statistics of the sampled values of the parameters of interest from PSA should match closely to the given summary statistics of the parameters of interest; (3) the induced correlation between the sampled values of parameters of interest should be plausible.

Introducing an additional parameter that presents either an absolute difference or a ratio of the ordered parameters is a common approach for modelling the ordered parameters explicitly. In Sect. 2, we apply this modelling idea in generating samples for PSA, and propose a new approach for sampling ordered parameters where summary statistics have been provided for the parameters of interest. We call this method the difference method (DM) approach. This approach is illustrated with an example sampling ordered parameters for utility and cost in Sect. 3, while Sect. 4 provides the results of comparing the performance of the DM approach with the typical approaches used in practice. Discussion is given in Sect. 5.

Methods

Unbounded Ordered Parameters

Suppose that there are two variables and , where the value of is greater than the value of , the distribution of has mean and variance , and the distribution of  has mean and variance . Let denote the difference between and , and and denote the mean and variance of the distribution of the difference , respectively. The DM approach samples from either and or and , depending on the magnitude of the variances of and . If , then definebut if , then defineAssuming and in Eq. 1 are independent, and and in Eq. 2 are independent, we get and from Eq. 1, and and from Eq. 2. Rearrange these equations, and we have and .

If both variable and are unbounded, we propose using a Gamma(,) distribution for so that the difference is always positive, where is the shape parameter and is the rate parameter. Hence, and . These can be solved simultaneously to give

Other distributions with two model parameters for a non-negative random variable such as log normal could also be used, and will produce sampled realisations with both clinical and statistical validity, but the induced correlations might be different.

The sampling procedure depends on the magnitude of the variances of and .

When , it involves sampling from Normal (,) and from Gamma (,). Sampled values of are derived from sampled values of and using Eq. 1.

When , it involves sampling from Normal(,) and from a Gamma(,). Sampled values of are derived from sampled values of and using Eq. 2.

The normal distribution is chosen because both and are unbounded and can take any values on the real line. In addition, the input parameter of interest in an economic model is typically the mean of a random variable, which is approximately normally distributed because of the central limit theorem. We have proposed to use a simulation method to derive sampled values using either Eqs. 1 or 2; however, an analytic solution might exist.

Bounded Ordered Parameters

If both and are bounded between 0 and 1, such as probabilities and most HRQoLs, or bounded to be positive, such as cost, we suggest a four-step sampling procedure.

Step 1: Given mean and variance of and , sample and from Beta distributions if the , and sample from Gamma distributions if . The Gamma parameters can be derived using Eqs. 3 and 4. To derive the model parameter for a Beta distribution given the mean and variance, let us assume is from a Beta() distribution. Hence

These two equations can be solved simultaneously to give

If Y is from a , the Beta model parameters can be derived using Eqs. 5 and 6 by replacing and with and .

Step 2: Transform the sampled and to the real line, using and if the ; and using and if the ). Calculate the mean and variance for sampled and , where and are the logit transformed and if , and the log transformed and if .

Step 3: Use the DM approach for unbounded variables described in Sect. 2.1 to redefine to be the difference between and . First, sample from a Gamma distribution, where the model parameters can be derived using Eqs. 3 and 4. Then either using sampled values for from step 2 and sampled values for to derive sampled values for , or using sampled values for from step 2 and sampled values for to derive sampled values for , depending on the magnitude of the variance of and .

Step 4: Back transform sampled values for and to obtain sampled values for and . If , then use and . If ), then use and .

The logit or log transformation for the bounded and is to make sure that the sampled realisations will be in the appropriate bounded range. Depending on the given summary statistics, some samples may fall outside of the bounded ranges without the transformation.

Example

In this section, we illustrate how the proposed DM approach can be implemented in a hypothetical example. Suppose that there is an active (worse) and remission (better) state in an economic model for a particular condition. The input parameters are mean HRQoL and mean cost. In health technology assessments, it is common that an analyst does not have access to the individual patient-level data (IPD), but only summary statistics derived from the IPD. For ordered parameters, often only the summary statistics such as mean and standard deviation/variance for each of the parameters are provided, but not the correlation structure. Suppose that the analyst is given the following information regarding the input parameters.

Active (worse) state: the input parameter for HRQoL () has mean and standard error 0.138 (variance ); the input parameter for cost () has mean and standard error 3.872 (variance ).

Remission (better) state: the input parameter for HRQoL () has mean and standard error 0.126 (variance ); the input parameter for cost () has mean and standard error 3.162 (variance ).

Suppose the analyst requires 5000 samples of realisations for each parameter in each state. It was known that the HRQoL in the remission state is higher than the active state, , and HRQoL is bounded between 0 and 1, . It was also known that the cost in the remission state is lower than in the active state, , and cost is positive, . The four-step sampling procedure for sampling mean HRQoL and mean cost is given in Table 1.

Table 1

Four-step sampling procedure for generating mean health-related quality of life and mean cost

Sampling procedure	Sampling mean HRQoL	Sampling mean cost
Step 1: Sample Y and X	Sample 5000 values of Y_u from Beta(6.52,5.55) and 5000 values of X_u from Beta(8.49,3.64), where the Beta distribution parameters were calculated using Eqs. (5) and (6).	Sample 5000 values of Y_c from Gamma(1000,0.100) and 5000 values of X_c from Gamma(806.67,0.136), where the Gamma distribution parameters were calculated using Eqs. (3) and (4).
Step 2: Transform sampled Y and X from step 1 to unbounded range	Transform sampled Y_u and X_u from step 1 to unbounded range using logit function: Yu′=logit(Y_u) and Xu′=logit(X_u).The mean and variance for the transformed variables are μY_u′=0.928, σY_u′2=0.436, μX_u′=0.186, and σX_u′2=0.364.	Transform sampled Y_c and X_c from step 1 to unbounded range using log function: Yc′=log(Y_c) and Xc′=log(X_c).The mean and variance for the transformed variables are μY_c′=4.60, σY_c′2=0.00101, μX_c′=4.70, and σX_c′2=0.00124.
Step 3: Sample the difference between transformed Y and X	Since σY_u′2>σX_u′2, we define Yu′=Xu′+Δu′, where the mean and variance of Δu′ is 0.742 and 0.072, respectively.Sample Δu′ from Gamma(7.64,0.10), where the Gamma distribution parameters were calculated using Eqs. (3) and (4). Compute sampled values of Yu′ by adding sampled Δu′ and sampled Xu′ from step 2.	Since σY_c′2<σX_c′2, we define Xc′=Yc′+Δc′, where the mean and variance of Δc′ is 0.10 and 0.0002, respectively.Sample Δc′ from Gamma(41.07,0.002), where the Gamma distribution parameters were calculated using Eqs. (3) and (4). Compute sampled values of Xc′ by subtracting sampled Δc′ from sampled Yc′ from step 2.
Step 4: Back transform	Back transform sampled Yu′ and Xu′ to Y_u and X_u using Y_u=e^Yu′1+e^Yu′ and X_u=e^Xu′1+e^Xu′.	Back transform sampled Xc′ and Yc′ to X_c and Y_c using X_c=e^Xc′ and Y_c=e^Yc′.

HRQoL health-related quality of life

For comparison, we also performed the independent sampling approach, the modified independent sampling approach, and the common random number generator approach to sample mean HRQoL and mean cost in the active and remission states. For the mean HRQoL parameters, the independent sampling approach generates and by sampling from and from independently. The modified independent sampling approach samples independent realisations of and first and then excludes the draws where the ordering assumption is violated until it reaches the required number of realisations. The common random number generator approach samples and from the same Beta distributions as the other two approaches, but the same random number generator was used for both parameters. For the mean cost parameters, the distributions used in these three standard sampling approaches were and for and , respectively.

We also developed a Microsoft Excel workbook (Microsoft Corporation, Redmond, WA, USA) to implement the DM approach, which is included as online supplementary material to this paper (Online Resource 1). The Microsoft Excel workbook used the example presented in Sect. 3 for an illustration.

Results

Table 2 shows that the mean and variance of 5000 sampled realisations using four sampling approaches (the DM, independent sampling, modified independent sampling, and common random number generator approaches). The generated mean and variance of the sampled realisations using all the approaches closely matched the given summary statistics, with the exception of the modified independent sampling approach, which produces biased sampled realisations and should be avoided in practice for handling the logical problem in PSA. This method will always underestimate the uncertainty in the ordered parameters, overestimate the mean of the parameter with a higher value, and underestimate the mean of the parameter with a lower value. The magnitude of the bias depends on the percentages of the sampled realisations that violated the constraint. In the example, 20% of the sampled values in 5000 samples did not meet the logical order for the HRQoL parameter, whereas 2% of the samples violated the order for the cost parameter. The generated summary statistics were closer to the given values for mean cost than mean HRQoL.

Table 2

Mean and variance of 5000 sampled realisations using four sampling approaches

Variable	Health state	Given mean (variance)	Generated mean (variance)
			Difference method	Independent sampling	Modified independent sampling	Common random number
HRQoL	Remission	0.70 (0.016)	0.70 (0.016)	0.70 (0.016)	0.73 (0.012)	0.70 (0.016)
	Active	0.54 (0.019)	0.54 (0.019)	0.54 (0.019)	0.51 (0.016)	0.54 (0.020)
Cost	Remission	100 (10)	100.02 (9.94)	100.02 (9.94)	99.9 (9.54)	100.02 (9.95)
	Active	110 (15)	109.98 (15.29)	109.98 (15.10)	110.15 (13.87)	110.03 (14.93)

HRQoL health-related quality of life

The scatterplots of sampled pairs of values using the three methods that produce unbiased sampled realisations are provided in Fig. 1. These scatterplots show that the DM approach guarantees to maintain the constraint that one value is greater than another for each sampled pair, and induced a positive correlation between the ordered variables. The independent sampling approach is likely to lack both clinical and statistical validity since (1) it may produce a sampled value of mean HRQoL in the worse state that is higher than the sampled value of mean HRQoL in the better state (20% of the samples violated the order), and a sampled value of mean cost in the better state that is higher than the sampled value of mean cost in the worse state (2% of the samples violated the order); and (2) there are no correlations between the sampled realisations in the two states. The common random number generator approach is likely to lack statistical validity regarding the correlations since this method implies that given the value of one variable, the value of the other is fixed/determined, i.e. two variables are perfectly correlated.

Fig. 1

Scatterplots of pairs of samples generated by three methods: a for HRQoL, where is HRQoL in the disease-active state and is HRQoL in the disease-remission state; b for cost, where is cost in the disease-remission state and is cost in the disease-active state. HRQoL health-related quality of life, DM difference method, Ind independent sampling method, RN common random number sampling method

Using different sampling approaches in generating samples for ordered parameters will have an impact on the uncertainty of the incremental cost-effectiveness ratio (ICER), and hence also the CEAC/CEAF; however, we believe that this would rarely affect conclusions of policy. Hence, we did not compare the methods based on the change in uncertainty in ICER or CEAC/CEAF. Nevertheless, we should not stop improving the rigour of a method just because it may not affect the conclusions.

Discussion

Sampling from ordered variables is a common task that an analyst faces in conducting a PSA in cost-effectiveness analysis (CEA). Given the summary statistics for the ordered parameters, the only way to handle the logical problem is by using an appropriate sampling approach in the PSA. We have illustrated that typical approaches used in practice, such as independent sampling, modified independent sampling, and using a common random number generator, lack either statistical or clinical validity. This problem may only rarely affect conclusions or decisions made based on economic evaluation results, but we believe there is a need for a new sampling method to handle the logical problem to improve the rigour of PSAs in CEA.

The proposed DM approach has been shown to be effective in generating bivariate variables that satisfy both clinical and statistical validity in the given examples. It provides a solution to an issue that in theory may have important implications for the interpretation of economic evaluations of health technologies. An earlier version of the method without the transformation step has been used in recent work in health techonology assessment [5, 6], but has been refined and made more generalisable in the process of writing this paper.

When performing a PSA, often only the summary statistics of the sampled realisations are compared with their given values to check the statistical validity. We contend that when sampling from ordered variables, it is also important to consider the clinical validity of the induced correlations between the sampled values. If it is believed that the ordered variables are neither independent nor perfectly dependent, then the independent sampling approach or the common random number generator approach should be avoided. Assuming different parametric distributions in the proposed DM approach may result in induced correlations of different magnitudes; whether there is a best choice in specific scenarios is subject to further research. The modified independent sampling approach should be avoided in all cases because it produces biased sampled realisations.

One limitation of this paper is that we have not conducted a review of the methods used in handling the logical problem in ordered parameters, and have only undertaken one example to illustrate the properties of the proposed method. One drawback of the DM approach is that it does not work if utilities are believed to be below 0. A modification of the method is needed and this requires future research. Some ordered parameters may have a multiplicative relationship rather than an additive. A multiplier approach could be an alternative approach to handle the logical problem. Assigning an appropriate distribution for the multiplier parameter in PSA may be a challenge because it may not have properties as convenient as those in the difference parameter.

We have illustrated how the DM approach works in the case of two ordered parameters. It can be extended to the case of more than two ordered parameters by taking an iterative approach of the proposed method. For example, if there are three variables and , where the value of is greater than the value of , and the value of is greater than the value of , then we need to define two difference parameters: represents the difference between and , and represents the difference between and . We firstly sample and using the proposed DM approach, then apply the DM approach again to sampled and to derive samples for .

This logical problem of one variable being strictly greater than another also exists in one-way sensitivity analysis. The proposed method can be used in probabilistic one-way sensitivity analysis to handle ordered parameters. For deterministic one-way sensitivity analysis, making the same change in the related variables will make sure the parameter order remains unchanged; this ‘same change' could either be relative (i.e. a multiple) or absolute (i.e. an increment). This approach is similar to the common random number generator approach in PSA.

Conclusions

When producing PSA samples, both clinical and statistical validity should be checked. Where there is a strong belief that variables are constrained in that one value is greater than another, the DM approach should be considered as a method of ensuring the clinical and statistical validity of PSA samples, and the analyses derived from these samples.

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSX 4018 kb)

In health economic models, uncertain ordered parameters are common. For example, utility and treatment costs associated with different severity levels of a disease are commonly ordered.

For ordered parameters, which are believed to be related such that the value of one variable is always greater than another, this information on ordering should always be appropriately incorporated within PSA.

The proposed sampling approach generates ordered parameters with both clinical and statistical validity for use in PSA.