Analyzing a Cost-Effectiveness Dataset: A Speech and Language Example for Clinicians.

Cost-effectiveness analysis, the most common type of economic evaluation, estimates a new option's additional outcome in relation to its extra costs. This is crucial to study within the clinical setting because funding for new treatments and interventions is often linked to whether there is evidence showing they are a good use of resources. This article describes how to analyze a cost-effectiveness dataset using the framework of a net benefit regression. The process of creating estimates and characterizing uncertainty is demonstrated using a hypothetical dataset. The results are explained and illustrated using graphs commonly employed in cost-effectiveness analyses. We conclude with a call to action for researchers to do more person-level cost-effectiveness analysis to produce evidence of the value of new treatments and interventions. Researchers can utilize cost-effectiveness analysis to compare new and existing treatment mechanisms. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/).

Learning Outcomes: As a result of this activity, the reader will be able to (1) explain the components of a cost-effectiveness dataset; (2) describe how to estimate the cost-effectiveness of a new treatment or intervention using a dataset; and (3) characterize uncertainty about cost-effectiveness of a new treatment or intervention using a dataset.

Nearly two decades ago, O'Brien et al published a seminal article explaining how to analyze clinical studies that collect patient-specific cost and effect data. 1 Such data present an opportunity to analyze cost-effectiveness using conventional statistical methods. While the analytical techniques are relatively straightforward, researcher uptake has been slow. For example, a recent review of the cost-effectiveness of telehealth (a new mode of practice) for the delivery of speech- language pathology services to children found no economic evaluations. 2 There are few articles in speech and language that analyze cost-effectiveness datasets, but some examples do exist. 3 4 Indeed, de Sonneville-Koedoot and colleagues observed that, “rapidly increasing health care expenditures force policy makers to make explicit decisions about expenses of the health care budget… [but] in the field of speech and language pathology economic evaluations are relatively uncommon.” 5 Given the usefulness of cost-effectiveness evidence to understanding the value of new treatments and interventions, it is surprising more researchers do not collect cost data alongside clinical trials.

Value is a key consideration for decision-makers deciding how to allocate their fixed budgets. 6 Importantly, value is different from cost. Cost is what the payer pays, and effect or outcome is what is received or produced. In contrast, value is about worth (e.g., is the extra outcome worth the extra cost). Whether a new treatment or intervention is a good value rests on a judgment about the worth of both its additional cost and its additional outcomes. Consequently, researchers must go beyond a focus on effectiveness and instead provide evidence of value (e.g., whether funding a new treatment or intervention is a prudent use of scarce resources). 7 However, to show that a new treatment or intervention represents good value for money requires more than just data on costs and outcomes. After obtaining the cost-effectiveness data, one must analyze them correctly and explain the findings clearly so that the implications may be communicated correctly.

This article illustrates a simple method to analyze a cost-effectiveness dataset. As recommended by experts, 1 8 we focus the cost-effectiveness analysis on creating estimates and characterizing uncertainty. After introducing our hypothetical dataset, we demonstrate how to use the net-benefit regression framework to estimate a simple linear regression with ordinary least squares (OLS). We then provide an example of how to interpret the findings. This article concludes with a call to action for researchers to do more person-level cost-effectiveness analysis to produce evidence of the value of new treatments and interventions.

Methods

Data Requirements

A dataset on which a cost-effectiveness analysis can be performed (which we call a cost-effectiveness dataset) should contain at least three variables: cost, effect, and a treatment indicator. The relevant cost data to collect are related to the perspective of the analysis. The costing perspective (i.e., whose costs are counted) can be quite broad (e.g., a societal perspective) or focused (e.g., the drug budget of an employer's health insurer). When trying to influence the one who will pay for a treatment or intervention, it is best to include the payer's perspective in the analysis. This means collecting data on the costs that will be incurred by the payer because of the new treatment. There may be other relevant perspectives (e.g., those reflecting the interests of other stakeholders who are affected by the treatment) to consider. For example, Liu et al 9 studied the cost-effectiveness of speech and language therapy plus scalp acupuncture versus speech and language therapy alone for community-based patients with Broca's aphasia after stroke. The analysis was performed from a societal perspective, including all expenses related to the interventions irrespective of who paid. Nevertheless, the payer's perspective provides essential information to support a funding decision for a new treatment or intervention. For this reason, most economic evaluations conducted for the purpose of supporting funding requests for a new option estimate the extra costs of the new option from the funder's perspective. In addition to these cost data, a cost-effectiveness analysis also considers outcome, also known as effect.

There can be tension in choosing a variable to represent the effect of a new treatment or intervention. Sometimes a condition-specific outcome is appealing from a clinical perspective; however, from a policy perspective, a broader outcome is needed (e.g., like a quality-adjusted life-year [QALY], introduced later). For example, Ellis et al 10 studied the cost-effectiveness analysis of treatments for aphasia using proficiency of performance as the “effect” variable. i The researchers estimated an average overall gain in proficiency of 43.19% which came at an additional cost of $412.15. They calculated $9.54 as the extra cost of a 1% gain (since $412.15/43.19% ≈ $9.54). If this finding were shared with a decision-maker to inform a decision about whether to fund a treatment for aphasia, it would be challenging to put the extra cost for a 1% gain in a broader context. This is the tension surrounding the choice of the “effect” variable. With a goal of improving aphasia outcomes, choosing an “effect” like proficiency of performance makes good sense. However, from a broad health policy perspective, a general health outcome is often more helpful, particularly if one is competing for funding dedicated to improving health outcomes in general, rather than aphasia outcomes specifically. For this reason, researchers often employ an additional, more general outcome like the QALY.

Measurement Tools: QALYs

The QALY is a measure commonly used to demonstrate overall improvement in health. QALYs can be measured for patients with any type of disease or condition. Palmer et al 11 proposed to use the QALY for their outcome in a cost-effectiveness analysis of computerized speech and language therapy or attention control added to usual care for people with long-term post-stroke aphasia. The QALY gain was 0.017 for computerized speech and language therapy compared with usual care. If the outcome were life years, then 0.017 life years gained is approximately 6 additional days of life (since 0.017 years gained is 0.017 years × 365 days per year ≈ 6 days). Using life years as the outcome assumes that the main goal of the intervention is to extend length of life. In contrast, rather than life year, the study's outcome is QALYs. This suggests the primary outcome is driven by improvements in quality of life (as opposed to length of life). If one believes that computerized speech and language therapy or attention control added to usual care for people with long-term post-stroke aphasia is unlikely to increase length of life, then most/all of the 0.017 gain in QALYs is due to quality-of-life improvement.

Dividing the extra cost by this 0.017 QALY gain produces an estimated extra cost of about $57,000 for a 1-unit QALY gain. ii The $57,000 can be interpreted as an efficiency rate: the new intervention provides extra outcome at a rate of $57,000 for a 1-unit gain in QALYs. The intervention appears even more economically attractive for specific patient subgroups. For both mild and moderate word-finding difficulty subgroups, the estimates of the extra cost for an additional QALY gain were both less than $40,000. This extra cost for a 1-unit QALY gain can now be compared easily to other treatments and interventions for other diseases or conditions that used QALYs as their outcome measure.

For example, the median extra cost for an additional QALY gain is nearly $150,000 for FDA-approved cancer drugs from 2015 to 2020. 12 In comparison, Palmer et al's cost-effectiveness estimates appear to be a quite reasonable rate to pay for an additional QALY. The median cancer drug is almost three times more costly than estimates for computerized speech and language therapy or attention control added to usual care for people with long-term post-stroke aphasia. The practical implication is that by choosing an outcome measure like the QALY that is “more general” (i.e., one that could be used in the study of any disease), the economic efficiency of speech and language treatments and interventions can be compared with other healthcare investments (e.g., those for cancer).

Measurement Tools: Willingness-to-Pay

Often, a cost-effectiveness analysis does not determine whether a new treatment or intervention is cost-effective. For example, in the research by Ellis et al 10 studying the cost-effectiveness of aphasia treatments, the extra cost was $9.54 for a 1% gain in outcome. While this is almost an order of magnitude lower than the one previous study that attempted to examine the cost-effectiveness of aphasia treatment, there is still the issue of how much an improvement is worth. 13 In fact, the title of Boysen and Wertz's study of aphasia therapy includes the question “How much is a word worth?” They estimate a 1% gain costing between $206 and $567. 13 Clearly, Ellis et al's 1% gain for $9.54 is more economically attractive than Boysen and Wertz's. In reality, it is the decision-maker who determines an outcome's worth (i.e., it is the decision-maker's willingness-to-pay). Thus, the funding implications of a study turn on the decision-maker's unknown willingness-to-pay value, which we call λ.

When the willingness-to-pay value λ is ≥ $10, the aphasia treatment studied by Ellis et al is cost-effective. For example, if the decision-maker is willing to pay $10 (i.e., λ = $10) for a 1% gain, then the extra effect of 43.19% is worth $431.90 (i.e., $10 × 43.19). The extra cost is $412.15 for this extra benefit of $431.90. The incremental net benefit (INB) is extra benefit – extra cost. In this case, INB = $431.90 − $412.15 = $19.75. The extra benefits (ΔB) outweigh the extra costs (ΔC) by $19.75 when a 1-unit gain of extra effect (ΔE) is valued at a willingness to pay (λ) of $10. iii In this example, λ = $10, the incremental cost-effectiveness ratio (ICER) is ΔC/ΔE = $9.54, so a 1-unit gain costs less than the willingness to pay for a 1-unit gain. To summarize, if INB > 0, the new treatment or intervention is cost-effective; however, to compute INB, one needs ΔC and ΔE (which can be estimated from the data) as well as a value for λ (which is unknown).

When choosing a general outcome measure such as the QALY to be the effect or outcome variable, researchers often turn to commonly used willingness-to-pay values (e.g., λ = $50,000). 14 However, when using a condition-specific outcome measure, as is more common in clinical studies, it is less clear what value to use for λ. As a result, we look at different values for the unknown willingness-to-pay value λ (from $0 to something large) to see how the conclusions could be affected. This is accomplished by graphing the estimated INB against a variety of potential values for the unknown λ in an INB by willingness-to-pay plot (e.g., see Fig. 1 ). Next, we review the steps to make the estimate of the INB and create the recommended graphs for cost-effectiveness analysis.

Figure 1

Incremental net benefit by willingness-to-pay plot.

The Analysis of a Hypothetical Dataset

Let us consider a hypothetical situation in which our dataset contains costs represented as a sum of the product of service units and unit costs plus any additional relevant costs. We denote this as cost . Our effect variable is a condition-specific outcome measure. It could be any of the outcomes collected in the study by Palmer et al 11 including but not limited to a (1) change in word finding ability; (2) change in functional communication; (3) generalization to untreated words (e.g., as measured by a score on the Comprehensive Aphasia Test Naming Objects subtest) 15 ; or (4) carer's perception of change in communication. We assume the outcome measure is a continuous measure and will call it effect . Lastly, the dataset includes a treatment indicator (or dummy) variable called tx that equals 1 for study participants receiving the new treatment or 0 otherwise (e.g., receiving standard practice or usual care). The hypothetical data shown in Table 1 will be used to illustrate how researchers can apply net-benefit regression to estimate and explain a new treatment's cost-effectiveness.

Table 1

Hypothetical Cost-effectiveness Dataset

Observation number	New treatment ( tx )	Cost	Effect
Usual care (N = 7)
1	0	3,000	100
2	0	2,000	100
3	0	2,500	100
4	0	2,000	150
5	0	2,000	150
6	0	1,500	200
7	0	1,500	300
New option (N = 10)
8	1	4,000	170
9	1	3,000	150
10	1	2,000	200
11	1	4,000	275
12	1	3,000	300
13	1	3,000	300
14	1	2,000	300
15	1	2,500	360
16	1	3,000	360
17	1	3,000	360

Cost and Effect Regression Analysis

When analyzing a cost-effectiveness dataset using regression analysis, start by estimating extra cost (ΔC) and extra effect (ΔE). By using OLS to estimate coefficients for both of the simple linear regressions on cost data ( cost ) and effect data ( effect ):

cost  = β c0  + β ΔC tx

effect  = β e0  + β ΔE tx

one obtains the estimates for ΔC and ΔE from β ΔC and β ΔE , respectively. These are the most important parts of any cost-effectiveness analysis since they are used to make the ICER and the INB.

Cost-Effectiveness Estimates: Net Benefit Regression

By using OLS to estimate coefficients for the simple linear regression:

nb  = β nb0  + β ΔNB tx ,

where nb is calculated as λ ×  effect – cost , we get the INB of new treatment which equals the β ΔNB estimate. In net benefit regression, the coefficient estimate on the tx variable is the INB. 16 If β ΔNB  > 0 then INB > 0 indicating cost-effectiveness. Likewise, the 95% confidence interval for β ΔNB is the 95% confidence interval for INB. These are the two dashed lines in Figure 1 .

Cost-Effectiveness Uncertainty: Net Benefit Regression with 95% Confidence Intervals

A simple linear regression (with λ = $30) produces an INB estimate of about 2,700 with lower and upper 95% confidence limits of −4 and 5,460. This 95% confidence interval just excludes zero and this is reflected in the p  = 0.050. One can see this brush with significance in Fig. 1 as the lower dashed line touches the horizontal axis when λ = 30. Table 2 shows the results for a variety of willingness-to-pay values for λ = 0 to 40.

Table 2

Using the Results from Net Benefit Regression to Characterize Uncertainty

Assumed	From regression				Calculated
Willingness–to-payλ value	Treatment indicator coefficient estimate a	Lower 95% confidence limit a	Upper 95% confidence limit a	2-sided p -value	1-sided p -value	Probability of cost-effectiveness
$0	−880	−1,510	−250	0.010	0.005	≈ 1%
$5	−280	−1,170	610	0.517	0.259	26%
$10	325	−900	1,550	0.580	0.290	71%
$15	930	−660	2,510	0.232	0.116	88%
$20	1,530	−430	3,490	0.117	0.059	94%
$30	2,730	−4	5,460	0.050	0.025	97%
$40	3,940	430	7,440	0.030	0.015	98%

Number rounded to the nearest 10.

Fig. 1 is a graph of the columns in Table 2 labeled “treatment indicator coefficient estimate,” “lower 95% confidence limit,” and “upper 95% confidence limit.” The last row of Table 2 , where λ = $40, has positive values in all three of these columns. This corresponds to the two dashed lines and one solid line being above the horizontal axis in Fig. 1 . With these results, we can reject the null hypothesis that INB = 0. Likewise, the first row of Table 2 has a p -value < 0.05, and we can reject the null hypothesis that INB = 0. However, more relevant is whether INB > 0. This is equivalent to a one-sided hypothesis test to reject the null hypothesis that INB < 0. iv We use the p -value from net benefit regression to create a measure of the probability that INB > 0 (i.e., that the new treatment is cost-effective).

Cost-Effectiveness Uncertainty: Net Benefit Regression p -Values

Table 2 also has the information necessary to make a cost-effectiveness acceptability curve (CEAC) using the technique described by Hoch et al. 17 A CEAC illustrates the probability of cost-effectiveness in relation to a variety of possible willingness-to-pay λ values. Fig. 2 is an example of this. One can make Fig. 2 from the hypothetical data by converting the two-sided p -value into a one-sided p -value (dividing by 2) and either plotting the resulting number (if INB < 0) or plotting that amount subtracted from 100% (if INB > 0).

Figure 2

Cost-effectiveness acceptability curve.

Results

The descriptive statistics from the hypothetical cost-effectiveness data appear in Table 3 . Table 4 shows the incremental cost (ΔC), the incremental effectiveness (ΔE), and the ICER (ΔC/ΔE). To assess the cost-effectiveness of the new option, one must compare the ICER to the willingness-to-pay value λ, and we conduct this comparison using the INB. As the willingness-to-pay value λ is unknown, we plot the INB against a λ with values varying from 0 to something large. When λ = $0, the INB is INB = $0 × 120–$880 = − $880. When λ equals $880/120, the INB = ($880/120) × 120–880 = 0. This means that if a decision-maker values an extra unit of patient outcome at exactly the new option's ICER, the extra benefits will equal the extra costs. The graph of INB versus λ is a line with slope equal to ΔE and y -intercept equal to −ΔC.

Table 3

Descriptive Statistics from a Hypothetical Cost-effectiveness Dataset

Variable	Mean	SD	SE	Correlation
Usual care (N = 7)
Cost	2,071.43	534.52	202.03	−0.76
Effect	157.14	73.19	27.66
New option (N = 10)
Cost	2,950.00	685.16	216.67	−0.20
Effect	277.50	78.71	24.89

Table 4

Incremental Cost, Incremental Effectiveness, and Incremental Cost-effectiveness

Variable	Mean a	SE	Correlation	ICER a
Incremental (N = 17)
Cost (ΔC)	880	295.86	−0.48	−0.76
Effect (ΔE)	120	37.09

Number rounded to the nearest 10; ICER = incremental cost-effectiveness ratio = ΔC/ΔE.

The solid line in Fig. 1 illustrates the INB with λ ranging from $0 to $40. In this example, the INB line runs through the vertical and horizontal intercept points at (0, −880) and (7.33, 0), respectively. The vertical intercept is at −ΔC (i.e., −880). The slope of the INB line equals ΔE (i.e., 120). The horizontal intercept is at the ICER (i.e., ΔC/ΔE = 880/120 = 7.33). The horizontal intercepts for the dashed 95% confidence limits for the INB are the 95% confidence limits for the ICER . When the willingness-to-pay value is λ = $40, all three lines in Fig. 1 are above the horizontal axis. After λ > $40, the study conclusions do not change; there is little uncertainty that the new intervention is cost-effective (i.e., has a positive INB when compared with usual care). We can identify this point since it occurs when the lower dashed line (the lower confidence interval) rises above zero. Net benefit regression is thus a simple way to produce both the INB estimate and the 95% confidence interval for the INB.

The CEAC provides another way to characterize uncertainty. In Table 2 when λ = $5, the INB < 0 and the two-sided p -value (reported by regression) is 0.517. The one-sided p -value is 0.517/2 = 0.259, and the probability of cost-effectiveness is 26%. Alternatively, when λ = $40, the INB > 0 and the two-sided p -value is 0.030. The one-sided p -value is 0.030/2 = 0.015, and the probability that the new treatment is cost-effective is 98%. Fig. 2 shows that the probability of cost-effectiveness starts quite low and then quickly increases before plateauing. Vertical axis values less than 50% in the CEAC ( Fig. 2 ) correspond to the INB < 0 in the INB by willingness-to-pay plot ( Fig. 1 ). When the probability of cost-effectiveness is very sensitive (i.e., changes dramatically when there is a change) to the choice of willingness-to-pay λ values, the curve in Fig. 2 will have a steep slope. In our example, for λ values from $15 to $40, the CEAC increases only a bit from 88 to 98%. However, for λ values between $5 and $15, the CEAC shows that decision-makers might change their opinions about the likelihood that the new treatment is a good use of money. Thus, even when the λ value is unknown, it is clear for which scenarios uncertainty about the value of λ leads to uncertainty about the findings' implications.

Discussion

This article described net benefit regression as a method to estimate cost-effectiveness and characterize uncertainty. Given the conclusions from a cost-effectiveness analysis involve considering cost and effect simultaneously, it is often more convenient to analyze both cost and effect together using net benefit regression. The technique can produce estimates of both the ICER and the INB which is useful because the INB has much nicer statistical properties than the ICER. 18 19 20 A challenge with using the INB is that the “right” willingness-to-pay is unknown; however, figures like the CEAC and the INB by willingness-to-pay plot circumvent the issue by varying the unknown λ to provide a sense of the value of a new treatment or intervention.

In our hypothetical example, Fig. 1 shows the INB estimate as a solid line and the ICER value where the solid line intersects the horizontal axis. The solid line has a positive slope; this means the new treatment or intervention is more effective than usual care (i.e., ΔE > 0). The solid line has a negative y -intercept indicating a more costly investment (i.e., ΔC > 0). When the solid line is above the horizontal axis (INB > 0) which occurs when WTP > 7.33, the new treatment or intervention is cost-effective. Therefore, if the decision-maker is willing to pay $8 or more for an additional 1-unit gain in outcome, the new option is good value. In addition to this information from the cost-effectiveness estimates, there is important information from examining the uncertainty.

Uncertainty is conveyed by the dashed lines for the INB in Fig. 1 and the CEAC in Fig. 2 . The dashed lines for the INB in Fig. 1 show the 95% confidence interval for the ICER, and where they intersect the horizontal axis (i.e., near $0 and $30). Thus, we can reject that the ICER is more than $35 for a 1-unit gain (or less than $0). The graph of the CEAC illustrates the probability of cost-effectiveness as we vary the unknown value λ. Given its shape, one can conclude the recommendations from the analysis are only sensitive to the unknown value λ when it is between $5 and $15. For scenarios when the λ value is unknown, the INB by WTP plot and the CEAC convey relevant information about a study findings' implications.

Limitations

This article covers methods for the analysis of cost-effectiveness data. but it does not describe how one might obtain the cost-effectiveness data, While the difficulties with measuring costs (e.g., lack of standardization around costs) are not often addressed in the literature, there does exist guidance on data collection tools that can be used to obtain reliable data for estimating costs. 21 Another challenge that researchers face is that funding decisions are political in nature. While economic efficiency may be one of the criteria by which a decision is made, it certainly is not the only one. Trenaman and colleagues studied how cost-effectiveness, contextual considerations, and other benefits were viewed in recommendations about value in the United States. 22 They found that while cost-effectiveness was important, judgments as to the value of a new treatment or intervention were influenced by other benefits and contextual considerations. Thus, even if there were a single willingness-to-pay number and it were known, a new treatment or intervention with INB > 0 (or INB < 0) might not be perceived as high (low) value.

But researchers should not despair. It is critical to take the initial steps to create an evidence base showing the value of new treatments and interventions. Speech and language professionals know the importance of communication, and communication about the value of what the field does must be strengthened with results from cost-effectiveness analysis. Based on the paucity of cost-effectiveness analysis showing the value of new treatments and interventions, the field knows more than it can say with its research about the value of what it does. Cost-effectiveness analysis is the right vehicle to express the extra cost of a new treatment or intervention considering its extra effect. Discussions about “what are we trying to achieve” and “how much should we be paying to get a 1-unit increase” are necessary in a world where not everything can be done. Evidence-based advocacy can be strengthened by participating in this discussion about value for money in healthcare.

Conclusion

The field of speech-language pathology has embraced research to generate new evidence about the effectiveness of new treatments or interventions. The next generation of research studies must embrace the collection of cost and effect (outcome) data. The analysis of a cost-effectiveness dataset is not an insurmountable challenge. Using net benefit regression, researchers can use regression to produce cost-effectiveness estimates and characterize their uncertainty. The required assumptions for regression to produce unbiased estimates are quite straightforward (i.e., relevant variables must be included in the analysis). For analysts concerned with the parametric assumptions necessary for characterizing uncertainty, nonparametric methods like bootstrapping can be employed and compared with standard parametric methods. 22 In addition, one can display the findings using the INB by willingness-to-pay plot and the CEAC. The construction of both graphs is facilitated by net benefit regression. Since both graphs vary willingness-to-pay along the horizontal axis, researchers do not need to know a decision-maker's willingness-to-pay (they just need to vary λ from $0 to something suitably large). In this way, speech-language pathology researchers can embrace their role of providing policy-relevant evidence, and then delegate decision making to the decision-makers.

There may be good reasons not to use a general health outcome measure (e.g., the QALY) v ; however, work is being done in this area to close the gap between where the research is and where it needs to be. 23 Until then, the field should consider using disorder-specific outcome measures as the effect and tracking costs from the perspective of a healthcare payer in each study of a more costly new treatment or intervention. Researchers can then use net benefit regression to quantify whether the new treatment or intervention is a good use of money. This can be accomplished by reporting an estimate of the INB and its 95% confidence interval. Modeling beyond the data (either to link a surrogate outcome to a more relevant one or to extend the length of the study period) can be used to address potential study shortcomings. However, if the findings of a clinical study are intended to influence treatment, researchers must go beyond effectiveness and into the arena of value.