Affordability of oncology drugs: accuracy of budget impact estimations.

Background: In many countries, Budget Impact (BI) informs reimbursement decisions. Evidence has shown that decision-makers have restricted access based on high BI estimates but studies show that BI estimates are often inaccurate. Objective: To assess the accuracy of BI estimations used for informing access decisions on oncology drugs in the Netherlands. Study Design: Oncology products for which European Medicines Agency Marketing Authorisation was granted between 1-1-2000 and 1-10-2017 were selected. Observed BI data were provided by FarmInform. BI estimates were extracted from the reimbursement dossiers of the Dutch Healthcare Institute. Products without an estimated BI in the reimbursement dossier were excluded. Accuracy is defined as the ratio observed BI/estimated BI. Setting: General community, the Netherlands.
Results: Ten products were included in the base case analysis. Mean accuracy was 0.64 and observed BI deviated by more than 40% and 100% from the estimated BI for 4 and 5 products, respectively. For all products together, €141 million BI was estimated and €82 million BI was observed, a €59 million difference. Conclusions: The findings indicate that BI estimates for oncology drugs in the Netherlands are inaccurate. The role and use of BI in reimbursement decisions for these potentially life-saving drugs should therefore be considered carefully, as well as BI estimation methodology.

Introduction

Increased spending on oncology drugs is a global problem of major concern [1,2]. The global annual spending on oncology drugs was around US$ 35 billion (€28.5 billion) in 2006, compared to around US $100 billion (€81.5 billion) in 2017. This figure is expected to rise to $150 billion (€122 billion) in 2020 [2]. The increasing number of oncology approvals for new and existing drugs, higher prices for personalized medicine and potential off-label use causes an increasing burden to health-care budgets and therefore results in growing affordability [3-5].

Budget impact analyses (BIA) are performed and submitted as part of reimbursement applications in many countries to quantify the potential budget impact (BI) [5-7]. Next to clinical evidence and cost-effectiveness, BI has a formal role in many countries in assessing affordability and therefore a role in informing reimbursement decisions (e.g., for orphan and specialty drugs) [6,8,9]. Various recent cases have shown that decision-makers have restricted access to new drugs based on high BI estimates [2,6,10-13].

BI estimates used by decision-makers for informing these reimbursement decisions are however often inaccurate [11,14]. Keeping et al. described that BI estimates used by the All Wales Medicines Strategy Group (AWMSG), the Welsh Health Technology Assessment (HTA) agency, deviated from observed BI with more than 40% in 80% of the cases [14]. Similar or even higher deviations were reported by others [15,16].

In the Netherlands, the estimated BI of hepatitis C drugs was nearly twice as high as the observed BI [11]. Also, among other types of information (e.g., cost-effectiveness ratios), BI estimates have contributed to decisions for initial access restrictions as well as price negotiations [11,17]. For oncology drugs, access restrictions are currently imposed [13,18]. The accuracy of BI estimates for oncology is however unknown, potentially hampering decision-makers in considering the uncertainty in BI outcomes. Especially in the oncology field, various schemes have been implemented by decision-makers to manage affordability and BI, such as the Cancer Drug Fund in the UK, and various managed entry schemes frequently used for oncology drugs in, for example, Sweden, Belgium and Italy [19-21]. In order to provide more insight into this source of uncertainty and to allow for a better appraisal of BI, this study aims to quantify the accuracy of BI estimations using Dutch reimbursement decisions on oncology products as an example.

Methods

Data source for observed and estimated BI

The Dutch National Healthcare Institute (ZIN) is the Dutch HTA agency and conducts BIA for new medicines [22]. The manufacturer of a new (expensive) medicine is obliged to provide ZIN with a reimbursement dossier, stating the product’s price and containing a description of estimated relative effectiveness, cost-effectiveness and budget impact [22]. The manufacturer’s dossier is assessed and appraised ZIN after which ZIN provides the data, findings, comments and advice to the Minister of Health (MoH) in a publicly available reimbursement dossier. The results (e.g., BI) presented in this final reimbursement dossier can therefore differ from the results supplied by the manufacturer.

BI typically pertains to the additional costs associated with the implementation of a new medicine in a healthcare system [5,23]. BI therefore not only includes the additional costs of the new medicine but also encompasses savings due to the replacement of another treatment. ZIN estimates BI by estimating the volume of the new medicine, composed of the size of the patient population, the market penetration and the treatment duration. This volume is then multiplied by the stated medicine price. Savings due to replacement is then subtracted to obtain the estimated BI. As the medicine price is regarded as a constant, only the volume-related factors have to be estimated and are therefore uncertain. As market uptake of a new medicine is gradual, ZIN estimates BI for the first 3 years after market introduction where the BI in the third year serves as the benchmark for the reimbursement advice to the MoH [22].

As (savings by) replacement of alternate therapies is hard to measure and as the relative influence of replacement of the existing treatment vs the introduction of the new treatment is expected to be low in oncology, we henceforth define BI solely as the additional expenses for the new treatment [17,24]. This methodology has successfully been used in other studies describing BI estimation accuracy [11].

The observed BI data were provided by FarmInform [25]. This Dutch population-level data source contains the monthly volume of all in- and outpatient prescription drugs. BI was calculated by multiplying the volume by the Dutch list price [26]. This BI data were available for the period from 1 January 2000 to 1 March 2018. The validity of the data was assessed by crosschecking with a patient-level data source that is representative of the Netherlands [27].

BI estimates were extracted from the publicly available ZIN reimbursement dossiers [28]. The date of publication of the dossier was denoted as index date. The estimated BI in the entire third year after the index date was used as estimated BI. Costs or savings due to the replacement of alternate therapies were then subtracted. This yields an estimated BI that is solely composed of the costs (as price * volume) of the new medicine and is therefore comparable with the observed BI.

Product inclusion

Products were selected that obtained a European Medicines Agency (EMA) Marketing Authorization (MA) between 1-1-2000 and 1-10-2017. To select oncology products, only products were included that belonged to the L01 (antineoplastic agents) ATC category and had an initial oncology indication as specified by the European Public Assessment Report (EPAR) [29]. Biosimilars, generics and products that were not designated as a ‘New Active Substance’ by the EMA were excluded. Furthermore, a ZIN BIA should be available. Finally, products without the full market data of the third year available were also excluded.

Assessment of BI estimates from reimbursement dossiers

In some cases, a BI range is given in the reimbursement dossiers. In these instances, the average of the minimum and maximum BI estimate was included.

Oncology products can have multiple indications and Dutch reimbursement decisions are typically on indication level instead of on product level. Therefore, a BIA is typically tailored to a specific indication. The observed BI data source does not contain data on indication. In order to accurately assess the BI estimation accuracy, all observed BI must be attributable to the indication covered in the BIA. In other words, there must be no second indication that generates BI as this will falsely be attributed to being generated by the indication covered in the BIA and thus alter estimation accuracy.

BI can be generated (and recorded) sometime before the index date as a form of early access or compassionate use scheme. Similarly, the actual availability of a product (and consequent BI) can lag behind the index date. We therefore assumed that when the first BI record in the dataset was at most 6 months from a product’s index date, all observed BI was generated by the indication for which BI was estimated. Only the products that meet this criterium were included in the primary analysis. The products that did not meet this criterium were included in a secondary analysis.

When an additional reimbursement dossier (including BIA) was published within the 3 years after the index date, we adapted estimated BI to reflect this. In this case, we added the estimated BI of the second indication to the estimated BI of the initial indication. This addition was performed on the publication date of the second indication’s dossier.

Accuracy definition

Estimation accuracy was calculated as:

We used this description of accuracy as primary outcome as it is frequently used and easy to interpret. When however using Equation (1), underestimations yield ratios from 1 to infinity whilst overestimations yield ratios from 0 to 1 so averaging these ratios gives biased results [30,31]. To overcome this, a symmetric accuracy in the form of Equation (2) was used as a secondary outcome [30,31].

In order to allow for a comparison with other literature, estimation accuracy was also calculated as percentage difference, defined in Equation (3).

To illustrate the uptake and estimation accuracy over time, we plotted accuracy against the index date. For improved interpretability, loess smoothing was applied and was performed using R version 3.5.2 and ggplot2 version 3.1.1 [32,33].

Results

Nineteen products were included of which 10 are included in the primary analysis and 9 in the secondary analysis. Table 1 displays the characteristics and outcomes of all the included products. In Table 2, the aggregated outcomes are presented.

Table 1.

Characteristics and budget impact of 19 oncology products

Product	Active substance	Analysis	Estimated BI (€)	Observed BI (€)	Net difference (€)	Accuracy	Symmetric accuracy	Percentage difference	Index date	First BI record
Arzerra	Ofatumumab	Primary	1,210,556	446,061	−764,495	0.37	2.71	171.39	23 May 2011	1 December 2010
Caprelsa	Vandetanib	Primary	2,500,000	626,653	−1,873,347	0.25	3.99	298.95	1 November 2012	1 January 2013
Jevtana	Cabazitaxel	Primary	5,900,000	8,214,398	2,314,398	1.39	1.39	−28.17	26 September 2011	1 April 2011
Nexavar	Sorafenib	Primary	28,155,960	3,803,336	−24,352,624	0.14	7.40	640.30	27 October 2006	1 January 2007
Sprycel	Dasatinib	Primary	14,470,548	6,896,229	−7,574,319	0.48	2.10	109.83	6 March 2007	1 February 2007
Sutent	Sunitinib	Primary	27,751,023	11,860,601	−15,890,422	0.43	2.34	133.98	27 October 2006	1 November 2006
Tarceva	Erlotinib	Primary	14,573,357	11,318,537	−3,254,821	0.78	1.29	28.76	9 March 2006	1 January 2006
Vectibix	Panitumumab	Primary	4,977,360	5,355,496	378,136	1.08	1.08	−7.06	28 April 2008	1 August 2008
Vidaza	Azacitidine	Primary	10,881,360	6,705,395	−4,175,965	0.62	1.62	62.28	25 May 2009	1 March 2009
Yervoy	Ipilimumab	Primary	30,240,000	26,885,500	−3,354,500	0.89	1.12	12.48	23 January 2012	1 March 2012
Alimta	Pemetrexed	Secondary	21,600,000	26,742,714	5,142,714	1.24	1.24	−19.23	22 June 2009	1 November 2004
Avastin	Bevacizumab	Secondary	6,504,960	46,243,910	39,738,950	7.11	7.11	−85.93	22 October 2007	1 February 2005
Erbitux	Cetuximab	Secondary	10,154,100	2,010,656	−8,143,444	0.20	5.05	405.01	29 January 2007	1 May 2004
Herceptin	Trastuzumab	Secondary	18,000,000	68,700,329	50,700,329	3.82	3.82	−73.80	26 April 2010	1 October 2000
Mabcampath	Alemtuzumab	Secondary	6,237,840	351,045	−5,886,795	0.06	17.77	1676.93	28 August 2006	1 January 2002
Torisel	Temsirolimus	Secondary	2,506,837	606,085	−1,900,752	0.24	4.14	313.61	25 August 2008	1 January 2008
Tyverb	Lapatinib	Secondary	10,498,488	234,418	−10,264,070	0.02	44.79	4378.54	26 January 2009	1 September 2011
Velcade	Bortezomib	Secondary	9,921,500	12,289,775	2,368,275	1.24	1.24	−19.27	24 September 2007	1 June 2004
Zelboraf	Vemurafenib	Secondary	26,459,460	4,561,173	−21,898,287	0.17	5.80	480.10	24 February 2014	1 April 2012

Table 2.

Aggregated budget impact accuracy. Primary analysis set indicates that a product’s first BI record occurred within 6 months of the index date

Analysis (n) (first BI record <6 months from index date)	Estimated BI (€)	Observed BI (€)	Mean accuracy	Mean symmetric accuracy	Mean percentage difference (%)	Total net difference (€)
Primary (10)	140,660,163	82,112,205	0.64	2.50	142.2	−58,547,958
Secondary (9)	111,883,185	161,740,105	1.57	10.10	784.00	49,856,920
Combined (19)	252,543,348	243,852,310	1.08	6.10	446.25	−8,691,038

Primary analysis

Table 1 shows that the BI estimate for Nexavar was least accurate with an accuracy of 0.14 and a €24 million net overestimation. As can be seen in Table 2, the mean accuracy is 0.64, the mean percentage difference is 142% and the mean symmetric accuracy is 2.50. These results illustrate that, on average, for each €1 estimated only €0.64 is observed. For each individual product, an estimated €1 compared to a mean observed BI of €0.40 or €2.50, dependent on whether BI is over- or underestimated, respectively. The total observed BI of €82 million was overestimated by €58.5 million.

In Figure 1, the development of accuracy over time is presented. Time is defined as the months from the index date. Accuracy of 1 implies that observed BI exactly matches estimated BI. As the ZIN BI estimates are for the whole third year, the average ratio in months 24–36 should ideally be 1. Figure 1 shows that BI uptake is rather gradual and that products that are overestimated at around 12 months generally do not reach their estimated BI.

Figure 1.

Estimation accuracy of the primary analysis. Primary analysis indicates that a product’s first BI record was within 6 months of the index date

Secondary analysis

In the secondary analysis, individual accuracy (see Table 1 and Figure 2) is lower than for the products included in the primary analysis. This is however at least partly due to the previously described difference in indication. For Herceptin, for example, no reimbursement dossier was available for the initial (and very large) indication of HER2-positive breast cancer [34]. The Herceptin BIA, which resulted in inclusion in this study, only covered out-patient Herceptin use [34]. The relative BI of this subgroup was relatively small and therefore impacts the results. For Avastin, high off-label use likely caused a very high observed BI relative to the indication for which BI was estimated.

Figure 2.

Estimation accuracy budget impact in the secondary analysis. Secondary analysis indicates that a product’s first BI record was separated from the index date by more than 6 months

Five products (Erbitux, Tyverb, Zelboraf, Torisel, Mabcampath) display a very low ratio. Of these products, Mabcampath, Zelboraf and Erbitux had early market exposure relative to their index date. Despite this potential availability of market data, their BI estimates were still rather inaccurate.

Discussion

This study aimed to quantify the accuracy of BI estimates reported in reimbursement dossiers, using oncology products in the Netherlands as an example. Of the 10 products included in the primary analysis, the estimated BI was €140.7 million whilst only €82.1 million was observed. This difference would imply an aggregated accuracy of 0.58 with a mean accuracy for the individual products of 0.64. The mean symmetric accuracy and the mean percentage difference were 2.50% and 142%, respectively.

Keeping et al. assessed the accuracy of BIAs conducted by the AWMSG [14]. They reported that only 18% of estimations were within 40% of observed BI as defined by percentage difference. In the third year after introduction, Keeping et al. reported that 7 (14%) and 13 (27%) products of a total of 49 were within 40% and 100% of estimated BI. In our (primary) analysis, these figures were 4 (40%) and 5 (50%). The results are not directly comparable as Keeping and colleagues only included two oncology products.

Cha et al. and Broder et al. also investigated BIA accuracy but their relevance in this regard is limited as they did not target BIAs for informing reimbursement decisions [15,16]. They did, however, report considerable estimation inaccuracy (5.5-fold overestimation by Broder et al. and 60% of products deviated by >40% by Cha), strengthening the evidence base that BI estimates are often inaccurate and thus pose a source of uncertainty to decision-makers that need to take BI into account.

The secondary analysis shows some interesting findings, with, for example, Mabcampath and Zelboraf, that show BI records preceding the index date by more than 6 months. For these products, observed BI remained well below the BI estimations reported in the eventual BIA. If the initial BI was generated by a different indication than covered in the BIA, the over-estimation relative to the indication in the BIA would be even greater. In other words, an indication that generated BI but is not covered in the BIA contributes to the observed BI, thereby increasing the value of the accuracy outcome (see Equation 1). Therefore, for products with overestimated BI (accuracy <1; Mebcampath, Zelboraf, Tyverb, Torisel, Erbitux), it could be assumed that the reported accuracy is a best-case scenario. These five products incurred BI that deviated by >100% from estimated BI. We can therefore conclude that at least 10 of 19 (53%) products deviated by >100%, and potentially even 14 of 19 (74%) products.

Our findings thus show that BI estimates are generally inaccurate. It is known that BI estimates are often used for confidential price negotiations between manufacturers and payers [35]. In that regard, a higher BI estimate could be an incentive and argument for a payer to attempt to negotiate a discount, although we have no evidence to support this suggestion. Our findings regarding BI estimation inaccuracy should, however, warrant careful consideration of the role of BI in access decisions. Given the inaccuracy and large deviation in accuracy between products, uncertainty in BI is evident and should definitely be considered when BI informs decision-making.

Furthermore, as indicated by a review by Vooren et al., limitations in current BIA methodology and/or lack of adherence to the ISPOR budget impact guidelines might be a cause for the relative lack of accuracy [8,36]. We believe that, given the importance of BI in decision-making, efforts should be undertaken to improve BIA methodology and, for example, to properly address uncertainty in BI estimates.

We have some limitations to address. First, our sample size with regard to the number of products was quite small. We, however, chose to impose limitations on the deviation of the first BI record from the index date to assure that the comparison of the estimated BI is accurate. Our secondary analysis shows that for at least five more products, we can conclude that their BI assessment deviated by >100%.

Second, we were unable to account for (off-label) indication extensions for which ZIN did not publish a BIA. As BIAs typically are conducted for expected BI over €2.5 million, we assume that indication extensions without BIA would have generated a relatively low BI and thus had little influence on our results [23,37].

Conclusion

The 10 products included in the primary analysis resulted in total BI estimates of €140.7 million whilst only €82.1 million was observed. For at least 53% of the included products, the observed BI differed from the estimated BI by more than 100%. These findings, combined with the large deviation in accuracy between different products, lead us to conclude that Dutch BI estimates for oncology drugs are often inaccurate. They are thus an inevitable source of uncertainty to the decision-maker impacting reimbursement decisions and therefore potentially leading to restricted access for patients. The role and use of BI in reimbursement decisions for these potentially life-saving drugs should therefore be considered carefully, as well as the need for improvements in BI estimation methodology.