Acceptance of health technology assessment submissions with incremental cost-effectiveness ratios above the cost-effectiveness threshold.

OBJECTIVES: In health technology assessment (HTA) agencies where cost-effectiveness plays a role in decision-making, an incremental cost-effectiveness ratio (ICER) threshold is often used to inform reimbursement decisions. The acceptance of submissions with ICERs higher than the threshold was assessed across different agencies and across indications, in order to inform future reimbursement submissions.
METHODS: All HTA appraisals from May 2000 to May 2014 from National Institute for Health and Care Excellence (NICE), Scottish Medicines Consortium (SMC), Pharmaceutical Benefits Advisory Committee (PBAC), and Canadian Agency for Drugs and Technologies in Health (CADTH) were assessed. Multiple technology appraisals, resubmissions, vaccination programs, and requests for advice were excluded. Submissions not reporting an ICER, or for which an ICER could not be determined were also excluded. The remaining appraisals were reviewed, and the submitted ICER, recommendation, and reasoning behind the recommendation were extracted.
RESULTS: NICE recommended the highest proportion of submissions with ICERs higher than the threshold (34% accepted without restrictions; 20% with restrictions), followed by PBAC (16% accepted without restrictions; 4% with restrictions), SMC (11% accepted without restrictions; 14% accepted with restrictions), and CADTH (0% accepted without restrictions; 26% with restrictions). Overall, the majority of higher-than-threshold ICER submissions were classified into the "malignant disease and immunosuppression" therapeutic category; however, there was no notable variation in acceptance rates by disease area. Reasons for accepting submissions reporting ICERs above the threshold included high clinical benefit over the standard of care, and addressing an unmet therapeutic need.
CONCLUSION: Acceptance of submissions with higher-than-threshold ICERs varied by HTA agency and was not significantly influenced by disease category. Such submissions must be accompanied by robust, concrete, and transparent evidence in order to achieve patient access.

Introduction

Health care decision-makers are faced with the challenge of identifying the optimal allocation of limited resources to optimize health benefit. However, due to constant budgetary constraints, most health care systems cannot make all new interventions developed for the detection, prevention, and treatment of diseases available.1

Governments are increasingly required to manage limited health care resources strategically, by investing only in those services that deliver the best health outcomes while still providing care that is affordable, effective, safe, and patient-centered. In response to these pressures, health care decision-makers have developed explicit systems, in the form of a health technology assessment (HTA), to identify interventions that provide the best value and aid resource allocation decisions.

Over the last 10 years, HTA has become an increasingly important part of the overall assessment system of new health technologies, and has the potential to assist payers in making informed decisions about allocating resources across the health care system. HTA examines the full range of clinical and economic evidence of a new intervention following market authorization, focusing not only on the health effects but also on the broader societal implications of introducing the intervention within the health care system.2

The objective of HTA programs is to make decisions regarding the acceptance, restriction, or rejection of interventions on a rational basis. Many HTA agencies use cost-effectiveness analysis to inform such decisions, where cost-effectiveness is expressed using incremental cost-effectiveness ratios (ICERs). ICERs are calculated by dividing the incremental cost of an intervention by the incremental benefit, which is often expressed in quality-adjusted life years (QALYs). QALYs are captured by applying a quality-adjusted weight to each health state of a disease (by balancing the quantity of life [or life-years gained] against the quality of that life), multiplying this by the time in each health state, and summing. Alternatively, disability-adjusted life-years or life-years gained may be used to estimate health benefit in cost-effectiveness analyses.1

Agencies that consider cost-effectiveness tend to use either a formal or informal ICER threshold to aid the decision-making process;3 however, there are some situations where applying an ICER threshold is insufficient. The concept of an ICER threshold refers to a balance between costs and health benefits that an intervention must achieve in order to be considered acceptable within a given health care system.1 By using an ICER threshold in HTA, the decision rule for any intervention is based on achieving a prespecified ratio of monetary costs per measure of health gain. The use of ICER thresholds in the HTA decision-making process ensures the consistency, transparency, and predictability of the process. ICER thresholds dictate that the results of the cost per health gain should be the sole decision criterion for resource allocation. However, this limits the incorporation of other considerations, such as societal preferences, budget impact of the intervention under evaluation, and the value of innovation.

Establishing the economic value of an intervention can be difficult in certain situations, such as in the case of orphan drugs, which are therapies indicated for the treatment of rare diseases. There is no globally accepted definition for an orphan disease, but the European Medicines Agency considers it to be a disease that affects fewer than 5 per 10,000 of the population, while the US Food and Drug Administration defines it as a disease affecting fewer than 200,000 people.4 Due to the small patient populations, orphan drugs are often associated with a high cost, making it unlikely that the drug will be able to demonstrate cost-effectiveness according to traditional thresholds. As a result, many HTA programs have established specific criteria with which to judge the cost-effectiveness of orphan drugs.5,6 Similarly, allowances may be made in some cases for drugs that extend life for patients with a short life expectancy (the so-called “end-of-life” criteria).7 The policy decisions around orphan drugs and end-of-life care demonstrate some of the difficulties of using ICER thresholds as the sole decision criterion of HTA.

While many HTA programs use formal or informal ICER thresholds to make decisions, whereby the calculated ICER for an intervention is a key criterion for decision-making, there remains an opportunity to incorporate other criteria within the process.3

The objective of this study was to identify the key rationale provided by four different HTA agencies – the National Institute for Health and Care Excellence (NICE, England), Scottish Medicines Consortium (SMC), Canadian Agency for Drugs and Technologies in Health (CADTH), and Pharmaceutical Benefits Advisory Committee (PBAC, Australia) – when their decisions went against the assumed ICER thresholds, and evaluate any differences between disease areas. We hypothesized that no significant differences in acceptance rates of submissions with ICERs higher than the threshold are observed between disease areas.

Methods

Data sources

All publicly available HTA appraisals from NICE, SMC, CADTH, and PBAC were searched from January 2000 to May 2014. These agencies were chosen as they represent key agencies that use cost-effectiveness criteria to inform decision-making around the globe and release publicly available, transparent, English-language appraisal documents for each intervention evaluated. Agency ICER thresholds (expressed in cost per QALY) were assumed for each of the four HTA agencies included in this analysis, based on published literature, agency publications, and inference based on previous decision-making trends.3,8,9 The following agency thresholds were assumed in the analysis: £30,000 per QALY for NICE and SMC; CAN$50,000 for CADTH; and AUS$42,000 for PBAC.

Appraisal selection

All single technology appraisals across the four agencies were assessed in this analysis. Appraisals were not excluded on the basis of disease area, indication, or type of intervention. Not all HTA submissions present a cost-effectiveness analysis, instead using a cost-minimization analysis or budget impact analysis. Since this analysis considered the rationale for decisions that went against the assumed ICER thresholds, other types of economic evaluations were not relevant to this analysis. Therefore, submissions that did not present a cost-effectiveness analysis expressed in cost per QALY, or for which an ICER could not be determined, were excluded. Multiple technology appraisals, resubmissions, vaccination programs, and requests for advice were also excluded, due to the practical difficulties of extracting singular ICERs for such submissions.

Data extraction

The full responses for all appraisals that met the inclusion criteria of this analysis were reviewed. The manufacturer’s base case ICER, recommendation, and rationale for the recommendation were extracted for all appraisals. The base case ICER was extracted to ensure consistency in the approach; while the ICER may have been adjusted during the appraisal process, the base case ICER reflected a common starting point from which to compare appraisals (being the most consistently reported value). Appraisals accepted in-line with the submission indication were classified as accepted. Appraisals accepted in a restricted population or for which agency-defined restrictions were placed were classified as restricted. Appraisals that were not recommended were classified as rejected.

Where multiple base case ICERs were submitted, the ICER was selected based on the proposed indication of the drug. When submissions included patient access schemes, ICERs both with and without the patient access schemes were extracted (Figure 1).

Figure 1

Summary of methods.

Abbreviations: BNF, British National Formulary; CADTH, Canadian Agency for Drugs and Technologies in Health; HTA, health technology assessment; ICER, incremental cost-effectiveness ratio; MTA, multiple technology assessment; NICE, National Institute for Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; SMC, Scottish Medicines Consortium.

Disease area analysis

Submissions were classified by disease area based on British National Formulary (BNF) categories. To determine whether differences in acceptance rates varied by disease area, a funnel plot was used with limits ±3 standard error (SE) of the mean. This method allows for differences in sample sizes between categories. A chi-square test was also carried out to test any potential significant differences.

Results

Overall acceptance trends

A total of 679 submissions were identified that met the inclusion criteria between January 2000 and May 2014: 114 from NICE, 310 from SMC, 68 from CADTH, and 187 from PBAC.

NICE recommended the highest proportion of submissions overall, with an acceptance rate of 76% (87/114); 66 of these were accepted unconditionally and 21 with restrictions. SMC recommended 52% (162/310) of submissions, 94 without restrictions and 68 with restrictions. CADTH recommended 53% (36/68) of submissions, though just 4 of these were accepted without restrictions; the remaining 32 were restricted. PBAC recommended the lowest proportion of submissions (42% [79/187]), 73 of which were accepted without restrictions and 6 with restrictions (Table 1).

Table 1

Acceptance rates for total submissions and for submissions with ICERs higher than the threshold (N=679)

	NICE	SMC	CADTH	PBAC
Total submissions, N	114	310	68	187
Recommended, n (%)	66 (58)	94 (30)	4 (6)	73 (39)
Restricted, n (%)	21 (18)	68 (22)	32 (47)	6 (3)
Rejected, n (%)	27 (24)	148 (48)	32 (47)	108 (58)
Submissions with ICERs higher than the threshold, N	41	70	27	83
Recommended, n (%)	14 (34)	8 (11)	0 (0)	13 (16)
Restricted, n (%)	8 (20)	10 (14)	7 (26)	3 (4)
Rejected, n (%)	19 (46)	52 (74)	20 (74)	67 (81)
Submissions with ICERs lower than the threshold, N	73	240	41	104
Recommended, n (%)	52 (71)	86 (36)	4 (10)	60 (58)
Restricted, n (%)	13 (18)	58 (24)	25 (61)	3 (3)
Rejected, n (%)	8 (11)	96 (40)	12 (29)	41 (39)
P-value (acceptance of ICERs above the threshold vs ICERs below the threshold)	<0.0001(χ2=18.18; df =1)	<0.0001(χ2=25.53; df =1)	0.00029(χ2=13.12; df =1)	<0.0001(χ2=32.27; df =1)

Abbreviations: ICERs, incremental cost-effectiveness ratios; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; CADTH, Canadian Agency for Drugs and Technologies in Health; PBAC, Pharmaceutical Benefits Advisory Committee; df, degrees of freedom; χ2, chi-square value.

The influence of ICER thresholds on acceptance rates was then examined. Across all agencies, acceptance rates were statistically significantly lower for submissions with ICERs higher than the threshold than for submissions with ICERs lower than the threshold (P<0.0005 for all agencies). Nevertheless, a considerable proportion of submissions were accepted with ICERs higher than the threshold, especially in the case of NICE (Table 1). A list of all submissions accepted with ICERs higher than the threshold can be found in Table S1.

Rationale for acceptance of submissions reporting ICERs higher than the threshold

A total of 28/63 (44%) accepted submissions with ICERs higher than the threshold had restrictions imposed on them (8 for NICE; 10 for SMC; 7 for CADTH; and 3 for PBAC), mostly by limiting the eligible patient population. In submissions that were accepted unconditionally, a convincing argument was required to justify the added costs, and patient access or risk-sharing schemes were often agreed to bring costs down. In other cases, drivers for acceptance included addressing an unmet therapeutic need, demonstrating a clear and certain clinical benefit over the current standard of care, and meeting end-of-life or orphan drug criteria (Table S1). In the case of NICE, innovation was also a key consideration for some submissions.

The effect of therapy area on appraisal outcome

Acceptance rates by disease area were assessed to determine whether submissions in some disease areas were more likely to be accepted than others. ICER thresholds were also applied to these findings, to explore whether ICER thresholds vary by disease area (Table S2).

NICE

Interventions in the malignant disease and immunosuppression disease category (specifically oncologics) accounted for the majority of NICE submissions, with an acceptance rate slightly lower than the average across all disease areas. Over half of submissions in the malignant disease and immunosuppression category reported ICERs higher than NICE’s £30,000 cost-effectiveness threshold (32/56; 57.1%). Technologies in the cardiovascular system category had the highest overall acceptance rate, with 13/14 (92.9%) submissions accepted without restrictions. However, all of these submissions reported ICERs in line with the threshold (Figure 2A).

Figure 2

HTA submission outcomes by disease area, for total submissions (top, darker bars) and submissions with ICERs higher than the threshold (bottom, lower bars).

Notes: The lower bar in each disease category represents outcomes for submissions with ICERs higher than the agency’s cost-effectiveness threshold ((A) NICE: £30,000; (B) SMC: £30,000; (C) CADTH: CAN$50,000; (D) PBAC: AUS$42,000). Note that the malignant disease and immunosuppression category includes cytotoxic drugs, drugs affecting the immune response, and sex hormones/hormone antagonists in malignant disease.

Abbreviations: ICER, incremental cost-effectiveness ratio; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; CADTH, Canadian Agency for Drugs and Technologies in Health; PBAC, Pharmaceutical Benefits Advisory Committee.

SMC

As for NICE, malignant disease and immunosuppression accounted for the majority of SMC submissions overall as well as submissions with ICERs higher than the £30,000 cost-effectiveness threshold (Figure 2B). However, the rejection rate was higher than for NICE, regardless of disease area or ICER threshold compliance.

CADTH

CADTH does not typically assess oncologics, which was reflected in the low number of submissions in the malignant disease and immunosuppression disease category. Notable for CADTH was the high proportion of restricted submissions regardless of disease area; few submissions were accepted unconditionally (Figure 2C).

PBAC

Following the trend for NICE and SMC, the majority of PBAC submissions (overall as well as submissions with ICERs higher than the AUS$42,000 cost-effectiveness threshold) fell into the malignant disease and immunosuppression category. There were no notable disease-area specific variations in acceptance or rejection rates (Figure 2D).

Pooled analysis

Finally, the results across all agencies were pooled to determine whether submissions in some disease areas were more likely to be accepted with above-threshold ICERs than in other disease areas.

Based on the outputs of a funnel plot with limits ±3 SE of the mean, there was no significant difference in acceptance rates between disease areas compared with the average acceptance rate of 29%, for submissions with ICERs higher than the threshold (Figure 3). The funnel plot shows decisions for each BNF category plotted with the number of decisions above the standard threshold across the x-axis and the percentage successful on the y-axis. Confidence intervals ±3 SEs of the mean (∼99% confidence interval) are plotted to help the reader see the variability that might be expected by chance. The fact that the rates we observe fall within these lines indicates that the results are unsurprising.

Figure 3

Funnel plot demonstrating difference in acceptance rates for submissions with ICERs higher than the threshold for different disease areas.

Note: Funnel plot with illustrative 95% confidence intervals.

Abbreviations: CNS, central nervous system; ICER, incremental cost-effectiveness ratio; O and G and UTDs, obstetrics, gynecology, and urinary tract disorders.

A chi-square test was also carried out on this data set to test any differences in acceptance rates by disease area. The chi-square test of these data produced a P-value of 0.967 (χ2=4.093, df =11) (Table S3); therefore, the hypothesis that differences in the proportion of decisions accepted above the threshold are purely due to sampling error cannot be rejected.

Discussion

The aim of this study was to evaluate the prevalence of submissions to different HTA agencies that are accepted with ICERs higher than the threshold and investigate any variation by disease area. We found that, overall, over a quarter of submissions with ICERs higher than the threshold were given a positive reimbursement submission; however, 42% of these submissions were restricted in their use within the relevant health system. In cases where submissions were accepted unconditionally, a strong evidence base and convincing argument was required to warrant a positive decision. A key finding was that a large number of submissions with ICERs higher than the threshold fell into the malignant disease and immunosuppression category; however, submissions in this disease category were not more likely to be accepted than submissions in other disease areas (based on the outputs of the funnel plot and chi-square test).

Why are submissions accepted with ICERs higher than the cost-effectiveness threshold?

The concept of the ICER puts in place a threshold for cost-effectiveness, which is necessary to ensure that adequate health gains can be obtained within the fixed budget of a health care system.1,8 For new technologies to effectively bypass this system by being accepted with higher ICERs, it can be assumed that there are additional criteria in place.

Some HTA agencies focus only on clinical effectiveness, and do not consider cost-effectiveness.10 While the agencies included in this analysis all use cost-effectiveness criteria to inform decision-making, in-depth review of individual submissions suggests that demonstration of clinical benefit can be a contributing factor to the acceptance of submissions for which standard cost-effectiveness criteria are not met.11–13 Examples of a strong clinical evidence base in such cases include positive head-to-head trial evidence against a relevant comparator or adjusted indirect comparison with low levels of uncertainty and bias, and/or a clearly defined improvement in treatment effect.8 Previous research has also suggested that the number of randomized trials can strengthen the clinical evidence from the perspective of HTA agencies.14

Additionally, unmet therapeutic needs and lack of alternative options can override ICER thresholds. Under the value-based pricing framework proposed in the UK, assessments would be formally weighted by criteria, including level of unmet need and the wider societal benefit.15 Considering that value-based pricing proposals have been heavily criticized and stalled,16–18 it is interesting to note that these criteria are already informally incorporated into the decision-making process in several markets.

Another priority is orphan drug designation. Due to the need of manufacturers to recover development costs in indications with small patient numbers, orphan drugs tend to be more expensive than therapies in more prevalent disease areas.5 As a result, orphan drugs do not usually meet standard cost-effectiveness criteria, and reported ICERs in manufacturers’ reimbursement submissions are often higher than the market’s willingness-to-pay (WTP) threshold.5,6 To mitigate these challenges, HTA agencies may consider higher WTP thresholds for orphan drugs, though no formal threshold or process exists.19 The results of our research support the theory that orphan drugs may have higher WTP thresholds, as several submissions with higher-than-threshold ICERs were accepted on the basis of meeting orphan drug criteria. Further research is warranted to characterize the potential ICER allowance for orphan drugs.

The wider societal view in respect to end-of-life therapies should also be considered. It is recognized that society values the humanistic appeal of life-extending therapies, and NICE has published specific guidance on appraising end-of-life drugs.7 Although end-of-life therapies were not considered in this analysis as such, many of these would have fallen into the malignant disease and immunosuppression disease category. Interestingly, while a large number of these submissions reported ICERs higher than the threshold, the proportion of accepted submissions in this category did not differ significantly to that in other disease areas, based on the outputs of the funnel plot and chi-square test. These findings do not support the theory that end-of-life drugs have higher ICER thresholds; however, not all cancer drugs are considered end-of-life therapies, and these criteria can be met by drugs in other indications. Therefore, further research into the effect of end-of-life criteria on ICER thresholds would be informative in this respect.

Why are acceptance rates variable between HTA agencies?

Acceptance of above-threshold ICER technologies varies from country to country, highlighting the differing approaches each country takes to adopting new technologies.

NICE accepted the highest proportion of submissions with ICERs higher than the threshold out of all the HTA agencies. This is likely to be due to the selective nature of NICE appraisals,20 with horizon scanning potentially acting as an early bottleneck for technologies unlikely to be accepted. Other HTA agencies adopt the approach of evaluating a larger range of potential technologies, with the result that the rejection rate is higher. Another potential reason for the high rate of recorded acceptance is that NICE considers a wider range of criteria than other agencies. For example, NICE recognizes the innovative nature of technologies and makes allowances for end-of-life drugs, whereas other HTA agencies may not.8 It has also been suggested that patient group submissions, the number of randomized trials, and the number of systematic reviews supporting NICE submissions can positively affect the appraisal outcome.14 A further reason for the higher acceptance rate may be the fact that NICE’s formal cost-effectiveness threshold is in fact lower than the threshold applied in practice; previous modeling of NICE decisions has suggested an ICER threshold closer to the range of £35,000–£40,000.21

The acceptance rate for SMC was lower than for NICE, both overall and for submissions with ICERs higher than the threshold. This is likely to partly reflect differences in the volume of drugs being appraised, as SMC considers all health technologies for appraisal while NICE considers only a selection.20,22 Nevertheless, the fact that SMC approves a smaller proportion of technologies with ICERs above £30,000 than NICE does implies a lower willingness-to-pay for such interventions. This should be considered by manufacturers when seeking reimbursement in the UK.

CADTH did not unconditionally accept any submissions with ICERs higher than the threshold, instead placing restrictions on all such submissions that gained a positive reimbursement decision. The reason for the high rate of restriction remains unclear, though CADTH also restricts a high proportion of submissions overall. Previous research in this area has highlighted clinical and economic uncertainty, as well as nonrelevant clinical endpoints as negative drivers of CADTH decisions.23 In addition, CADTH does not tend to assess oncologics, which are instead evaluated under the pan-Canadian Oncology Drug Review, and this may have affected the recorded acceptance rates.24

PBAC rejected the highest proportion of higher-than-threshold ICER submissions overall, but accepted more drugs unconditionally than CADTH or SMC. It has been suggested that clinical significance, clinical and economic uncertainty, relevance of endpoints, indication (specifically, drugs for life-threatening conditions), and cost to government (in terms of financial impact) can affect the likelihood of acceptance of PBAC submissions, in addition to cost-effectiveness criteria,23,25–27 which may influence the decision-making process.

Strengths, limitations, and scope for further research

This study incorporated the decisions made by multiple HTA agencies in the analysis, which allowed for direct comparison of reimbursement trends across different markets. Further investigation across additional HTA agencies could help expand the context of the results detailed here. Similarly, more detailed comparisons between HTA agencies, perhaps in relation to individual products, would yield more information on some of the nuances of HTA decision-making when it comes to above-threshold ICER submissions.

The limitations of the analysis primarily surround uncertainty in the definition of the ICER threshold. Different ICER thresholds for orphan drugs and therapies meeting end of life criteria were not considered; although such drugs may have higher WTP thresholds,19 these are neither defined in the literature nor by HTA agencies. In addition, reporting of multiple ICERs for some submissions made selection of the appropriate base-case ICER difficult in some cases. Further, the “ICER threshold” itself is not a concrete concept; in reality, ranges are more likely to be applied.21 For example, NICE formally uses a threshold range of £20,000–£30,000 per QALY,8 (and in practice, this threshold range may be even higher21) and formal thresholds for other agencies are not usually published. The results are therefore sensitive to the assumed ICER thresholds. In light of this, reasons for differences between HTA acceptance rates must be contextualized. The application of different assumed ICER thresholds to the analysis would certainly affect the results presented here, and this limits the potential for objective comparisons between HTA agencies.

The rationale for acceptance in the submissions extracted was not quantified, as it proved difficult to objectively categorize decision drivers. In reality, a wide range and combination of reasons will contribute to the overall appraisal outcome, which will be specific to the treatment landscape of the technology being appraised as well as agency-specific criteria and policies. Further, HTA agencies vary in the level of detail included in their publically available summary documents for each appraisal, and so extraction of such information is a subjective exercise that is unlikely to capture all of the decision drivers behind acceptance. Nevertheless, such information would be informative to manufacturers seeking acceptance of technologies with high ICERs in order to best demonstrate their value, and therefore further research beyond the exploratory analysis detailed in Table S1 is warranted.

No significant differences in acceptance rates of submissions with ICERs higher than the threshold by disease area were observed in the pooled analysis, based on the funnel plot and chi-square test output. This outcome may have been skewed by the large number of submissions with higher-than-threshold ICERs in the malignant disease and immunosuppression category, compared with other disease categories where low numbers of submissions were recorded. We cannot prove that there is no difference present, just that it is not sufficiently large to be considered surprising. There are, however, marked differences in the number of decisions across BNF categories, and some of the cells in the chi-square analysis had a small number of decisions in them. Nevertheless, the results do appear to be robust to distributional assumptions, and the expected values are very close indeed to those observed. More granular analysis or use of a disease classification method other than BNF categories could help to further inform differences in disease area-specific trends.

Conclusion

While the submitted ICER is the most influential factor in the HTA decision-making process in the four HTA agencies evaluated, additional criteria are important in determining the success or failure of a submission. Differing criteria of HTA bodies should be considered, as well as the nature of the technology being appraised. Nevertheless, submissions reporting a higher-than-threshold base-case ICER must be supported by transparent, robust, and concrete evidence in order to achieve patient access.

Table S1

List of all submissions that were accepted with ICERs higher than the threshold, by HTA agency

Drug	Indication	BNF Category	Manufacturer’s base-case ICER	Nature of acceptance	Key rationale provided by HTA bodya
					PAS/risk sharing schemeb	Unmet need/lack of alternative therapeutic options	High clinical benefitc	Orphan/rare disease	End of life considerations
NICE (£)
Abiraterone + prednisolone	Castration-resistant metastatic prostate cancer previously treated with one docetaxel-containing regimen	Malignant disease and immunosuppression	52,851	Accepted	X				X
Adalimumab	Plaque psoriasis	Skin	30,500	Restricted
Azacitidine	Myelodysplasic syndromes, chronic myelomonocytic leukemia and acute myeloid leukemia	Malignant disease and immunosuppression	47,432	Accepted	X		X		X
Bortezomib	Multiple myeloma	Malignant disease and immunosuppression	38,000	Accepted	X	X	X
Certolizumab	Rheumatoid arthritis	Musculoskeletal and joint diseases	34,930	Restricted	X
Fingolimod	Highly active relapsing-remitting multiple sclerosis	Malignant disease and immunosuppression	55,634	Accepted	X		X
Gefitinib	Locally advanced or metastatic non-small-cell lung cancer	Malignant disease and immunosuppression	35,992	Accepted	X
Golimumab +methotrexate	Rheumatoid arthritis	Musculoskeletal and joint diseases	31,464	Restricted	X
Ipilimumab	Previously treated advanced (unresectable or metastatic) melanoma	Malignant disease and immunosuppression	60,737	Accepted	X				X
Mannitol dry powder	Cystic fibrosis	Respiratory system	47,095	Restricted		X	X
Mifamurtide	High-grade resectable nonmetastatic osteosarcoma in children, adolescents and young adults	Malignant disease and immunosuppression	56,683	Accepted	X	X	X
Natalizumab	Multiple sclerosis	Malignant disease and immunosuppression	44,600	Restricted		X
Pazopanib	Metastatic renal cell carcinoma	Malignant disease and immunosuppression	32,898	Accepted	X		X		X
Pemetrexed	Non-small-cell lung cancer (maintenance)	Malignant disease and immunosuppression	33,065	Accepted			X		X
Pemetrexed + cisplatin	Locally advanced or metastatic non-small-cell lung cancer	Malignant disease and immunosuppression	33,732	Accepted			X
Pirfenidone	Idiopathic pulmonary fibrosis	Musculoskeletal and joint diseases	36,327	Restricted	X
Ranibizumab	Visual impairment due to diabetic macular edema	Eye	30,277	Restricted	X		X
Sunitinib	Unresectable and/or metastatic malignant gastrointestinal stromal tumors	Malignant disease and immunosuppression	47,628	Accepted	X				X
Trabectedin	Advanced soft tissue sarcoma	Malignant disease and immunosuppression	56,985	Accepted	X				X
Trastuzumab + cisplatin and capecitabine or 5-FU	Gastric cancer (HER2-positive, metastatic)	Malignant disease and immunosuppression	51,927	Restricted					X
Ustekinumab	Plaque psoriasis	Skin	30,664	Accepted	X
Vemurafenib	Locally advanced or metastatic BRAF V600 mutation-positive malignant melanoma	Malignant disease and immunosuppression	56,410	Accepted	X	X	X		X
SMC (£)
Axitinib	Advanced renal cell carcinoma	Malignant disease and immunosuppression	33,837	Accepted	X	X	X
Carglumic acid	Hyperammonaemia	Nutrition and blood	81,000	Restricted		X	X	X
Crizotinib	Previously treated ALK-positive advanced non-small-cell lung cancer	Malignant disease and immunosuppression	42,295	Accepted	X	X	X
Dasatinib	Leukemia	Malignant disease and immunosuppression	44,456	Restricted			X
Dimethyl fumarate	Relapsing-remitting multiple sclerosis	Malignant disease and immunosuppression	73,724	Accepted	X	X	X
Elvitegravir + cobicistat + emtricitabine + tenofovir disoproxil (as fumarate)	Treatment-naïve HIV	Infections	1,332,746	Accepted	X
Infliximab	Ulcerative colitis in children and adolescents	Musculoskeletal and joint diseases	35,000	Restricted		X	X
Lenalidomide	Transfusion-dependent anemia	Malignant disease and immunosuppression	57,930	Accepted		X	X	X
Mecasermin	Severe primary insulin-like growth factor-l deficiency	Endocrine System	47,729	Accepted		X	X	X
Methylnaltrexone bromide	Opioid-induced constipation	Gastrointestinal system	32,284	Restricted			X
Nelarabine	T-cell acute lymphoblastic leukemia/lymphoma	Malignant disease and immunosuppression	56,107	Restricted			X	X
Parathyroid hormone	Severe osteoporosis in postmenopausal women	Endocrine System	107,000	Restricted			X
Pazopanib	Advanced renal cell carcinoma	Malignant disease and immunosuppression	62,414	Accepted	X
Pemetrexed	Malignant pleural mesothelioma	Malignant disease and immunosuppression	35,600	Restricted		X	X
Plerixafor	Lymphoma and multiple myeloma	Nutrition and blood	39,649	Accepted			X	X
Tacrolimus	Prophylaxis of transplant rejection in heart allograft recipients	Malignant disease and immunosuppression	35,000	Restricted
Ustekinumab	Plaque psoriasis	Skin	97,063	Restricted	X
Vemurafenib	BRAF V600 mutation-positive unresectable or metastatic melanoma	Malignant disease and immunosuppression	39,617	Restricted	X	X	X
CADTH (CAN$)
Erlotinib	Non-small-cell lung cancer	Malignant disease and immunosuppression	71,000	Restricted			X
Tipranavir	HIV infection	Infections	52,000	Restricted			X
Sunitinib	Gastrointestinal stromal tumor	Malignant disease and immunosuppression	80,000	Restricted			X
Deferasirox	Iron overload	Nutrition and blood	67,595	Restricted		X
Rufinamide	Lennox-Gastaut syndrome; adjunctive treatment of seizures	Central nervous system	55,715	Restricted		X	X
Asenapine	Bipolar I disorder	Central nervous system	72,623	Restricted		X
Tocilizumab	Juvenile idiopathic arthritis	Musculoskeletal and joint diseases	69,845	Restricted			X
PBAC(AUS$)
Afatinib	Locally advanced or metastatic non-small-cell lung cancer in patients with EGFR gene mutation(s)	Malignant disease and immunosuppression	45,000–75,000	Accepted	X
Bevacizumab	Metastatic colorectal cancer	Malignant disease and immunosuppression	45,000–75,000	Accepted	X		X
Carmustine	Malignant glioma	Malignant disease and immunosuppression	105,000–200,000	Restricted	X
Dabrafenib	BRAF V600 mutation positive melanoma	Malignant disease and immunosuppression	45,000–75,000	Accepted	X	X	X
Erlotinib	Locally advanced or metastatic non-squamous non-small-cell lung cancer in patients with evidence of activating EGFR mutation(s)	Malignant disease and immunosuppression	45,000–75,000	Accepted	X		X
Everolimus	Postmenopausal women with hormone-receptor positive, HER2 negative advanced breast cancer after failure of letrozole or anastrozole	Malignant disease and immunosuppression	45,000–75,000	Accepted	X
Everolimus	Subependymal giant cell astrocytoma	Malignant disease and immunosuppression	45,000–75,000	Accepted	X	X	X
Infliximab	Severe plaque psoriasis	Skin	45,000–75,000	Restricted			X
Pazopanib	Advanced (unresectable and/or metastatic) soft tissue sarcoma	Malignant disease and immunosuppression	45,000–75,000	Accepted		X
Raltegravir	HIV infection in antiretroviral experienced patients	Infections	45,000–75,000	Accepted			X
Somatropin	Prader-Willi Syndrome	Endocrine System	45,000–75,000	Accepted			X
Sunitinib	Malignant pancreatic neuroendocrine tumor	Malignant disease and immunosuppression	45,000–75,000	Accepted	X	X		X
Sunitinib malate	Gastrointestinal stromal tumor	Malignant disease and immunosuppression	45,000–75,000	Accepted	X	X
Sunitinib malate	Advanced/metastatic renal cell carcinoma	Malignant disease and immunosuppression	45,000–75,000	Accepted	X	X
Tamsulosin hydrochloride	Lower urinary tract symptoms associated with benign prostatic hyperplasia	Obstetrics, gynecology, and urinary tract disorders	45,000–75,000	Restricted	X
Verteporfin	Subfoveal choroidal neovascularisation	Eye	45,000–75,000	Accepted			X

Notes:

Reflects information available in the publically available summary document only. Additional criteria not stated in this document may also have contributed to acceptance;

includes discounts, patient-access schemes, price capping agreements, and outcomes-linked schemes;

covers improved efficacy improved safety, and/or improved patient quality of life.

Abbreviations: ICERs, incremental cost-effectiveness ratios; HTA, health technology assessment; BNF, British National Formulary; PAS, patient access scheme; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; CADTH, Canadian Agency for Drugs and Technologies in Health; PBAC, Pharmaceutical Benefits Advisory Committee.

Table S2

Acceptance of submissions with ICERs higher than the threshold by disease area

	NICEn (%)	SMCn (%)	CADTHn (%)	PBACn (%)	Pooled results n (%)
Total	22/41 (54)	18/70 (26)	7/27 (26)	16/83 (19)	64/223 (29)
1. Gastrointestinal system	–	1/4 (25)	0/1 (0)	0/1 (0)	1/6 (17)
2. Cardiovascular system	–	–	–	0/1 (0)	0/1 (0)
3. Respiratory system	1/1 (100)	0/1 (0)	–	0/3 (0)	1/5 (20)
4. Central nervous system	–	0/2 (0)	2/3 (67)	0/2 (0)	2/7 (29)
5. Infections	–	1/5 (20)	1/5 (20)	1/4 (25)	3/14 (21)
6. Endocrine system	–	2/3 (67)	0/3 (0)	1/5 (20)	3/11 (27)
7. Obstetrics, gynecology, and urinary tract disorders	–	–	–	1/3 (33)	1/3 (33)
8. Malignant disease and immunosuppression	14/32 (44)	10/45 (22)	2/6 (33)	11/47 (23)	37/131 (28)
9. Nutrition and blood	0/1 (0)	2/5 (40)	1/3 (33)	0/5 (0)	3/14 (21)
10. Musculoskeletal and joint diseases	4/4 (100)	1/3 (33)	1/4 (25)	0/3 (0)	6/16 (38)
11. Eye	1/1 (100)	0/1 (0)	0/2 (0)	1/2 (50)	2/6 (33)
12. Ear, nose, and oropharynx	–	–	–	–	–
13. Skin	2/2 (100)	1/1 (100)	–	1/5 (20)	4/8 (50)
14. Immunological products and vaccines	–	–	–	–	–
15. Anesthesia	–	–	–	–	–

Notes: That acceptance here includes restricted submissions; – indicates not applicable.

Abbreviations: ICERs, incremental cost-effectiveness ratios; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; CADTH, Canadian Agency for Drugs and Technologies in Health; PBAC, Pharmaceutical Benefits Advisory Committee.

Table S3

Chi-square test “Is there evidence that acceptance rates above the accepted thresholds are higher for some therapeutic areas than for others?”

	Number accepted	Number rejected	Total	Expected accepted	Expected rejected	Chi squared accepted	Chi squared rejected
Malignant disease and immunosuppression	37	94	131	37.00896861	93.99103139	2.17×10–6	8.56×10–7
Musculoskeletal and joint diseases	6	10	16	4.520179372	11.47982063	0.484465	0.020055
Infections	3	11	14	3.955156951	10.04484305	0.230667	0.090825
Nutrition and blood	3	11	14	3.955156951	10.04484305	0.230667	0.090825
Endocrine system	3	8	11	3.107623318	7.892376682	0.003727	0.001468
Skin	4	4	8	2.260089686	5.739910314	1.339455	0.52741
Central nervous system	2	5	7	1.977578475	5.022421525	0.000254	0.0001
Gastrointestinal system	1	5	6	1.695067265	4.304932735	0.285014	0.112224
Eye	2	4	6	1.695067265	4.304932735	0.054856	0.021599
Respiratory system	1	4	5	1.412556054	3.587443946	0.120493	0.047444
Obstetrics, gynecology, and urinary tract disorders	1	2	3	0.847533632	2.152466368	0.027428	0.0108
Cardiovascular system	0	1	1	0.282511211	0.717488789	0.282511	0.111239
Chi-square total	4.09353
P-value (χ2 test)	0.967101
P-value (df =11)	0.967101