A Review of Issues Affecting the Efficiency of Decision Making in the NICE Single Technology Appraisal Process.

BACKGROUND: Escalating demands upon the National Institute for Health and Care Excellence's (NICE's) Single Technology Appraisal (STA) programme require a 2.5-times increase upon 2015 capacity by 2020. This additional strain on committee resources threatens to compromise the rigour of the STA process. In 2018, NICE introduced changes to the appraisal process, aiming to expedite final decisions, including consultation opportunities prior to the company's evidence submission, a 'Technical Engagement' stage prior to the first committee meeting, and powers for committee chairs to recommend technologies without a second formal meeting.
OBJECTIVE: This study reviews recent STAs and aims to understand why appraisals require multiple meetings, and whether recent reforms can address the underlying issues.
METHODS: NICE STAs published between January 2010 and January 2018 were reviewed, excluding updates or re-considerations. Data on cost, clinical, and decision-making outcomes from 146 appraisals were extracted and analysed thematically.
RESULTS: Drugs for advanced cancers were least likely to be recommended (28/43 [65.1%] vs 71/74 [96%] for non-cancer) and took longer (2.36 meetings for a final decision vs 1.97 for non-cancer). The academic review increased upon the company's base-case incremental cost-effectiveness ratio by a median of 32.7%. Eighty-four technologies (57.5%) received a negative preliminary recommendation, deferring a final decision by an average of 142 days. Of these, 85.1% were not considered cost-effective. Uncertainty in economic (34.3%) and clinical (22.3%) data also prevented a positive decision. The majority (72.6% [61/84]) of negative preliminary decisions were overturned following further committee discussion; important considerations were Patient Access Schemes, decision optimisation, and the Cancer Drugs Fund.
CONCLUSIONS: Value considerations are the primary driver of negative preliminary recommendations. It is unclear if new opportunities for additional interaction between NICE, review groups, and manufacturers will meaningfully improve the efficiency of the appraisal process, particularly given the proportion of technologies requiring further committee discussion for decision optimisation or admission into the CDF.

Key Points for Decision Makers

Introduction

The National Institute for Health and Care Excellence’s (NICE’s) Single Technology Appraisal (STA) process is the principal route via which new pharmacological technologies are evaluated for a single indication in the English National Health Service (NHS). This constitutes a formal submission of evidence on the clinical effectiveness and cost-effectiveness of the technology by the manufacturer, which is reviewed, critiqued, and supplemented by an independent Evidence Review Group (ERG). This evidence is then considered by a NICE Technology Appraisal Committee, who provide a judgement on whether the technology should be recommended as an effective use of NHS resources. This may take the form of the Appraisal Consultation Document (ACD), in which the Appraisal Committee sets out its provisional recommendations, or the Final Appraisal Determination (FAD), which is issued after the committee’s final decision is made [1].

The technology appraisal programme has in recent years seen a rapid increase in demand; it is expected that the annual number of appraisals will increase to 75 over the next 2 years, from only 30 in 2015 [2]. NICE has suggested several potential reasons for this trend, principally the introduction of increasingly tissue-agnostic cancer drugs [3], which under current processes may have more than ten indications to be individually appraised [2]. Another challenge is presented by the growing tendency for marketing authorisation to be granted earlier in product development. While this can mean more timely access for patients, it also adds greater complexity and uncertainty to each topic. Given the increasing demands placed upon the appraisal committees, NICE has judged the substantial committee resources currently dedicated to each topic to be inefficient and unsustainable [2]. Each appraisal now requires an average of 2.5 meetings [2] and extensive consultation until a decision can be reached and final guidance produced, meaning an increase of 45 topics could require over 100 further committee meetings.

NICE anticipates that the accommodation of such a capacity increase will require fundamental changes to the appraisal process. There are currently no plans to increase the number of committees or meetings; therefore, as the amount of committee time allocated for the discussion of each topic decreases, it is possible that these meetings will become insufficient to fully consider the case for a decision. NICE have noted that while the majority of topics ultimately receive positive guidance, it is estimated that 60% of draft recommendations are negative [2]; however, the reasons for this are not fully resolved.

In recognition of the unsustainable required increases in resources and committee and ERG time, in April 2018, NICE introduced changes to the STA process aimed at reducing the need for multiple committee meetings. This is further to the earlier introduction of the Fast Track Appraisal process [4], which aimed to reduce time to approval and reduce resource use for highly cost-effective technologies [< £10,000 per quality-adjusted life year (QALY)] with an obvious and significant benefit over existing practice. However, it has been suggested that these changes may not significantly impact the overall efficiency of the process because of the small number of appraisals affected and other unexpected complications which often arise during STAs [5].

In these latest changes, NICE highlighted the increasing need for further evidence during consultation following provisional recommendations, both at the request of committees and companies—particularly when considering whether to recommend funding a technology through the recently reformed Cancer Drugs Fund (CDF). Companies are now invited to discuss their evidence submission earlier in the regulatory process, and with greater guidance from the NICE technical team, who along with the ERG will have more engagement in technical and methodological issues prior to the primary evidence submission [1]. Processes now also allow for a ‘technical engagement’ stage following the completion of the ERG report, in which the views of clinical, patient, and commissioning experts are sought. This offers an opportunity to address outstanding uncertainty in the evidence submission and to resolve key questions raised by the ERG. The company may provide new evidence and analysis prior to the first committee meeting to facilitate more focused discussion. NICE also aims to reduce the use of a second committee meeting when only commercial considerations have changed. The Committee Chair now has the authority to make a final decision in cases where the company provides a sufficiently improved commercial offer in response to the ACD [1].

While the factors addressed in these most recent revisions to the STA process may facilitate cooperation between stakeholders in an appraisal, these issues may only be a symptom of deeper issues in the way manufacturers engage with the appraisal process. A greater understanding of the underlying causes of negative preliminary decisions, and the relationship between disease areas and associated economic considerations could inform more targeted reform of STA processes, and sustainably and efficiently allow the inevitable increases in appraisal throughput, without compromising the rigour of the STA process. This study reviews 8 years of previous STAs to better understand why certain topics require more meetings and resources to reach a final decision, and how far the recent changes to the STA process may go to increasing its efficiency and sustainability.

Methods

Appraisal Selection

Researchers from the University of York and the University of Sheffield compiled a list of all NICE Technology Appraisal guidance for full STAs of any intervention or indication completed between January 2010 and 31 December 2017. STA documents (namely the Committee Papers and FAD) were independently screened by two authors (MW and JOC) against the following eligibility criteria:

Must be a full STA.

Published between 01/01/2010 and 31/12/2017.

Appraisals were excluded if:

They were reconsiderations or updates of previous STAs (including CDF rapid reconsiderations).

The appraisal had been terminated, e.g. due to failure to receive a licence.

Guidance had since been withdrawn.

Twenty-seven further appraisals were excluded after extraction as important information [typically incremental cost-effectiveness ratios (ICERs)] had been redacted or was otherwise missing from appraisal documents. An updated review of NICE’s decisions and use of the CDF was performed in August 2018 and included all appraisals of cancer drugs published between January and July 2018 inclusive.

Data Extraction

Relevant documentation for each appraisal was obtained from the NICE website, including the full Committee Papers and ERG reports, ACDs, and FADs. Data were extracted from these documents by a single reviewer (MW) into a standardised data extraction tool based on that used by Kaltenthaler and colleagues [6]. The data extracted included the following: technology appraisal title and number, disease area, appraisal committee, company and ERG base-case ICERs, appraisal committee–preferred ICER at ACDs and FAD, recommendations at each stage and the reasons behind them, the CDF, and decision optimisation (where a recommendation was restricted to more cost-effective sub-populations). Information about the introduction and progression of Patient Access Scheme (PAS) discounts, the number of appraisal committee meetings, and the time from first committee meeting to the FAD were also extracted. All data were independently checked for accuracy by a second reviewer (JOC) to ensure consistency.

Analysis

Descriptive summary statistics were generated for each of the quantitative items extracted as listed above (e.g. ICERs, number of appraisal committee meetings); these were also calculated for subgroups such as disease area and committee, with mean ICERs by preliminary decision and final decision also calculated. Linear and logistic regression analyses were performed to explore relationships between variables, such as the disease area, company and ERG base-case ICERs, the introduction of PAS discounts, year of appraisal, duration of appraisal, and number of committee meetings, and ACD and FAD decisions. For some analyses, technologies indicated for cancer were further split by whether or not they were indicated for cancers described as advanced, metastatic, or otherwise classed as ‘end-of-life’. All statistical analyses were performed using Stata 15.

A narrative synthesis of the extracted data was performed, which highlighted potentially important patterns and relationships within the dataset.

The factors influencing the likelihood of a negative preliminary recommendation and the reversal of negative decisions in the FAD were investigated through simple descriptive statistics and logistic regression analysis.

Results

A total of 146 STAs conducted between 1 January 2011 and 31 December 2017 met the study inclusion criteria. A further 24 studies were included following a rapid review of decisions on cancer drugs appraised between January and July 2018 inclusive. A flowchart of the selection process is presented in Fig. 1. See the Appendix in the electronic supplementary material for a list of the included STAs.

Fig. 1

PRISMA flow diagram of appraisal selection process. PRISMA preferred reporting items for systematic reviews and meta-analyses, STA single technology appraisal

The main disease areas covered by the STA process were cancer (72/146, 49.3%), cardiovascular conditions (14/146, 9.6%), autoimmune diseases (10/146, 6.8%), and eye conditions (9/146, 6.2%). Overall, 123 (84.2%) of STAs ended with a positive final recommendation; however, a significantly lower proportion (28/43, 65.1%) of drugs for advanced or ‘end-of-life’ cancers received a positive final recommendation (χ2 [2, N = 146] = 19.8, p < 0.001). Despite making up only 29.5% of included STAs, drugs for advanced cancers comprised 65% of all negative recommendations. Non-advanced cancer drugs were approved 82.8% of the time (24/29), while drugs indicated for conditions other than cancer were approved in 96% of cases (71/74). Notably, in the 2 years since the relaunch of the CDF, 81.1% of cancer drugs appraised have received some form of positive recommendation, compared to 63.3% between 2000 and 2016 [7].

The Determinants of Multiple Committee Meetings

An average of 2.16 (95% confidence interval [CI] 2.01–2.32) committee meetings were required to reach a final decision across all disease areas. A final decision was reached in one meeting for only 20.5% of technologies, with the majority (54%) requiring two (see Fig. 2). Cancer drugs required the most meetings (2.36, 95% CI 2.12–2.60), while drugs for all other indications took 1.97 meetings (95% CI 1.79–2.15) to reach a final decision. There were no statistically significant trends over time in the number of committee meetings dedicated to each appraisal, and no other factors were found to be influential.

Fig. 2

Number of committee meetings taken to reach a final decision by disease area

In most cases (84/146, 57.5%), the committee issued a negative preliminary recommendation following the first committee meeting. The implications of a negative preliminary decision are significant, typically delaying the publication of final guidance by a further 142 days. Final guidance is published in an average of 98 days (95% CI 86–111) following a positive preliminary recommendation, increasing to 240 days (95% CI 202–274) after a negative preliminary recommendation. Cancer drug appraisals took 224 days (95% 182–266) from the first committee meeting to publication of final guidance, while this was 148 days (95% CI 120–176) across all other disease areas.

The reasons for a negative preliminary decision can be grouped into three categories, with some decisions citing all three. Most (85.1%, 57/67) technologies were judged not to be a cost-effective use of NHS resources using the committee’s preferred ICER. The committee also often considered the uncertainty of the presented cost (34.3%, 23/67) or clinical effectiveness (22.3%, 15/67) too great to make a reliable decision and decide upon a preferred set of assumptions, citing immature, limited, or inappropriate clinical data which had been inadequately analysed or modelled.

The divergence in cost-effectiveness estimates presented to the committee by the company and ERG was a significant factor in many negative preliminary decisions. At the time of the first committee meeting, the mean base-case ICER as presented by the manufacturer was £31,635 per QALY gained (95% CI 25,947–37,324) across all indications. The mean ICER preferred by the ERG was significantly higher, at £51,644 per QALY gained (95% CI 42,675–60,612). Following the ERG’s review of the company submission, the base-case ICER increased by a median of 32.7% (mean 63%) on the company’s estimate; this was 41.1% for advanced cancer drugs and 22.5% for other types of non-advanced cancer. This excludes instances where a judgement of dominance by the company was not agreed with by the ERG, which means these figures may overestimate the mean company base-case ICER and underestimate the increase in the ERG preferred ICER.

The magnitude of the difference between the company and ERG cost-effectiveness estimates was influential in the preliminary decision; logistic regression analysis indicated that each £1000 difference in the ERG and company base cases was associated with a decrease of 0.04 (95% CI 0.02–0.07) in the odds of a positive preliminary decision, i.e. greater uncertainty in value estimates made a negative decision more likely.

Across all indications, NICE approved drugs at £26,431 per QALY gained on average (95% CI 23,268–29,594). The mean ICER at FAD is presented by indication in Table 1. There were no apparent trends over time in the ICER at which technologies were approved by NICE, and regression analysis adjusting for committee case mix in terms of disease area showed no statistically significant difference in the ICERs at which the different committees approved technologies.

Table 1

Mean ICERs of technologies recommended by NICE

Disease area	No.	FAD ICER (£)	95% CI
Total	123	26,431	23,268–29,594
Non-advanced cancer	24	30,784	24,783–36,785
Advanced cancer	28	41,462	35,811–48,112
Other	71	17,075	14,519–19,631

CI confidence interval, FAD Final Appraisal Determination, ICER incremental cost-effectiveness ratio, NICE National Institute for Health and Care Excellence

Drugs receiving a positive preliminary decision or which went straight to a positive FAD were generally priced well below their respective willingness-to-pay (WTP) thresholds; NICE’s average preferred estimates of cost-effectiveness were £17,010 per QALY gained for non-advanced cancers, £31,123 for advanced cancers, and £8,653 for other indications.

Reversal of Negative Preliminary Decisions

Out of 84 negative provisional decisions, 61 technologies (72.6%) went on to receive a positive final recommendation after further committee discussion. The main reasons for the reversal of a negative preliminary ACD recommendation are presented in Fig. 3, and comprised the following: the introduction or enhancement of a PAS discount (37.7%), approval in more clinically effective or cost-effective subgroups (32.8%); the decision to fund the drug through the CDF, i.e. clinical uncertainty which could reasonably be resolved through a period of data collection (18%); and provision of new clinical evidence or analysis (11.5%).

Fig. 3

Primary reason for reversal of a negative preliminary decision. CDF cancer drugs fund

Technologies indicated for advanced cancers were less likely to have a negative preliminary decision reversed (odds ratio [OR] 0.19, 95% CI 0.067–0.56), with 17 out of 32 drugs (53.1%) receiving a negative ACD recommendation going on to be approved, while this was the case for 15 out of 20 non-advanced cancer drugs (75%) and 29 out of 32 drugs for other indications (90%).

PAS discounts are an important means of improving the likelihood of a positive recommendation. While inclusion of a PAS in the company’s first evidence submission did not affect the likelihood of a positive ACD recommendation, the introduction or enhancement of a PAS following a negative ACD recommendation meant a technology was 4.32 times more likely (95% CI 1.16–16.15) to receive a positive final recommendation, increasing to an OR of 6.4 (95% CI 1.57–26.31) in cancer drugs. The introduction of a PAS following the ACD was associated with a median 38.7% (Interquartile range [IQR] 31.8%, range 0–77.4%) reduction in the FAD ICER based on the ERG’s preferred base case for recommended technologies.

The FAD ICER was in excess of the committee’s preferred WTP threshold in 23 out of 24 of the drugs whose negative ACD recommendation was not reversed, with strength of uncertainty alone responsible for only one negative recommendation at FAD.

The Impact of the Cancer Drugs Fund

Following the introduction of the new CDF in 2016, 19 drugs have been recommended for use within the CDF, a mechanism which allows uncertainty around the efficacy of a technology to be resolved through a period of data collection. Between 2017 and July 2018, 54% (19/35) of all cancer drug recommendations were through the CDF. In 2017, the mean preferred ICER at FAD for drugs approved in advanced cancers was £53,887 (95% CI 46,849–60,925), slightly above the increased threshold for technologies meeting NICE’s criteria for life-extending end-of-life therapies of £50,000. The mean FAD ICER was £41,810 for all other non-advanced cancer drugs funded through the CDF. The mean duration of the committee appraisal process for cancer drugs increased from 208 days in 2015 to 244 days in 2016, and 275 days in 2017.

Discussion

Not unexpectedly, the NICE STA process works most efficiently when appraising technologies with well-characterised efficacy profiles, priced well below NICE’s WTP thresholds. Unfortunately, the distribution of such technologies is inequitable across disease areas, with the greater cost and uncertainty associated with drugs indicated for cancer (particularly advanced forms) leading to slower and more commonly negative final decisions. This review identified the outcome and decision determinants of the first appraisal committee meeting as the best indicator of the obstacles to decision making faced by the appraisal committees, and the evidence provided in the subsequent consultation documents and FAD was used to understand how these were overcome.

The results of this analysis demonstrate that the most consistently cited decision factor in NICE’s negative provisional decisions, and therefore the most significant barrier to rapid decision making is cost-effectiveness; strongly supportive of earlier reviews of NICE’s decision-making processes [8, 9]. This appears to originate from consistent mischaracterisation of a technology’s value and effectiveness by manufacturers, leading to uncertainty or doubt surrounding its cost-effectiveness following independent academic review.

The issues preventing a positive decision cited in the ACDs were satisfactorily resolved in the vast majority of cases, principally through decision optimisation and the introduction or enhancement of commercial offers. The recently introduced changes to the STA processes, namely the technical engagement step, may facilitate earlier communication regarding the committee’s requirements and preferences. If implemented effectively, this could help resolve key issues prior to the first committee meeting and focus discussion on the key issues. If the number of negative preliminary decisions could be reduced, a significant reduction in the time taken to produce final guidance, and thus patient access to treatment, could be achieved.

However, it is unlikely that a resolution to the majority of issues raised by the ERG and NICE would be achieved through the technical engagement step as currently implemented. As a strategic player, manufacturers are unlikely to pre-emptively reduce their price to increase the likelihood of an early approval, or re-focus their submission on a more cost-effective subgroup. At this stage in the appraisal, there is no ‘correct’ cost-effectiveness estimate or set of preferred assumptions, and there is no guarantee that the committee will favour those proposed by the ERG. Therefore, a company has little to lose by maintaining their position and arguing against the validity of the ERG’s analysis, rather than acting against their own interests by way of discounts. Furthermore, it is the prerogative of the committee alone to make considerations on the equitability of decision optimisation—i.e. placing restrictions upon the access to a drug for particular patient groups—and it is natural for companies to seek to market their product to the entire population covered under the licence. Where the committee is minded to approve a technology in more cost-effective subgroups, or where the suitability of a technology for routine commissioning is under consideration (i.e. possible use of the CDF), it is highly likely further meetings will be required regardless of recent process changes.

In 38% of reversed negative preliminary decisions, the basis upon which the committee changed their decision was the introduction or enhancement of a PAS discount, which sufficiently reduced the ICER to below the WTP threshold. In these cases, the new powers of committee chairs to recommend a technology when cost is the only concern may yield a substantial reduction in unnecessary meetings.

Historically, technologies indicated for cancers, and particularly advanced cancers, have been less successful in gaining approval from NICE than those for other indications. The CDF was reformed in 2016 in an attempt to curb the approval of highly cost-ineffective treatments in light of rapidly escalating costs, which stood at £416 million in 2014, an increase of £185 million on the previous year [10]. The CDF is now limited to drugs which have plausible potential to satisfy the criteria for routine commissioning [11], but around which there is significant remaining clinical uncertainty, typically due to immature trial data, which could be resolved through ongoing data collection. However, the application of the concept of ‘plausible potential’ can be difficult to determine, particularly where an expensive treatment’s cost-effectiveness relies upon extrapolated long-term benefits.

As most negative preliminary recommendations due to clinical uncertainty are for cancer drugs, it is likely that the reformed CDF will account for a significant proportion of resolvable clinical uncertainty. Indeed, since its inception, the CDF has been used as a vehicle for over half of all cancer drug recommendations, and appears to be responsible for a marked increase in drug approvals in this period. The CDF is a valuable tool for resolving clinical uncertainty, and accommodates both the protection of NHS resources from potentially ineffective products and the timely delivery of patient access to new technologies. The increased approval rate of drugs earlier in the development cycle is positive for UK patients, where drugs have historically been made available later than in other jurisdictions without Health Technology Assessment bodies such as NICE [12, 13]. However, this system may provide companies with an incentive to withhold evidence to exploit existing clinical uncertainty in order to gain access to the NHS and set prices high, despite unproven long-term cost-effectiveness.

Unless the technical engagement step can identify when evidence is too immature for decision making and advise against routine commissioning at an early stage, rather than following a negative preliminary recommendation and further committee discussion, it is unlikely that this process will significantly reduce the number of required committee meetings. Furthermore, the extent of integration of CDF eligibility assessment with new NICE consultation processes is currently unclear, and indeed may not be possible without wider restructuring of the appraisal processes. Thus, new opportunities for further technical engagement and consultation prior to appraisal committee discussion are not likely to substantially reduce the time to a final decision for cancer drugs.

A number of factors linked to the use of secondary data sources may have limited the accuracy of some analyses featured in this study. A key limitation was the exclusion of dominant ICERs from analyses. Companies often presented a base case in which their technology dominated its competitors, while the ERG’s preferred assumptions produced a numerical ICER, which may have led to the underestimation of ICER changes. Imprecision in the cost-effectiveness estimates reported in NICE’s decision documents was common, with preferred ICERs often presented as a range or described as ‘around’ or ‘below’ the WTP threshold. Inconsistent reporting across NICE and ERG documents may also have led to inappropriately narrow conclusions around the reasons for decisions, as full explanations were rarely presented in the ACD or FAD. This was particularly the case when one factor overshadowed the others, which consequently were not reported.

Conclusions

While it does not appear to be the case that more committee meetings are becoming necessary for each topic, greater uncertainty and complexity around an increasing number of topics is placing an ever-greater burden upon NICE and its resources. Challenging trends in the immaturity of the evidence base around many new technologies mean fundamental issues can take many months to resolve, often requiring further analysis from the company, the ERG, or in some instances, the Decision Support Unit [6]. Commercial negotiations between the company and NHS England also introduce further delays, particularly when there is a large anticipated budget impact. This defers a final decision and ultimately the availability of a new technology to patients. This is apparent in the increases in cancer drug appraisal duration seen here.

The recent changes introduced to the STA protocol and new powers of chairs to approve on price will help alleviate delays and inefficient use of committee time for repetitive cost and commercial discussion. Significant challenges to efficiency and rigour still remain, including uncertainty in value assessment and application of WTP thresholds, decision optimisation, commissioning decisions, and perverse incentives promoting obfuscation and pricing to thresholds. It is important that committees are adequately resourced to enable them to make appropriate, informed, and timely decisions, ensuring the NHS continues to maximise net population health through efficient allocation of funds, and ensure patient access to effective new technologies.

Data Availability Statement

All data included in this study are in the public domain and are available through the NICE website (https://www.nice.org.uk/guidance).

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 21 kb)

National Institute for Health and Care Excellence (NICE) has introduced further opportunities for technical engagement and co-operation with manufacturers during the Single Technology Appraisal process to help the Appraisal Committee make more timely decisions. It remains uncertain if this will have any substantive impact on overall process efficiency.

The vast majority of NICE’s negative decisions stem from an often wide disparity between manufacturer and Evidence Review Group estimates of the value of a new technology.

The introduction or enhancement of a Patient Access Scheme discount and focus on a more clinically effective or cost-effective subgroup were the two most effective ways of overturning a negative preliminary decision.

Over half of all cancer drugs have been approved through the reformed Cancer Drugs Fund in the 18 months from January 2017, resulting in a greater overall proportion of approvals. However, the time it takes to reach a final decision appears to be increasing.