Decision Making Under Uncertainty: Comparing Regulatory and Health Technology Assessment Reviews of Medicines in the United States and Europe.

Assessments of clinical evidence vary between regulators and health technology assessment bodies, but precise differences remain unclear. To compare uncertainties raised on the clinical evidence of approved drugs, we analyzed assessments of regulators and health technology assessment (HTA) bodies in the United States and Europe. We found that US and European regulators report uncertainties related to safety for almost all drugs (85-94%), whereas HTA bodies reported these less (53-59%). By contrast, HTA bodies raised uncertainties related to effects against relevant comparators for almost all drugs (88-100%), whereas this was infrequently addressed by regulators (12-32%). Regulators as well as HTA bodies reported uncertainties related to the patient population for 60-95% of drugs. The patterns of regulator-HTA misalignment were comparable between the United States and Europe. Our results indicate that increased coordination between these complementary organizations is necessary to facilitate the collection of necessary evidence in an efficient and timely manner.

WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

☑ Regulators evaluate the benefit‐risk balance of drugs while health technology assessment (HTA) bodies evaluate drug effects compared with available alternatives. Approved drugs are often rejected for reimbursement by HTA bodies because of uncertainties related to clinical evidence.

WHAT QUESTION DID THIS STUDY ADDRESS?

☑ This study quantified and compared the uncertainties that regulators and HTA bodies raise during their evaluations of clinical evidence.

WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

☑ There is a clear, quantifiable gap in reported uncertainties between regulators and HTA bodies regarding clinical evidence on safety, effects vs. relevant comparators, and clinically relevant, long‐term outcomes. Both regulators and HTA bodies report uncertainties related to the patient population for a majority of assessed drugs. Differences in reported uncertainties between regulators and HTA bodies are similar between the United States and Europe.

HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

☑ Increased coordination between regulators and HTA bodies may facilitate the collection of necessary evidence to resolve the demonstrated gaps in an efficient and timely manner, benefiting patients’ access to drugs.

Regulators must balance the size of clinical benefits of medicines against their harmful effects, taking into account uncertainties in both measures. , , , After regulatory approval, use of a drug will depend on its incremental clinically relevant effects in relation to already available treatments. Use will also depend on the willingness or ability to pay for the drug and coverage by insurers. The process of clinical and economic value assessment is formalized in Europe through evaluations by Health Technology Assessment (HTA) bodies. In the United States, it often relies on assessments of independent clinicians or organizations, or individual insurance plans.

Having to accept some uncertainties at approval is inherent to the limited information on benefits and risks available at the time of marketing authorization, but current trends suggest that drugs are being approved by regulators on the basis of testing in fewer patients and with fewer pivotal trials. , In recent years, over two‐thirds of all trials submitted to the US Food and Drug Administration (FDA) did not have an active comparator, and 20% did not have a comparator at all. Furthermore, 59% of pivotal trials submitted to the FDA used surrogate measures as their primary trial end points. In Europe, only 35% of oncology indications approved by the European Medicines Agency (EMA) had evidence of survival benefits over available treatments. Uncertainties relating to the evidence of newly approved drugs can remain unresolved for multiple years. With a median follow‐up postapproval between 3 and 5.4 years, there was no evidence of survival benefits in 28–58% of approved oncology indications in the United States and Europe. , , , , , Correlation of surrogate measures with survival is often low.

Regulators and HTA bodies have key similarities in their evaluations of new drugs, as they are usually based on the same clinical evidence. However, an important difference is that HTA bodies aim to evaluate effects against relevant comparators, whereas regulators evaluate a single treatment’s benefit‐risk balance, which does not necessarily involve comparison to an active treatment. Additionally, the relevant comparators may differ between subsets of the approved population. This means that HTA bodies may differ in opinion from regulators about which populations will benefit from the treatment. If no active comparator is available, HTA bodies evaluate drug effects against best supportive care, which may be different from the care provided in placebo‐controlled clinical trials. HTA bodies have repeatedly indicated that the evidence regulators find sufficient for establishing a positive benefit‐risk balance can be too limited to accurately establish comparative effectiveness and value. , , , , , , ,

Because most new drugs are now being evaluated by regulators and HTA bodies in the United States and Europe, it is important to know whether—and how—these reviews agree or disagree in their assessment of a drug. We, therefore, sought to investigate and quantify the uncertainties that regulators and HTA bodies raise during their evaluations of clinical evidence.

Methods

Sample construction

To perform a pairwise comparison between regulatory and HTA evaluations in the United States and Europe, we included medicines approved by the FDA and the EMA that were subsequently assessed by HTA bodies in both jurisdictions. In the United States, where there is no national HTA body, the Institute for Clinical and Economic Review (ICER) was used as the HTA body of choice. ICER is an independent organization that conducts HTAs on a selection of high‐impact and high‐profile drugs and drug classes. Included European HTA bodies were the Institute for Quality and Efficiency in Health Care (IQWiG, Germany), the National Institute for Health and Care Excellence (NICE, England and Wales), the National Health Care Institute (ZIN, The Netherlands), as well as the collaborative European network for HTA (EUnetHTA). EUnetHTA is not an HTA body itself but a consortium of HTA bodies that collaborate on, among others, joint assessments. These HTA bodies were selected because they provided comprehensive public reports in languages understood by the investigators (English and Dutch). The final cohort consisted of all EMA and FDA‐approved new drugs (i.e., excluding generics and biosimilars) that were subsequently evaluated by ICER and at least two of the four included European HTA bodies. Reports must have been published between January 1, 1995, and December 31, 2018. Data from reports of European HTA bodies were combined into an aggregated European HTA database (AGGR‐EUR).

Data sources

Although the regulatory and HTA bodies in the study had different evaluation processes, the key document used was the one summarizing the pivotal trials supporting a drug’s clinical utility. Included sections of final reports were all those in which an interpretation of the clinical evidence was provided by the evaluating entity. We excluded sections that did not relate to clinical evidence or that summarized clinical evidence without interpretation, as well as cost‐effectiveness sections. The reports used, including the relevant sections and subsections, are outlined in the Supplementary Table .

Outcome measurements

From the relevant sections of the reports for each drug in the cohort, “uncertainties” related to the clinical evidence were extracted with NVivo 12. An uncertainty was defined as any piece of text that explicitly or implicitly reported an unresolved shortcoming, question, or issue in the clinical evidence. Only unresolved uncertainties were extracted; thus, it would not count as an uncertainty if the report pointed out missing data on a subgroup, but it later turned out that the subgroup was not clinically relevant because the drug was not approved for that subgroup. An example of an explicit uncertainty would be the reporting of insufficient long‐term data to interpret adverse events. An example of an implicit uncertainty would be a statement saying the only reported outcome was an unvalidated laboratory test, as this implies the lack of a clinically relevant outcome. Within the analysis, no distinction was made between implicit or explicit uncertainties. Uncertainties mentioned within the same report on multiple occasions were included once. For some drugs, HTA bodies published a report on a whole drug class or indication rather than per drug. If an uncertainty within those reports was about the drug class or about all drugs within the indication (e.g., a lack of any comparative data), that uncertainty was included for each drug separately.

In addition, basic regulatory characteristics were collected about approval date, expedited approval pathways, special rare disease treatment status (e.g., via the Orphan Drug Act), and evidence considered for evaluation (control arm and end points).

Analysis

Uncertainties were classified into six categories related to safety, efficacy, and effectiveness (Table ). The first category included all uncertainties related to safety. The second category encompassed all uncertainties concerning the validity of the clinical trial results (efficacy). The rest of the categories were dedicated to the generalizability of the outcomes (effectiveness) observed in the assessed trials and related to the relevance of the studied patient population, the applicability of the intervention in practice, the drug’s effects in relation to relevant comparators, and the clinical relevance and long‐term effects of the studied outcomes. If an uncertainty originated from one category (e.g., trial validity) but only applied to a specific other category (e.g., a specific outcome or subgroup), this uncertainty was assigned to the category of its application and not the category of its origin. An example is the effect of crossover on long‐term extension trials. Crossover relates to trial validity, but only affects the specific outcome of long‐term results and was, thus, assigned to the outcomes category. The assignment of all uncertainties to categories was done independently and in duplicate by two investigators (R.A.V. and H.N.; kappa statistic: 0.85), discussing discrepancies until reaching consensus. Statistical significance of differences in the number of reported uncertainties was assessed via t‐tests.

Table 1

Categories of uncertainties and examples of the types of uncertainties that were assigned to each category

Category name	Examples of the types of uncertainties
Safety issues	Safety sample size too small
	Causality of adverse events uninterpretable
	Long‐term safety unclear
Trial validity	Selection bias
	Performance bias
	Detection bias
	Attrition bias
	Reporting bias
Population	Population does not match practice
	Relevant subgroups not adequately studied or reported
Intervention	Unreliable or missing information on interactions with other medication
	Unreliable or missing information on monotherapy or combination regimens
	Unreliable or missing information on appropriate treatment duration
Comparators	Unreliable or missing information on effects against relevant comparators
	Unreliable indirect comparisons
	Unreliable or missing information on appropriate treatment line
Outcomes	Unreliable or missing information on long‐term effects
	Relevant outcomes not measured or reported

After all uncertainties were assigned a category, we analyzed the alignment of regulators and HTA bodies within each jurisdiction (United States/Europe) by comparing per category the percentage of drugs for which both stakeholders reported at least one uncertainty. Uncertainties were extracted for each of the four included European HTA bodies, but for the European comparative analysis of regulator vs. HTA body, all uncertainties of the four institutions were combined and duplicates were removed. This resulted in an AGGR‐EUR. Differences between regulators and HTA bodies were expressed as misalignment per category. Misalignment per category was defined as the percentage of drugs for which one stakeholder reported at least one uncertainty within the category, but the other stakeholder reported none.

Subgroup analyses

Six subgroup analyses were performed on the differences between regulators and HTA bodies. The subgroups included drugs approved by regulators using surrogate measures as primary end points, drugs approved based on pivotal trials without active control arms, drugs approved with special designations for treating rare diseases, drugs approved with at least one expedited testing or regulatory review designation, drugs that were first‐in‐class, and drugs approved before December 2013 (5 years before inclusion end date). Previous research has highlighted the presence of clinical uncertainties in these subgroups. , , , , Expedited designations included fast‐track, accelerated approval, breakthrough status, and priority review for the United States, as well as conditional marketing authorization and accelerated assessment for Europe.

Results

Evaluations were available for 33 drugs, covering 34 indications. Characteristics of the included indications are provided in Table . Over half were antineoplastic or immunomodulatory agents. Almost half benefited from at least one expedited pathway in the United States, whereas only seven benefited from such a pathway in Europe (44% vs. 21%). In the United States, there were 12 drugs (35%) qualifying for fast‐track procedures, 4 (12%) for accelerated approvals, 5 drugs (15%) for breakthrough status, and 12 drugs (35%) for priority review. In Europe, one drug (3%) received a conditional marketing authorization and six drugs (18%) qualified for accelerated assessment. In most cases, the evidence base available for review was similar between regulators, but there were four cases in which the FDA approved drugs earlier than the EMA and with more limited evidence. No European HTA body assessed all 34 indications. The IQWiG (Germany) assessed 26 (76%) indications, NICE (England + Wales) assessed 33 (97%), ZIN (The Netherlands) assessed 18 (53%), and EUnetHTA (Europe) assessed 4 (12%) indications.

Table 2

Basic characteristics of the 34 included indications

Therapeutic class	Number of indications (%)
Alimentary tract and metabolism	2 (6)
Cardiovascular system	3 (9)
Dermatological	1 (3)
Anti‐infective	4 (12)
Antineoplastic or immunomodulatory	20 (59)
Respiratory system	3 (9)
Sensory organs	1 (3)
Subgroup	United States	Europe
Approved with at least one special regulatory designation	15 (44)	7 (21)
Rare disease designation status	7 (21)	2 (6)
Approved on surrogate primary end points	7 (21)	6 (18)
Active control pivotal trial not available	18 (53)	15 (44)
First‐in‐class	17 (50)	17 (50)
Approved > 5 years ago	22 (65)	21 (62)

Number of reported uncertainties

In total, 1,121 uncertainties were identified in assessment reports for all included drugs. Examples of extracted uncertainties for one of the drugs (evolocumab) are presented in Table . When aggregating European HTA organizations, after removal of duplicates, a total of 1,007 uncertainties remained for all institutions. Of those, the FDA raised 149 uncertainties, the ICER 286, EMA 250, and the combined European HTA bodies (AGGR‐EUR) raised 322 uncertainties. At least one uncertainty was reported for each assessed drug by each institution. An average of 7.4 uncertainties (SD 3.8) were raised per drug per institution. Figure shows the distributions of uncertainties per assessed drug for regulators and HTA bodies in the United States and Europe. The FDA raised, on average, 4.4 (SD: 2.2) uncertainties per drug, ICER raised 8.4 (SD: 2.6). The mean difference was 4.2 uncertainties per drug (P < 0.01). The EMA raised an average of 7.4 (SD: 3.6) uncertainties per drug, the aggregated European HTA bodies raised 9.5 (SD: 4.4). The mean difference was 2.1 uncertainties per drug (P = 0.02).

Table 3

Examples of uncertainties extracted from the regulatory and health technology assessment reports for evolocumab

Safety category: The trials were 6 months or less in duration. Serious adverse events may be identified in the large 5‐year outcome trials that are currently in progress

Population category: The overall trial population does not represent a population at high cardiovascular risk with substantial cardiovascular disease burden on maximally tolerated statin therapy, arguably the most appropriate patient population for add‐on therapy to a statin

Comparators category: No valid comparison was available with ezetimibe, evolocumab was only compared with placebo or standard of care

Outcomes category: The RCTs primarily measured surrogate end points (such as LDL‐C) and were not powered to measure cardiovascular outcomes, which was considered to be an important limitation of the evidence base

LDL‐C, low‐density lipoprotein cholesterol; RCTs, randomized controlled trials.

Figure 1

Distribution of the number of uncertainties raised per drug as reported by the United States and European regulators and health technology assessment bodies. The horizontal lines indicate the medians and the boxes indicate the interquartile range. Single points indicate outliers. AGGR‐EUR, aggregated European HTA database; EMA, European Medicines Agency; FDA, US Food and Drug Administration; ICER, Institute for Clinical and Economic Review.

Categories of reported uncertainties

Figure shows the percentage of drugs by category for which at least one uncertainty was raised (solid bars). Additionally, Figure indicates the percentage of drugs within the respective categories for which only one stakeholder (either regulator or HTA body) within a jurisdiction raised uncertainties and the other stakeholder did not (translucent bars).

Figure 2

Percentage of drugs for which stakeholder raised uncertainties, by category. Blue is for regulators, orange for health technology assessment (HTA) bodies. Solid bars indicate drugs for which both stakeholders raised uncertainties within that category, translucent bars indicate drugs for which only one stakeholder raised uncertainties (e.g., blue translucent bars indicate the percentage of indications for which regulators reported uncertainties but HTA bodies did not). AGGR‐EUR, aggregated European HTA database; EMA, European Medicines Agency; FDA, US Food and Drug Administration. [Colour figure can be viewed at wileyonlinelibrary.com]

Safety issues—such as those related to sample size or uncertainties in causality—were raised by regulators for almost all drugs assessed (94% for the FDA and 85% for the EMA). HTA bodies raised safety issues for only 59% (ICER) and 53% (AGGR‐EUR) of drugs. Thus, for 35% (United States) and 32% (Europe) of drugs, regulators, and HTA bodies were not aligned with respect to reporting uncertainties related to safety.

HTA bodies raised uncertainties related to effects against relevant comparators for almost all drugs (100% in the United States and 88% in Europe), whereas this category was hardly addressed by the FDA (12%) and only slightly more by the EMA (32%). Misalignment between regulators and HTA bodies was present for 88% (United States) and 56% (Europe) of indications, respectively. Uncertainties related to the relevance of the end points considered and long‐term effects were also raised for more drugs by HTA bodies than by regulators (82% and 91% for ICER and AGGR‐EUR, respectively, and 41% and 62% for the FDA and EMA, respectively). There was misalignment in 41% (United States) and 29% (Europe) of indications.

The percentages of indications for which uncertainties were raised related to the categories of patient population and intervention applicability were similar between regulators and HTA bodies. The percentage of indications with raised uncertainties related to trial validity was similar between the FDA and ICER but differed more between the EMA and the aggregated European HTA bodies: Misalignment was present for 12% of indications in the United States and for 32% of indications in Europe. All stakeholders raised uncertainties related to the patient population for > 60% of assessed drugs.

Overall, the patterns of differences between regulators and HTA bodies were very similar between the United States and Europe.

Figure shows the percentage of total categories throughout all indications in which uncertainties were raised by each stakeholder overall and in the predefined subgroups. The FDA reported uncertainties in 40% of all categories and ICER in 62%. The EMA reported uncertainties in 56% of categories and the European HTA bodies in 72%. None of the subgroups showed statistical differences in number of categories with uncertainties that were or were not raised by one of the stakeholders. Some of the subgroups were small (Table ) and the subgroup results were confounded by indication; for example, only two rare disease drugs were included for Europe and all anti‐infective drugs in our sample received priority review (FDA) or accelerated assessment (EMA).

Figure 3

Percentage of categories throughout all indications with raised uncertainties overall and in each of the subgroups. Darker gray is for regulators, lighter gray for health technology assessment bodies. Solid bars indicate the percentage of categories throughout all indications in which both stakeholders raised uncertainties, translucent bars indicate the percentage in which only one stakeholder raised uncertainties. AGGR‐EUR, aggregated European HTA database; EMA, European Medicines Agency; ERP, expedited regulatory pathway; FDA, US Food and Drug Administration; ICER, Institute for Clinical and Economic Review.

Discussion

In this systematic analysis of assessment reports from regulators and HTA bodies in the United States and Europe, many of the uncertainties raised by regulators on safety were not mirrored in HTA processes, whereas regulators did not report the majority of uncertainties raised by HTA bodies related to effects against relevant comparators and long‐term clinically relevant effects. In a majority of indications, both regulators and HTA bodies reported remaining uncertainties related to multiple drug features. The differences between regulators and HTA bodies have similar patterns in the United States and Europe.

Policy implications

The results of this study reflect the different objectives that drive the evaluations of regulators and HTA bodies. Regulators often make a judgment about a drug’s benefit‐risk balance without active‐controlled trials or trials confirming a clinically relevant long‐term effect, particularly if known side effects are relatively mild or can be managed. However, such data naturally provide HTA bodies with little information on incremental benefit over existing treatments. Thus, there is a gap between the evidence that is sufficient for regulators to assess benefits and harms, and the evidence that HTA bodies require to evaluate effects against relevant comparators, which we quantified through the differences in reported uncertainties.

Currently, only regulators have the authority to determine the quality and size of the pre‐approval clinical evidence package deemed appropriate for decision making and any additional studies that are required postapproval. Although previous studies have recognized that postapproval testing commitments recommended by regulators are often changed, delayed, or not fulfilled, it is also true that most of these postapproval obligations are not intended to increase insight into comparative or clinically relevant long‐term effects. , , , , , , , , Although some HTA bodies in Europe may ask for additional studies, most HTA bodies do not have the authority to compel these trials to be completed.

To address the gap in reported uncertainties between regulators and HTA bodies, these groups could coordinate better when setting postapproval testing recommendations. For example, regulators from the EMA have recently indicated their awareness of the fact that a positive benefit‐risk decision does not necessarily translate to relative benefit and have indicated their intent to be more explicit about negative, neutral, or positive added benefit in relevant patient subgroups. A first step for regulators may be to collaborate with HTA bodies and other stakeholders in the development process to start identifying the uncertainties that are relevant to downstream stakeholders and to discuss the necessary evidence and timing of its generation. Joint scientific advice procedures may benefit the generation of evidence that resolves the most important uncertainties for both stakeholders. Previous research has shown that regulators and HTA organizations reached a high level of agreement in evidence requirements during joint scientific advice procedures. Industry tended to implement changes to the development program in terms of primary end points and comparators based on feedback from both stakeholders. , The process of joint scientific advice is now formalized in Europe between the EMA and EUnetHTA and is called parallel consultation.

HTA bodies often did not raise the same uncertainties that regulators reported related to safety. This is interesting because usually HTA bodies have regulatory reports readily available during their evaluation process, and they often refer to findings discussed by regulators. The lack of alignment on uncertainties related to safety suggests that HTA bodies do not systematically re‐assess safety and instead rely on a positive benefit‐harm assessment, or do not find such uncertainties relevant enough to report them. However, for the determination of added benefit, as for the benefit‐risk balance, an inappropriately sized safety database or potentially increased safety risks have important implications. Further study is needed on how HTA bodies assess drug safety and integrate those assessments into their determinations.

There were some similarities in the uncertainties raised by regulators and HTA bodies. Both regulators and HTA bodies raise uncertainties related to the generalizability of the trial population and a lack of reliable information for important subgroups in 65–97% of indications. By definition, the majority of approved drugs have unresolved uncertainties related to how clinical trial effects will translate to patients who will use the drug in practice. All stakeholders, including clinicians and patients, could benefit from a coordinated effort to prospectively define the patient subgroups that need to be studied more extensively postapproval.

Limitations

The inclusion of drugs approved by both regulators and reviewed by ICER and at least two European HTA bodies may have caused selection bias. This selection procedure was necessary to provide a comparison between the United States and Europe, but, therefore, cannot be extrapolated to other drugs. However, the distribution of drugs in the cohort reflect recent approval trends and we found no evidence of modification of differences by any of the subgroups, although subgroup sample sizes were often small and differences were confounded by other factors.

The included reports differed in timing. Thus, some uncertainties reported by regulators may have been resolved at the time of HTA. However, this effect is at most minor and will not influence the conclusions related to effects against relevant comparators and clinically relevant outcomes because uncertainties were raised by HTA bodies for almost all drugs within these categories. Size and detail of reports also differed, including the option for HTA bodies to perform assessments at the drug‐class or indication level. We assumed that all relevant uncertainties were reported.

We chose to combine uncertainties raised by multiple institutions into an aggregated European HTA dataset. This increases the validity of the United States vs. Europe comparison, but also means results do not necessarily apply to each individual HTA jurisdiction in Europe. Because HTA bodies do not always raise the same uncertainties, the misalignment between the EMA and individual HTA bodies may differ from the aggregated set. Nevertheless, for pre‐approval and postapproval evidence generation within Europe, uncertainties raised by any of the HTA bodies are relevant, which makes our method of comparison the most appropriate one.

Conclusion

In the United States as well as in Europe, in order to fulfill their respective decision‐making mandates, some overlap exists in evidence needs between regulators and HTA bodies, but there is also substantial discordance, particularly regarding drug safety, drug effects vs. appropriate comparators, and the suitable time horizon for effects on relevant outcomes. Increased coordination between these two important, complementary organizations may facilitate the collection of necessary evidence in an efficient and timely manner.

Funding

No funding was received for this study. Dr. Kesselheim’s work is supported by Arnold Ventures, as well as the Harvard‐MIT Center for Regulatory Science. Dr. Naci’s work is supported by the Commonwealth Fund.

Conflicts of Interest

All authors have completed the International Committee of Medical Journal Editors (ICMJE) uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; H.G.M.L. reports that he is a member of the Lygature Leadership Team. A.S.K. reports serving on the FDA Peripheral and Central Nervous System Advisory Committee. S.G.S. is the principal investigator of investigator‐initiated grants to the Brigham and Women’s Hospital from Bayer, Vertex, and Boehringer Ingelheim unrelated to the topic of this study. He is a consultant to Aetion, a software manufacturer of which he owns equity. His interests were declared, reviewed, and approved by the Brigham and Women’s Hospital and Partners HealthCare System in accordance with their institutional compliance policies. All other authors declared no competing interests for this work.

Author Contributions

R.A.V. and A.S.K. wrote the manuscript. R.A.V., W.G.G., A.K.M.‐T., S.G.S., H.G.M.L., and A.S.K. designed the research. R.A.V., H.N., and A.S.K. performed the research. R.A.V., W.G.G., A.K.M.‐T., S.G.S., H.G.M.L., and A.S.K. analyzed the data.

Table S1

Click here for additional data file.