Benefit-risk evaluation: the past, present and future.

In the last two decades there has been a shift in the approach to evaluating the benefit-risk (BR) profiles of medicinal products from an unstructured, subjective, and inconsistent, to a more structured and objective, process. This article describes that shift from a historical perspective; the past, the present, and the future, and highlights key events that played critical roles in changing the field.

Introduction

In the last two decades there has been a shift in the approach to evaluating the benefit–risk (BR) profiles of medicinal products from an unstructured, subjective, and inconsistent, to a more structured and objective, process. This article describes that shift from a historical perspective; the past, the present, and the future, and highlights key events and initiatives that played critical roles in changing the field.

The past: subjective assessment

In the late 1990s and early 2000s, while there was an increasing pressure from regulatory agencies for pharmaceutical companies to perform BR evaluation more routinely and systematically, there were only a few publications in the literature and a few guidelines from the regulators on how to perform BR analyses. The majority of the publications in the literature were line listings of benefits and risks which might be useful for clinicians but, without taking into account the relative importance of benefits to risks (or preference weight), these could be interpreted differently by different stakeholders, leading to inconsistency in the interpretation.[1-4]

In 1998, the need for a more systematic and consistent approach to combining the benefits and risks was first introduced by the Council for International Organizations of Medical Sciences (CIOMS), in the report of CIOMS Working Group IV, Benefit–risk balance for marketed drugs: evaluating safety signals:[5] ‘If the benefits of the various options under consideration can be assumed to be equal, benefit–risk evaluation can rely on measures of relative risk. Otherwise, in the absence of a readily available and quantitative relationship between benefits and risks, which is commonly the case, evaluation usually comes down to analyses and conclusions that rely on indirect, informal and unavoidably subjective processes.’ CIOMS recognized that the common practice of BR evaluation at that time was subjective in nature and further mentioned that there were no methods to evaluate BR profiles of medicinal products comparing different treatments with relative merits, ‘There are no accepted general methods for deriving a benefit–risk ratio or another composite metric, or for using such measures to compare relative merits of alternative treatments. As ordinarily used, therefore, the BR ratio compares figuratively, but not often quantitatively, the relative magnitudes of benefits and risks.’ The terms ‘semiquantitative’ and ‘quantitative’ in this report referred to the quantification of the benefits and risks, such as severity of events for semiquantitative and cumulative incidence for quantitative measures, and not for the preference weight comparing benefits and risks.[5]

By 1999, there was still no guidance from the regulatory perspective. The European Committee for Proprietary Medicinal Products recommended methods to evaluate risks in the postmarketing settings (such as observational studies) but offered no clear guidance on how to perform BR evaluation, taking into account the preference weight between the benefits and the risks and: ‘. . .[w]henever possible, both benefits and risks should be considered in absolute terms and in comparison to alternative treatments. The degree of risk that may be considered acceptable is dependent on the seriousness of the disease being treated.’[6] The US Food and Drug Administration (FDA) provided guidance on the benefits and risks but not on BR evaluation.[7]

Two papers that proposed a more structured and quantitative BR evaluation were published in the early 2000s, proposing a few quantitative BR methods taking into account the preference weight of benefits and risks.[8,9] The papers proposed number needed to treat (NNT) and number needed to harm (NNH) to measure the effects for benefits and risks, respectively, and relative value adjusted NNT to not only take into account benefits and risks but also the preference weight of benefits and risks. While the concepts of NNT, NNH, and relative value to adjusted NNT were not new and introduced as early as 1988,[10-14] these two papers by Holden and colleagues[8,9] offered a different perspective and generated a lot of interest for further evaluation of the BR field, followed by key public–private partnerships in this field.

The present: structured and quantitative framework

Starting in the mid 2000s, the BR evaluation field has shifted toward a more structured and quantitative approach. In 2006, following the publication of the paper by Holden and colleagues,[8] the Benefit–Risk Action Team (BRAT), a collaborative project on BR evaluation sponsored by The Pharmaceutical Research and Manufacturers of America was initiated,[15] followed by several other regulatory [European Medicines Agency (EMA) and FDA] and public–private partnership projects a few years after. Some key projects in the field are highlighted in chronological order here.

The BRAT framework

The BRAT framework is a process to perform BR evaluation in a structured, transparent, and consistent way.[15] The process consists of six steps (define the context, identify outcomes, identify data sources, customize framework, assess outcome importance, and display and interpret key BR metrics);[15] it helps to inform stakeholders to make BR decisions, to communicate the decisions and the rationales for the decisions, and therefore increases the transparency of the whole process. The framework is flexible to incorporate benefits, risks, and preference weight from different perspectives, and standardized to ensure transparency and consistency. It is also simple and easy to understand, which is very important to improve the chance of successful application, especially in an organization with a complicated decision-making structure with numerous conflicting agendas.

Another framework that is comparable with BRAT is PrOACT–URL (problem, objectives, alternatives, consequences, trade-off, uncertainly, risk tolerance, and linked decisions).[16,17] Both BRAT and PrOACT–URL are standardized, yet flexible, frameworks suitable for incorporating outcomes and preference weight. Although BRAT is simpler (with fewer steps) than PrOACT–URL, there is no clear advantage of one over the other. In addition, both frameworks meet the necessary steps for an effective BR evaluation: define the context in which the decision is being made, identify the important relevant information and data regarding benefit and risk, assess the preference weight, make a decision from the information based on expert judgment, and communicate the decision and its rationale.[18] While BRAT and PrOACT–URL allow for structured BR evaluations, neither of them offer quantitative methods to integrate benefits and risks and incorporate preference weight into the BR evaluation.

EMA benefit–risk methodology project

The EMA recognized the need to develop a more structured approach to evaluating the BR profiles of medicinal products to ensure transparency and consistency across different stakeholders and started a 3-year ‘Benefit–Risk Methodology’ project in 2009.[19] Based on the early results from the project, the investigators introduced a two-level approach to performing BR evaluation: first, a qualitative approach, mainly consisting of key effects of the benefits and risks and their uncertainty, and second, recommended for more complex situations, a quantitative approach utilizing quantitative methods to incorporate preference weight. One specific quantitative method, the multicriteria decision analysis (MCDA), was specifically mentioned, although it was recognized that the method had significant challenges for successful implementation that needed to be addressed.[19]

In its final report in 2012, Benefit-risk methodology project, work package 4 report: benefit–risk tools and processes, the EMA’s suggestions were more specific:[20]

(1) PrOACT–URL model for qualitative approach which may be sufficient for most cases, or

(2) MCDA model for quantitative app-roach, following the eight steps consistent with PrOACT–URL, which are suitable for more complex situations.

Besides complicated situations, the MCDA model was also suggested to be used when BR balance was ‘marginal’ and there was no clear BR advantage of one treatment over the others. EMA also considered that the quantitative BR model could play a key role for European regulators to monitor the BR profile of a medicinal product postapproval and to update the model with new data to assess whether the BR profile has changed.[20]

FDA benefit–risk framework

In 2009, the same year the EMA started its BR methodology project, in response to feedback from FDA stakeholders and recognizing the need to improve the clarity and transparency of FDA’s BR evaluations, the FDA initiated an effort to explore more systematic approaches to BR assessment and communication.[18] In this effort, the FDA concluded that a structured qualitative approach would be more appropriate, as it was flexible to accommodate quantitative analysis and that the use of quantitative analysis was supporting, rather than replacing, judgment. The FDA also considered that quantitative methods might not capture the nuanced assessments and they might obscure subjective expert judgment.[18]

The FDA developed a structured BR framework with key decision factors as follow: analysis of condition, current treatment options, benefit, risk, and risk management; each factor with two components: (a) evidence and uncertainties; and (b) conclusions and reasons (Figure 1).[18] While the EMA suggested PrOACT–URL for qualitative, and MCDA for quantitative evaluation in its BR methodology project report, the FDA did not prescribe any methods in its structured BR framework.

Figure 1.

FDA benefit–risk framework.

Adapted from Structured approach to benefit–risk assessment in drug regulatory decision making: draft PDUFA V implementation plan – February 2013 fiscal years 2013–2017.[18]

FDA, US Food and Drug Administration, PDUFA V, fifth authorization of the Prescription Drug-User-Fee Act.

As part of the fifth authorization of the Prescription Drug User Fee Act (PDUFA V), in 2013, the FDA published the draft implementation plan of the BR framework for the fiscal years 2013–2017. As part of PDUFA VI, the FDA would continue the implementation of the structured BR framework in the fiscal years 2018–2022 and had made several commitments, including holding a meetings to gather stakeholder input and conducting a second evaluation of the implementation of the BR framework starting in 2021.[21]

IMI: PROTECT

Started in 2009, the Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT) was a collaborative European project under the umbrella of the Innovative Medicine Initiative (IMI) with an aim to address limitations of current methods in the field of pharmacoepidemiology and pharmacovigilance. One of its working programs, the BR group was established to develop methods for use in BR assessment, including both the methods and the presentation of the results, with a particular emphasis on graphical methods.[22] A total of 47 BR methods were identified, thirteen of which selected for further investigation in the case studies;[23] lessons learned from the PROTECT BR group were summarized and synthesized into the BR roadmap.[24]

IMI–PROTECT contributed significantly to the BR field especially by providing deeper and more careful evaluations of a wide range of BR methods that could be categorized as follows:[24]

(1) Frameworks: BRAT and PrOACT–URL;

(2) Quantitative methods: MCDA and stochastic multicriteria acceptability analysis;

(3) Metrics: NNT, NNH, impact numbers, and BR ratio;

(4) Estimation techniques: probabilistic simulation model, indirect/multiple treatment comparison;

(5) Utility survey technique: discrete choice experiment (DCE).

Results from these evaluations have been publis-hed.[25-33] The IMI – PROTECT BR group work has also been used as a reference in regulatory guidance documents and in their working plan.[34]

In addition, there were other projects that contributed to the shift of BR evaluation toward a more structured and quantitative approach: the IMI–ADVANCE (Accelerated Development of Vaccine Benefit–Risk) initiative which started in 2013 with an aim to develop and test methods and guidelines for BR evaluations vaccines that are on the market[35] and the UMBRA (Unified Methodologies for Benefit–Risk Assessment) initiative, which was started in 2012 by The Center for Innovation in Regulatory Science (CIRS), resulting in the UMBRA framework that has been used by regulatory agencies in Canada, Australia, Switzerland, and Singapore.[36]

All of these projects and initiatives contributed to a current practice of BR evaluation, not only among pharmaceutical industry but also among regulators in their decision-making process, which was more structured, transparent, and consistent. Most of the analyses were qualitative in nature, but quantitative methods were also available and accepted by regulators for a more complex situation where there was no clear advantage of one treatment over the other. When a quantitative approach was necessary, preference weight was needed from different stakeholders, including patients.

It was not clear how much these frameworks had made an impact among pharmaceutical industry and regulators, but a number of studies using the BRAT framework had been published.[37-40] The BRAT framework was also used in a number of IMI–PROTECT case studies[22] and it was consistent with the FDA structured BR approach.[18] Two BR studies using PrOACT–URL have been published[40,41] and it was used in the EMA BR Project.[19] A small number of publications of BR studies using these two frameworks might not fully capture the extent to which they have been implemented in the pharmaceutical industry and regulators. Because there were a large number of pharmaceutical companies as well as regulators in these initiatives, it was likely that the frameworks had been used for internal purposes or as part of their regulatory requirements, including FDA advisory committee meetings. The results were not published probably because of proprietary and confidentiality issues.

Results of BR evaluations using quantitative methods have also been published before.[40,42-44] It was not clear whether the regulators used quantitative BR methods. The FDA did not recommend any specific quantitative method, although the use was encouraged to support the qualitative approach.[18] Although one of the suggestions from the EMA BR project was to use MCDA for a more complex situation,[19] it was not clear whether MCDA had been used in the EMA decision-making process. One of the reasons for the regulators’ reluctance to formally use quantitative BR methods might be to maintain their neutral position and not to favor one method over the other. Another possible reason was that, given the complexity of many of the methods, they might not yet have the capability to implement them.

The future: patient perspective in BR evaluation

The role of patient preferences in the BR evaluation of medicinal products and devices has become increasingly important in the last several years and the patient perspective has become an integral part of the regulatory decision making. Patients’ roles not only include the expression of patients’ preferences to be used in the BR evaluation during regulatory approval or postmarketing evaluations, but also in discovery and development. In the discovery phase, patients’ feedback can help to identify unmet medical needs to inform the design of the target product profile, better understand the disease and acceptability of benefits and risks. In clinical development, patient preferences can inform clinical trial design through identification of patient-relevant outcomes, for example, but can also inform product design validation and provide insights on acceptability of benefits and risks.[45]

The shift toward a focus on patient preference started as early as 2009 when a case study on patient preference in BR evaluation was done by the IMI–PROTECT BR Group.[26] In this study, patients with obesity were contacted to participate in the BR evaluation of rimonabant (a CB1 receptor antagonist) with a comparison groups using DCE method.[26] Afterwards, there were a number of initiatives on patient preference by the FDA, EMA and a number of public – private partnerships.[45-51]

From the regulatory perspective, the shift toward a more patient-centered BR evaluation started in the early 2010s. In 2012, as part of PDUFA V, the FDA held the first ‘Public meeting and request for comments’ on patient-focused drug development[46] and a total of 24 meetings during the fiscal years 2013–2017 were planned. In 2012, the FDA Center for Devices and Radiological Health (CDRH) published a guidance that included quantitative patient-preference data on the trade-off between benefits and risks as a factor in the regulatory review; this guidance was revised in 2016.[52] The FDA also published a guidance for ‘Patient preference information: voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling’ in 2016.[47] An initiative on patient preference elicitation was announced by the EMA in 2015, although patient representatives have been participating in the EMA Scientific Advisory Group meetings since 2011.[48] As expected, this trend toward a focus on patient preferences among regulatory agencies impacted the research agenda, not only among regulators but also academia and the pharmaceutical industry.

The first public–private partnership with significant impact on patient preference in the BR evaluation was the Medical Device Innovation Consortium–Patient-Centered Benefit–Risk (MDCI–PCBR) Project in the US, with an objective of improving medical device regulatory science for patient benefit.[49] The MDIC, founded in 2012, performed the PCBR project in response to the FDA CDRH’s priority to focus on BR assessment as a central component of the medical device approval process. Representatives from the FDA, NIH, industry, nonprofit, and patient organizations participated in the project. The MDIC–PCBR project had two main components: (a) the development of a framework underpinning BR assessment with patient preference; and (b) the development of the catalog of methods that can be used not only to collect patient preference but also to incorporate into the BR analysis. Both the framework and catalog of methods were very useful and have been used by different public–private partnerships, pharmaceutical companies, and other stakeholders to perform patient-centered BR evaluations which have become increasingly important for meeting regulatory requirements.

In Europe, another public–private partnership was formed, focusing on ‘. . .education and training to increase the capacity and capability of patients to understand and contribute to medicine research and development and also improve the availability of objective, reliable, patient-friendly information for the public.’[50] This project, another one under the umbrella of IMI, the European Patients’ Academy (EUPATI) was a European consortium of representatives from the pharmaceutical industry, academia, not-for-profit, and patient organizations.[50] While EUPATI did not provide or propose any methods for the collection of patient-preference information or incorporating it in the BR evaluation, it plays an important role in the education and training of patients, critical to the success of a patient-centered BR evaluation.

Started in 2016, also under IMI, the Patient Preferences in Benefit and Risk Assessments during the Treatment Life Cycle (PREFER) project is a 5-year public–private research initiative with representatives from academia, the pharmaceutical industry, and patient organization and health technology assessment bodies, with an aim to strengthen patient-centric decision making throughout the life cycle of medicinal treatments by developing expert and evidence-based recommendations to guide the stakeholders.[51] One of the strengths of the PREFER initiative is that, learning from previous initiatives by the FDA, EMA, IMI PROTECT, MDIC and EUPATI, PREFER’s approach was broader, by incorporating perspectives from a wide set of stakeholders.[51] It will evaluate the stakeholders’ expectations and concerns about the assessment of patient preferences and their use in decision making. Based on this, a systematic and comprehensive review of patient-preference methods will be performed, with input from experts in the field. Another strength is that a number of methods would be further tested and evaluated in a number of clinical case and simulation studies focusing on different decision points in the medical product life cycle.[51] Results from the PREFER initiative will contribute significantly not only to the patient-preference research area but also to BR fields. Results on factors and situations that influence the value of patient-preference studies along the medical product lifecycle have been published.[45]

It is not clear how much impact these initiatives on patient preference have made in the pharmaceutical industry, and regulators in their decision-making processes, but several studies using patient-preference data in BR evaluations have been published before.[38,53-59] FDA CDRH had implemented patient preference in the BR evaluation of medical devices earlier than the Center for Drug Evaluation and Research (CDER). CDRH published the draft guidance for quantitative patient-preference data in 2012 which was then revised in 2016;[52] CDER published its draft guidance on patient-focused drug development 6 years later in 2018.[60] With its collaboration with the MDIC, the CDRH performed the first public-partnership project on patient preference;[49] CDER has not performed any public–private initiative on this topic. CDRH also performed the first study of quantitative patient-preference data to inform its approval decision.[58] In this study, discrete-choice experiments were used to quantify patient preference in comparing different weight-loss devices among US population with obesity.[58] There has been no published study on patient preference by CDER.

Final remarks

There has been great improvement in the BR field in the last 2 decades, from a subjective and inconsistent, to a more structured, transparent and consistent approach, with a number of quantitative methods to incorporate preference weight from various stakeholders. While this development is encouraging, there is still more work to be done.

Communication of BR evaluation to patients and other stakeholders is one of the areas that need to be improved. How could BR decision and the rationale behind it be communicated effectively and transparently with an opportunity for a routine dialog between decision makers and stakeholders? When should communication between regulators and stakeholders start in the drug lifecycle? In various public partnerships, less attention has been given to the communication than to other aspects of BR evaluation. Another area is the patient perspective; how, when, and what to elicit regarding the patient perspective and how to incorporate it in the BR evaluation. A number of past and ongoing initiatives will address these issues to a certain degree, but education and training of patients and other stakeholders will take a longer time, and more efforts will be required to enable patients to participate in these discussions. BR evaluations must be meaningful; not only to regulators and pharmaceutical companies but most importantly, to patients.