Study Design and Evaluation of Risk Minimization Measures: A Review of Studies Submitted to the European Medicines Agency for Cardiovascular, Endocrinology, and Metabolic Drugs.

INTRODUCTION: Studies measuring the effectiveness of risk minimization measures (RMMs) submitted by pharmaceutical companies to the European Medicines Agency are part of the post-authorization regulatory requirements and represent an important source of data covering a range of medicinal products and safety-related issues. Their objectives, design, and the associated regulatory outcomes were reviewed, and conclusions were drawn that may support future progress in risk minimization evaluation.
METHODS: Information was obtained from risk management plans, study protocols, clinical study reports, and assessment reports of 157 medicinal products authorized for cardiovascular, endocrinology, and metabolic indications. We selected observational studies measuring, as outcomes of interest, the relationship between the RMMs in place and (1) implementation measures, such as clinical knowledge or physicians` compliance to recommendations contained in the RMMs; and (2) occurrence or reduced severity of the adverse drug reactions for which the RMMs were required.
RESULTS: Of 59 eligible studies (24 completed, 35 ongoing), 44 assessed implementation measures, whereas only 15 assessed safety outcomes (1 study as a single endpoint and 14 studies with other endpoints). Fifty-one studies used non-experimental designs and 25 studies employed electronic healthcare databases for analysis. Of the 24 completed studies, 17 were considered satisfactory and supported immediate regulatory decision making, 6 were considered inconclusive and required new evaluations, and 1 was terminated early because new safety restrictions were required, thereby necessitating a new evaluation. Compliance with agreed deadlines was considered acceptable in 21 of 24 completed studies; the average time for a submission was 37 months (standard deviation ± 17), with differences observed by type of data source employed.
CONCLUSIONS: Three important gaps in the evaluation plans of RMMs were identified: lack of early feedback on implementation, limited evaluation of safety outcomes, and inability to provide information on the effectiveness from an integrated measurement of different elements of a set of risk minimization tools. More robust evidence is needed to advance regulatory science and support more rapid adjustment of risk minimization strategies as needed.

Key Points

Introduction

The European Union (EU) Risk Management Plan (EU-RMP) was introduced in 2005 as the instrument for planning pharmacovigilance activities and risk minimization measures (RMMs) for new chemical entities and biologicals, as well as for generics when additional risk management activities were identified for the reference product [1]. In July 2012, with the implementation of the current EU pharmacovigilance legislation, an EU-RMP became mandatory for all newly authorized products [2, 3]. The EU-RMP aims to ensure that the benefit-risk profile of a medicinal product is managed optimally during the product lifecycle, knowledge gaps at the time of authorization are elucidated, and risks are further quantified and characterized over time [4].

The three pillars of the EU-RMP are: (1) the safety specification that describes the safety concerns (any important risks and missing information that could affect the benefit-risk profile of the medicinal product or have implications for public health); (2) the pharmacovigilance plan to characterize these safety concerns (routine and additional activities); and (3) the risk minimization plan to mitigate the risks in clinical practice [5].

For all medicinal products, the EU-RMP includes routine RMMs, meaning the provision of information and recommendations in the summary of product characteristics (SmPC) and package leaflet (PL), the product labeling, a pack size appropriate to the anticipated treatment duration, and a risk-appropriate legal status of the product (e.g. prescription-only medicine) [5]. However, for some drugs routine risk minimization might not be sufficient and it may be necessary to implement additional measures, such as the provision of education materials for healthcare professionals (HCPs) and patients. These additional RMMs (aRMMs) aim to support a positive benefit-risk profile, for example, by targeted patient selection or exclusion, appropriate treatment management (e.g. specific dosing regimen, relevant testing, and patient follow-up), and/or early recognition of adverse drug reactions (ADRs) [6].

The assessment of the effectiveness of RMMs is of importance to ensure that their objectives are fulfilled or to provide evidence that further amendments are warranted. This knowledge can also be used to inform on optimal risk minimization strategies for other medicines to maintain a positive benefit-risk profile [5, 7, 8]. This activity has in fact been recognized as one of the key elements of a broad strategy established in January 2016 by the EMA Pharmacovigilance Risk Assessment Committee (PRAC) to measure the impact of pharmacovigilance activities that may enable the monitoring of important pharmacovigilance outputs and regulatory actions and contribute to the further development of the proactive EU pharmacovigilance system [9].

Available evidence shows that studies measuring the effectiveness of RMMs to date are mainly focused on medicinal products that were subject to extensive risk communication because of a serious safety-related issue [10], employed secondary data for collecting information on drug exposure and related health outcomes [11, 12], or used suboptimal research designs and analytic methods, making the results susceptible to bias [13, 14].

Studies submitted by pharmaceutical companies to the competent authorities represent an additional source of data covering a range of products and safety-related issues. The EU pharmacovigilance legislation [2] defines these as post-authorization safety studies (PASS) and envisages detailed guiding principles for their conduct, such as their inclusion in the EU-RMP of each medicinal product, the format of protocols, abstracts and clinical study reports (CSRs), and the mandatory submission of the CSR to the competent authorities for assessment by the agreed deadlines [5, 6].

The present study was conducted to provide an overview of industry-sponsored studies measuring the effectiveness of RMMs included in the EU-RMP of medicinal products in designated therapeutic areas (i.e. cardiovascular, endocrinology, metabolic), including their objectives, design, and the associated regulatory outcomes. It also aimed to assess how these studies were able to support decisions by EU regulatory authorities. This should in turn help to identify enablers and barriers associated with efforts to evaluate RMMs, to draw conclusions relevant to the future development in the area of risk minimization evaluation, including process and methodological improvement.

Methods

Identification of Centrally Authorized Products

All medicinal products authorized in the EU with a centralized procedure (i.e. centrally authorized products [CAPs]) [1] between 1 January 1995 and 31 July 2015 for cardiovascular, endocrinology and metabolic indications, and not withdrawn or suspended post-authorization, were identified. New and known active substances, well-established use products, and fixed combination products for which a full dossier was submitted and assessed by the European Medicines Agency’s (EMA’s) committees during the study period were considered eligible for this review. All products based on generic, biosimilar, or hybrid applications (i.e. applications that do not contain a full dossier but rely on a reference medicinal product) were excluded. From the eligible products, those with an EU-RMP either at the time of the marketing authorization or at any time post-authorization qualified for inclusion in the current study.

Data Sources

The EMA public website was used to collect information on medicinal products, including status after initial authorization (active or withdrawn), legal status (over the counter [OTC], subject to medical prescription), and authorization details (e.g. exceptional circumstances, orphan designation). Information on the type of risk minimization (routine vs. additional) and studies measuring the effectiveness of RMMs was retrieved from the most recent EU-RMP authorized by the EMA committees (i.e. PRAC or the Committee for Medicinal Products for Human Use [CHMP]) and stored in the EMA electronic records management system. The EMA electronic records management system was also checked to retrieve further relevant documentation on the studies measuring the effectiveness of RMMs, such as information on study protocols, CSRs, and EMA committees’ assessment reports to collect information on the regulatory outcome resulting from the assessment of the CSRs [15].

The EMA core regulatory processes database i.e. the Model System for Computer-Assisted Drug Registration (SIAMED) developed by the World Health Organization was used to retrieve information on medicinal products authorized on the basis of a hybrid application, as well as the type of application made by the pharmaceutical companies (i.e. regulatory procedure) resulting in these post-authorization measures. Information on the publication status of completed studies was retrieved using the EU electronic register of post-authorization studies (EU PAS Register) maintained by the European Network of Centers for Pharmacoepidemiology and Pharmacovigilance (ENCePP) [16] and the US registry of studies maintained by the National Institutes of Health [17]. A PubMed search was also conducted to check how many studies were published in the peer-reviewed scientific literature.

Data Extraction and Classification

Data were extracted by one researcher (DdS) using a prespecified standardized data extraction format. Study selection and classification was then conducted independently by two researchers (GM and DdS) and cross-checked for consistency. Discrepancies in study selection and classification were resolved by discussion.

All selected studies were classified according to the study endpoints, the research design, and the type of data source being used. The following study endpoints adapted from Prieto et al. [18] were considered: (a) clinical knowledge, aimed at determining whether the end users have correctly understood the purpose of the risk minimization tools and their key messages; (b) clinical behavior, aimed at measuring to what extent the behavior of the targeted stakeholders (e.g. HCPs, patients, or both) differs from the recommendations contained in the RMM; (c) safety outcomes, aimed at correlating the implementation of the RMM with reduced incidence or reduced severity of ADRs; (d) multiple outcomes when more than one of the above endpoints were proposed in the same study.

Research designs were classified into quasi-experimental and non-experimental. Quasi-experimental designs were further classified into before-and-after studies and interrupted time series (ITS). In both types of design, the evaluation of the effect of an intervention is performed by measuring the study endpoints before and after the introduction of the intervention in the same study population and by using baseline (pre-intervention) characteristics as a comparison group [13, 19]. Non-experimental designs only measure the study endpoints post-intervention. In the absence of a comparison group, the effect of the intervention can be estimated by using a predefined reference value, such as the expected outcome frequency in the general population or in the prelicensing clinical trials. Non-experimental designs were further classified as retrospective or prospective studies according to the time perspective for data collection.

Study Selection

Study selection was performed using a two-step approach. First, titles and objectives were reviewed for all interventional trials and other forms of study included in the pharmacovigilance plan of the relevant EU-RMPs, and only observational studies were selected. Second, for each of the selected studies, protocols and CSRs (where available) were examined to select the final list. At least one of the following eligibility criteria were considered for inclusion: (a) the assessment of the effectiveness of the RMM (e.g. major changes in the SmPC/PL or adoption of new educational material for HCPs/patients) was mentioned as the primary or secondary study objective; (b) the date of implementation of RMMs was used as the change point to analyze trends in drug exposure or incidence of adverse events; (c) the study was designed to assess the clinical knowledge or behavior regarding the recommendations contained in the RMM by using predefined measurable indicators [20].

Statistical Analysis

Descriptive statistics for the selected studies were analyzed as means (standard deviation [SD]), proportions, or count data, as appropriate. The studies were counted once regardless of whether they were related to more than one medicinal product. Conversely, more than one study could have been related to a single medicinal product or regulatory procedure, thus leading to different numbers for medicinal products and regulatory procedures associated with the selected studies.

Results

Of 297 medicinal products authorized in the observation period, 55 were withdrawn for commercial or safety reasons, 52 were excluded because they were generic, hybrid or biosimilar applications, and 33 were excluded because no RMP was present in the dossiers submitted by the pharmaceutical companies, neither at the time of the marketing authorization application nor in the post-authorization phase (Fig. 1). From the EU-RMP of the remaining 157 medicinal products, 436 studies were identified, of which 204 were not observational and 173 did not fulfill our selection criteria. A total of 59 studies (30 for routine risk minimization, 29 for additional risk minimization) qualified for the analysis (Fig. 1). The medicinal product characteristics associated with the selected studies are described in Table 1.

Fig. 1

Inclusion of eligible products and corresponding selected studies

Table 1

Medicinal product characteristics associated with the selected studies, stratified by type (routine vs. additional) of risk minimization measure

	Routine [N = 30] (%)	Additional [N = 29] (%)	Total [N = 59] (%)
Legal status
OTC	4 (13.3)	0	4 (6.8)
Medical prescription	22 (73.4)	20 (69)	42 (71.2)
Restricted medical prescription	4 (13.3)	9 (31)	13 (22)
Authorization details
Exceptional circumstances
Yes	0	3 (10.3)	3 (5.1)
No	30 (100)	26 (89.7)	56 (94.9)
Orphan status
Yes	1 (3.3)	6 (20.7)	7 (11.9)
No	29 (96.7)	23 (79.3)	52 (88.1)
Therapeutic group (ATC)a
Drugs used in diabetes	10 (33.3)	6 (20.7)	16 (27.1)
Antithrombotic agents	6 (20)	7 (24.1)	13 (22)
Other alimentary tract and metabolism products		7 (24.1)	7 (11.9)
Cardiac therapy		5 (17.2)	5 (8.5)
Drugs for treatment of bone diseases	4 (13.3)	1 (3.4)	5 (8.5)
Anti-obesity preparations, excluding diet products	4 (13.3)	1 (3.4)	5 (8.5)
Lipid-modifying agents	1 (3.3)	2 (6.9)	3 (5.1)
Othersa	5 (16.7)	0	5 (8.5)

OTC over the counter, ATC Anatomical Therapeutic Classification (second level)

aIncludes five therapeutic groups for which only one study was requested

Characteristics of the Selected Studies

Table 2 describes the main characteristics of the selected studies. Approximately half (n = 30) evaluated clinical behavior, 13 assessed clinical knowledge, and 1 evaluated multiple endpoints (without safety outcomes). Additionally, safety outcomes were assessed in 15 studies—in 1 study as a single endpoint and in 14 studies measured simultaneously with other endpoints (i.e. clinical behavior).

Table 2

Characteristics of the selected studies, stratified by type (routine vs. additional) of risk minimization measure

	Routine [N = 30] (%)	Additional [N = 29] (%)	Total [N = 59] (%)
Study endpoints
Clinical knowledge	4 (13.3)	9 (31)	13 (22)
Clinical behavior	22 (73.4)	8 (27.6)	30 (50.8)
Safety outcomes	0	1 (3.4)	1 (1.7)
Multiple (without safety outcomes)	0	1 (3.4)	1 (1.7)
Multiple (with safety outcomes)	4 (13.3)	10 (34.5)	14 (23.7)
Research design
Non-experimental (retrospective)	26 (86.7)	17 (58.6)	43 (72.9)
Non-experimental (prospective)	3 (10)	5 (17.2)	8 (13.6)
Quasi-experimental (before/after)	1 (3.3)	6 (20.7)	7 (11.9)
Quasi-experimental (ITS)		1 (3.4)	1 (1.7)
Data sources
Questionnaires	7 (23.3)	11 (37.9)	18 (30.5)
Electronic healthcare databases	16 (53.3)	9 (31)	25 (42.4)
Retrospective chart reviews	3 (10)	2 (6.9)	5 (8.5)
Primary data collection, including registries	3 (10)	5 (17.2)	8 (13.6)
Othersa	1 (3.4)	2 (6.9)	3 (5.0)

ITS interrupted time-series

aFocused usability testing (2), immunology testing (1)

A non-experimental design was used in 51 studies; among them, we observed 43 retrospective studies and 8 prospective studies. Quasi-experimental designs were applied in 8 studies (1 an ITS and 7 a before/after design); 7 of these studies measured the impact of aRMMs.

Of 59 studies, 25 used electronic healthcare databases (EHDs) as a source of information (19 for clinical behavior, 6 for multiple endpoints, including safety outcomes), and 18 used questionnaires (11 for clinical knowledge, 6 for clinical behavior, 1 for multiple endpoints, including safety outcomes). Of 13 studies using primary data and retrospective chart review, 8 evaluated safety outcomes. Stratification by RMM type indicated that EHDs were mostly used to assess the impact of routine RMMs (16 of 30), whereas questionnaires were mostly used to assess the effectiveness of aRMMs (11 of 29).

Regulatory Outcomes

The 59 studies that qualified for this review (both ongoing and completed) were part of 48 regulatory procedures for either initial marketing authorization (n = 23) or postmarketing requirements (n = 25). Among postmarketing requirements, 10 related to extensions of indication, 6 related to line extensions (i.e. strength, pharmaceutical form or route of administration), 6 were in EU reviews of the product’s benefit-risk assessment (i.e. EU referrals), 3 related to emerging safety concerns from two Periodic Safety Update Report (PSUR) assessments, and one was a re-analysis of a clinical trial (Table 3; see electronic supplementary material 1).

Table 3

Main regulatory information of completed studies by active substance and regulatory procedure, stratified by type of RMM

INN [no. of medicinal products]	Type of regulatory procedure [year]a	No. of studies	CSR submission (months)b	Study endpoint	Data source	Regulatory outcome	Study publication
Routine
Bivalirudin [1]	EXT [2007]	1	43	Multiple (with safety outcomes)	Primary data	Study considered unsatisfactory (methodological issues). Further expansion of the educational material; new study requested (see ongoing studies in electronic supplementary material 1)	PMID: 24680250
Denosumab [1]	MA [2010]	1	47	Clinical behavior	EHDs	Results considered satisfactory; no further follow-up required	NA
Fondaparinux [1]	EXT [2007]	1	49	Clinical behavior	Chart review	Results considered satisfactory; no further follow-up required	EUPAS6655 (no results posted)
Fondaparinux [1]	EXT [2010]	1	53	Clinical behavior	Chart review	Results considered satisfactory; no further follow-up required	NCT01406301 (no results posted)
Insulin glulisine [1]	EXT [2008]	3	50	[a] Clinical knowledge[b, c] Clinical behavior	[a] Questionnaire[b, c] EHDs	The first study reported evidence of non-compliance with safety restrictions; further requirement for risk minimization and new studies. Latest studies considered satisfactory; no further follow-up required	NA
Orlistat [1]	LE [2008]	2	24	[a, b] Clinical behavior	[a, b] Questionnaire	Studies reported evidence of non-compliance with safety restrictions. Further risk minimization measures and studies requested (see ongoing studies in electronic supplementary material 1)	NA
Pioglitazone (mono and FDC) [5]	EXT [2006]	1	36	Clinical behavior	EHDs	Results considered satisfactory; no further follow-up required	NA
Rivaroxaban [1]	MA [2008]	1	51	Clinical behavior	EHDs	Results considered satisfactory: no further follow-up required	PMID: 24858823
Strontium ranelate [2]	REF [2012]	1	15	Clinical knowledge	Questionnaire	Study was considered unsatisfactory (methodological issues). New risk minimization measures and studies requested due to additional regulatory actions taken during the conduct of this study (see ongoing studies in electronic supplementary material 1)	NA
Tolvaptan [1]	MA [2009]	1	65	Multiple (with safety outcomes)	Primary data	Study reported evidence of non-compliance with recommended dose. New studies required to evaluate the efficacy and safety of a lower starting/ maintenance dose	NCT01228682 (no results posted)
Vildagliptin (mono and FDC) [6]	MA [2007, 2008]	1	62c	Clinical behavior	EHDs	Results considered satisfactory: no further follow-up required	NA
Additional
Agalsidase beta [1]	LE [2011]	1	22	Clinical knowledge	Questionnaire	Study considered unsatisfactory (low recruitment/drug uptake). Company proposal to measure the impact of RMM by SRS accepted. Reconsideration for future surveys when considered feasible	NA
Dronedarone [1]	MA [2009]	1	30	Clinical behavior	Questionnaire	Study terminated early due to the outcome of an EU referral, where further risk minimization measures and new studies were requested (see ongoing studies in electronic supplementary material 1)	NA
Imiglucerase [1]	LE [2010]	1	27	Clinical knowledge	Questionnaire	Study considered unsatisfactory (low recruitment/drug uptake). Company proposal to measure the impact of RMM by SRS accepted. Reconsideration for future surveys when considered feasible	NA
Insulin degludec [1]	MA [2012]	1	23	Clinical knowledge	Other (focused usability test)	Results considered satisfactory; no further follow-up required	NA
Pioglitazone (mono and FDC) [5]	REF [2011]	2	35	[a, b] Multiple (with safety outcomes)	[a, b] EHDs	Results considered satisfactory; however, new studies requested to investigate the use and effectiveness of RMM in the management of the risk of bladder cancer (see ongoing studies in electronic supplementary material 1)	[a] EUPAS10001[b] EUPAS9998
Pioglitazone/alogliptin	MA [2013]	1d	21	Multiple (with safety outcomes)	EHDs	Study terminated early due to low recruitment/drug uptake. Information on this product included in the ongoing DUS for pioglitazone-containing products (see ongoing studies in electronic supplementary material 1)	NA
Prasugrel [1]	MA [2008]	2	64	[a] Clinical behavior[b] Multiple (with safety outcomes)	[a] EHDs[b] Primary data	Results considered satisfactory; no further follow-up required	[a] PMID: 27299993[b] PMID: 25794517
Ranolazine [1]	MA [2008]	1	84	Clinical knowledge	Questionnaire	Study considered unsatisfactory (methodological issues). Further expansion of the target population requested	NA

INN international non-proprietary name, CSR clinical study report, DUS drug utilization study, EHDs electronic healthcare databases, EXT extension of indication, EUPAS European Union Electronic Register of Post-Authorisation Studies identification number, FDC fixed-dose combination, LE line extension, MA marketing authorization, NA not available, NCT US registry of clinical trials, PMID PubMed identification number, REF referral, RMM risk minimization measure, SRS spontaneous reporting system

aThe year refers to the end date of the regulatory procedure

bTime to submission of the CSR from the end date of the regulatory procedure. In case of more studies for a single regulatory procedure, the date of the last CSR submitted has been considered

cCalculated from the date of the first authorized product

dStudy requested in the context of the marketing authorization of this product because of the referral conducted for pioglitazone-containing products finalized in July 2011

Table 3 describes the regulatory outcomes of 24 completed studies submitted to the EMA’s committees for assessment. Twenty-one CSRs were submitted by the pharmaceutical companies within the planned deadline. Nevertheless, in 8 cases the deadline initially planned was postponed upon request of the pharmaceutical companies and agreement from the EMA’s committees, mainly because of low recruitment or low drug uptake.

The average time from the date each study was formally requested until submission of the CSR to the EMA’s committees was 37 months (SD ± 17) with differences observed by type of data source employed (questionnaires: 30 months [SD ± 17]; EHDs: 37 months [SD ± 14]; retrospective chart review: 50 months [SD ± 3]; and primary data collection: 58 months [SD ± 12]).

The data presented in the CSRs were considered satisfactory and supported immediate regulatory decisions in 17 (70.8%) of the 24 completed studies. Further follow-up was not considered necessary in 13 studies, whereas evidence of non-compliance with safety restrictions was observed in 4 studies and prompted changes of the product information (orlistat), new risk communication (insulin glulisine), and a request for further studies to evaluate safety and efficacy using a different posology (tolvaptan). The results of six studies were considered inconclusive because of feasibility (i.e. low recruitment and low drug uptake) or methodological limitations (i.e. lack of external validity of the selected population and information bias). In two cases (agalsidase beta, imiglucerase) the EMA committees accepted the company proposal to monitor the effectiveness of the RMMs via spontaneous reporting of suspected ADRs, whereas in the other four cases, new studies were requested. One physician survey was terminated earlier and replaced with a drug utilization study (dronedarone) because new safety restrictions were requested during the conduct of the study.

Information on design, methods, and results was made publicly available in 9 of 24 studies, although the results were reported in only 6 studies (4 publications in peer-reviewed journals).

Discussion

Principal Findings

This is the first overview of industry-sponsored studies on the effectiveness of RMMs in designated therapeutic areas, with a focus on study design characteristics and regulatory outcomes. We identified 59 studies that measured the impact of RMMs. The analysis of these studies highlights the need for process and methodological improvement in the approaches to risk minimization effectiveness evaluation.

There are a number of different conceptual models published as guides for developing efficient strategies for measuring the effectiveness of RMMs [18, 21–23]. These models recommend for a comprehensive evaluation of risk minimization, a multifocal, integrated measurement, including each of the different elements of a set of RMMs, particularly the assessment of the implementation of risk minimization tools, such as acquired clinical knowledge of targeted stakeholders (i.e. patients and HCPs), as well as desired behavioral changes. They also emphasize that demonstrating the reduction of the occurrence or severity of ADRs (i.e. safety outcomes) is the ultimate endpoint to assess attainment of risk minimization objectives since this provides an overall measure of the level of risk control that has been achieved with any RMM in place. Therefore, robust evaluations are longitudinal in nature and should include pretesting of the risk minimization tools, the effectiveness of implementation, and the evaluation of the safety outcomes.

However, in designing an evaluation strategy, careful consideration should be given to the particular aspects of implementation and outcome which can be realistically measured to support regulatory decisions. Therefore, when assessment of safety outcomes is considered unfeasible or would incur unacceptable delay, pharmaceutical companies and regulators may agree on evaluation of implementation measures as a way to define the success of the RMM. These measures might provide insight into the effectiveness of the RMM, although there is no certainty that successful implementation corresponds with reduction in occurrence and/or severity of ADRs [6].

In our analysis, 15 studies were designed to measure safety outcomes and 8 of these required primary data collection or retrospective manual review of medical charts to retrieve information on either adverse events reactions reported or clinical outcomes that could reflect the occurrence or severity of ADRs. We have also demonstrated that studies using primary data require around 5 years for the finalization of the CSR, thus explaining the higher proportion of studies evaluating implementation measures and using alternative data sources. Evaluation of safety outcomes might rely on EHDs because they provide rapid information on drug exposure and patient clinical outcomes, mainly based on secondary data collection in an outpatient setting (i.e. primary care databases) [12, 24]. However, EHDs may not capture sufficient clinical outcome data [25] and integration with external sources such as hospital databases, disease/drug registries, and primary data may be challenging. This is reflected in our findings where only 6 of 15 studies used EHDs to measure safety outcomes.

Limited readily available information might also explain the differences observed in measuring the effectiveness of routine versus additional RMMs, particularly the higher proportion of studies using questionnaires and testing the clinical knowledge of the additional risk minimization tools and their key messages. Medicinal products with aRMMs are often used for treating serious diseases in a hospital setting, and information on drug exposure may be limited in EHDs [25]. The results of a recent study [22] showed that it remains challenging to perform aRMM effectiveness evaluation in a timely and appropriate way and that approaches that go beyond the use of EHDs are needed to allow the assessment of behavioral changes and clinical knowledge.

Questionnaire-based studies might be helpful for collecting information on patients’ or HCPs’ knowledge, attitudes, behavior, and medication use in the absence of readily available data. However, these surveys have challenges (e.g. recruiting participants, small or unrepresentative sample sizes) that make it difficult to draw robust conclusions [7]. Moreover, they rely on the respondent’s self-reporting and might provide socially acceptable responses, which have a large impact on the validity of proposed questionnaires and study findings [26].

The use of quasi-experimental designs to measure the impact of RMMs has been supported by a consensus panel discussing preferred methodologies for evaluating US FDA regulatory actions [13] because of their ability to estimate the effect size of outcome measures before and after risk minimization is introduced. The panel also highlighted the importance of using ITS, particularly when it is worth investigating the net effect of the intervention controlled for potential biases, such as underlying secular trends, seasonal effects, and random fluctuations [13, 27]. Our findings reported eight quasi-experimental studies and, among them, only one ITS design, thus suggesting that feasibility and operational reasons might have limited the ability to employ an appropriate design. However, there are certain conditions that should be met for a quasi-experimental study to be conducted, particularly the availability of pre-intervention information. In our analysis, 27 studies were requested at the time of the marketing authorization and 17 studies related to medicinal products with OTC or restricted medical prescription status for which pre-intervention information was unavailable. In the absence of pre-intervention information, the comparison of an outcome frequency measure obtained post-intervention against a predefined reference value (e.g. literature review, historical data, outcome frequency in the prelicensing clinical trials) may be considered [6, 18]. However, such an approach was not proposed in the analysis plan of any of the studies reviewed here.

With regard to the regulatory outcomes, 21 of 24 CSRs were submitted within the agreed timeframe and 17 provided results that supported immediate regulatory decisions. It should be noted that for 8 studies, the EMA’s committees acknowledged there were justified reasons for postponing the original due date, while the results of 7 CSRs were considered either outdated or of limited value, prompting the need for further evaluation of the RMMs. These findings suggest the requirement for more careful consideration and justification by the pharmaceutical companies of the proposed risk minimization evaluation in terms of objectives, design, and feasibility. They also suggest the need for a more comprehensive approach not limited to a single element of the risk minimization strategy and capable of generating different lines of evidence at relevant intervals to help regulators to promptly assess decision-relevant information. The utility of this approach has been recently emphasized in an EU referral on the fixed-dose combination of cyproterone and ethinylestradiol and the risk of thromboembolism. Here, the PRAC agreed that the assessment of the effectiveness of the aRMMs in place could benefit from both a drug utilization study and a questionnaire-based survey on clinical knowledge because of the complementarity of the information that would be provided [28]. Finally, our findings suggest that more effort is needed to ensure that study results are made publicly available to help advance the science of risk minimization evaluation by facilitating peer review of protocols and results, and encourage collaboration among stakeholders.

Strengths and Limitations of the Study

This is the first review of industry-sponsored studies measuring the impact of RMMs with access to relevant documents that also included published and unpublished CSRs, integrating the assessment of study design and regulatory outcomes. However, the study has some limitations. First, we selectively included medicinal products authorized for cardiovascular, endocrinology and metabolic indications for the review. These represent a substantial proportion (29.7%) of all medicinal products authorized during the study period and mainly focus on targeted populations (i.e. chronic and elderly users) potentially at higher risk for ADRs [29, 30]. However, there are differences compared with other drug classes in terms of patient characteristics, therapeutic indications, and posology, which might lead to differences in either the risk minimization strategy or the type of studies employed to assess effectiveness, as observed in a recent study from Zomerdijk et al. [31].

Second, the initial study selection was based on titles and objectives as reported in the EU-RMP and could have resulted in an incomplete selection of the eligible studies. Although the risk of overlooking studies has been minimized by the selection being undertaken by two independent reviewers, this bias cannot be completely ruled out. Third, the EU-RMP does not necessarily include all studies being performed to measure the impact of RMMs. However, the focus of this review was to give a snapshot of those studies submitted by pharmaceutical companies to the competent authorities as a part of a regulatory requirement and included in the EU-RMP. Therefore, we are confident that our sample provides a representative picture of the risk minimization evaluation strategies in place for the medicinal products authorized by the EMA in the designated therapeutic areas.

Conclusions

We have observed that studies measuring any single endpoint to assess the effectiveness of the RMM may have limitations, particularly uncertainties in predicting the overall success, inability to provide timely results, and feasibility issues. This indicates that due to the multilevel, multistakeholder nature of RMMs, no single methodology may be sufficient for conducting a robust evaluation of program implementation and impact on safety outcomes.

Therefore, for efficient and effective evaluation, consideration should be given to employing a range of methods that generate scientifically rigorous information on different elements of the implementation of RMMs within an agreed timeframe. These include enablers and barriers for optimal program delivery and success, stakeholders’ knowledge, attitudes and perception of risk, and intended and observed clinical behavior, with the ultimate focus ideally on safety outcomes.

Moreover, to facilitate assessment of a risk minimization program, besides the need for well-defined RMMs with clear objectives, it is essential that pharmaceutical companies and regulatory authorities agree on measurable indicators of success to be included in the evaluation plan. Optimal risk minimization evaluation may be hampered by the limited information on drug exposure and clinical outcomes when the RMM is introduced, therefore early decisions are needed on the most appropriate reference values.

Most importantly, a multifocal, integrated approach to evaluation of RMMs should provide an efficient way to allow rapid and timely assessment of decision-relevant information, which is necessary to modify RMMs and improve implementation at early stages, if needed, to further strengthen patient protection.

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 42 kb)

To measure the impact of pharmacovigilance activities, we reviewed industry-sponsored studies evaluating the effectiveness of risk minimization measures (RMMs) received by the European Medicines Agency.

Few studies were designed to measure the impact of RMMs in reducing the occurrence of adverse drug reactions, or used an appropriate study design to evaluate their effectiveness.

Optimal evaluation may be hampered by the limited data available when the RMM is introduced, and by the time required to obtain this information.

Efficient evaluation may benefit from an integrated measurement of the different elements of the RMMs. This should help regulators to gain timely information and undertake prompt adjustment of risk minimization strategies as needed.