Measuring the impact of medicines regulatory interventions - Systematic review and methodological considerations.

AIMS: Evaluating the public health impact of regulatory interventions is important but there is currently no common methodological approach to guide this evaluation. This systematic review provides a descriptive overview of the analytical methods for impact research.
METHODS: We searched MEDLINE and EMBASE for articles with an empirical analysis evaluating the impact of European Union or non-European Union regulatory actions to safeguard public health published until March 2017. References from systematic reviews and articles from other known sources were added. Regulatory interventions, data sources, outcomes of interest, methodology and key findings were extracted.
RESULTS: From 1246 screened articles, 229 were eligible for full-text review and 153 articles in English language were included in the descriptive analysis. Over a third of articles studied analgesics and antidepressants. Interventions most frequently evaluated are regulatory safety communications (28.8%), black box warnings (23.5%) and direct healthcare professional communications (10.5%); 55% of studies measured changes in drug utilization patterns, 27% evaluated health outcomes, and 18% targeted knowledge, behaviour or changes in clinical practice. Unintended consequences like switching therapies or spill-over effects were rarely evaluated. Two-thirds used before-after time series and 15.7% before-after cross-sectional study designs. Various analytical approaches were applied including interrupted time series regression (31.4%), simple descriptive analysis (28.8%) and descriptive analysis with significance tests (23.5%).
CONCLUSION: Whilst impact evaluation of pharmacovigilance and product-specific regulatory interventions is increasing, the marked heterogeneity in study conduct and reporting highlights the need for scientific guidance to ensure robust methodologies are applied and systematic dissemination of results occurs.

Introduction

Prescribing medicines is the most common health intervention globally and the safe use of medicines is paramount to public health. An estimated 3.5% of hospitalizations in Europe are caused by adverse drug reactions (ADRs), and up to 10% of hospitalized patients experience an ADR during their hospital stay 1.

To minimize the risks from medicines, pharmacovigilance systems have been established to continuously monitor their safety. These regulatory systems are designed to detect changes in the benefit–risk balance of a medicine which only become apparent during routine clinical use. Once safety signals have been evaluated and confirmed, appropriate regulatory action is taken to minimise the risks, such as labelling change, restriction, contraindication or withdrawal of a product or class of products.

Pharmacovigilance activities include monitoring of the effectiveness of risk minimization measures. The European Union (EU) pharmacovigilance legislation aimed to strengthen these activities and was found to lead to faster changes to product labelling and the conclusion of safety referrals 2. However, despite the potential for large global public health consequences, there is limited evidence about the effectiveness and consequences of regulatory actions at the population level, particularly relating to public health outcomes. To address this knowledge gap, the European Medicines Agency's Pharmacovigilance Risk Assessment Committee (PRAC) adopted in 2016 a strategy 3 aiming to assess whether pharmacovigilance activities achieve their intended objectives and to identify areas where performance could be enhanced 4.

To achieve their desired effect, regulatory interventions are expected to lead to changes in knowledge, attitudes and healthcare practices of individuals (i.e. patients, consumers and healthcare professionals) and organisations. However, the possibility of unintended consequences remains if measures are not properly implemented, which may give raise to criticism.

Measuring the impact of pharmacovigilance interventions is challenging as treatment and disease outcomes often overlap, and there may be significant time lags until clinical effects are seen with many existing studies being ecological in nature. It can also be difficult to evaluate decisions relating to single products if use is low and potential clinical outcomes are rare or when multiple interventions occur simultaneously. Nearly 50 years after the creation of the first national programmes for pharmacovigilance 5 there are no established guidelines for measuring the impact of regulatory interventions on public health 6, 7, 8.

Studies evaluating the effectiveness of risk minimization interventions often rely on surrogate measures such as changes in behaviour or prescribing rather than actual health outcomes 9. For example, measuring drug usage in population‐based electronic health records as a surrogate for changes in morbidity or mortality was one of various methods recommended at an international workshop exploring methodologies for measuring the impact of pharmacovigilance activities 10. Heterogeneity in study design and method of analysis also mean that proper interpretation and comparisons between regulatory systems are difficult.

We performed a systematic review of studies measuring the impact of pharmacovigilance regulatory interventions worldwide to highlight their methodological challenges and inform the conduct and reporting of future studies.

Methods

Literature screening

A protocol for a systematic search strategy was constructed a priori to identify articles evaluating the impact of regulatory interventions on healthcare utilisation, health knowledge and behaviour, or health outcomes. The search was performed in MEDLINE and EMBASE using Medical Subject Heading (MeSH) terms and keywords related to impact research in pharmacovigilance, regulatory policy, health outcome research, risk assessment, effectiveness of risk minimisation, health behaviour and health outcomes. The database search was supplemented with hand searching of references from systematic reviews, including articles and other known in‐house sources (snowballing). The protocol is available in the public European Union electronic Register of Post‐Authorisation Studies (EU PAS Register®) under study number EUPAS21337.

http://www.encepp.eu/encepp_studies/indexRegister.shtml .

Study selection

Eligible articles were initially screened by title and abstract by one of three reviewers with experience in regulatory science and pharmacoepidemiology (T.G., D.M., A.P.; (Figure 1). In a second stage, the eligibility of articles was independently evaluated after full text review and, where disagreement was present, discussions between the three reviewers were held to reach consensus.

Figure 1

Literature search and systematic review strategy. #Known literature and relevant references of published systematic reviews were included. *Duplicates, abstracts, letters to editors, commentaries and articles analysing the impact of other interventions (i.e. process and health policy related) were excluded

Inclusion and exclusion criteria

Articles in English language published up to 31 March 2017 evaluating regulatory interventions for medicines for human use were included. Duplicates, abstracts, letters to editors, commentaries and articles analysing the impact of health policy changes and studies investigating the impact of pharmacovigilance processes were excluded. We defined a regulatory intervention as any regulatory action taken by an EU or non‐EU competent authority to safeguard public health in relation to the use of medicinal products, including label changes, risk communication to the public or healthcare providers, product‐specific additional risk minimization measures defined in Good Pharmacovigilance Practices module XVI 11, withdrawal or suspension of a marketing authorization.

Data extraction and analysis

A standardized data extraction form was applied to obtain the following information: publication title, year, regulatory intervention and date/period, data source, study design, country, analytical method, outcome measure and drug therapeutic class (anatomical therapeutic chemical code). In addition, key findings, conclusions and any limitations of the studies were captured to support the review process. Data extraction was performed separately by each reviewer.

To synthesize information on the methodology for impact measurement of the studies identified, we categorized studies into one of the following mutually exclusive groups based on study design and analytical approach: before–after time series (defined as an evaluation at three or more time points crossing the date of the regulatory intervention); before–after cross‐sectional study (defined as an evaluation at one point in time before and after the date of the regulatory intervention); single time point cross‐sectional study (defined by a single time point after the date of the regulatory intervention); cohort study; and randomized controlled trial.

Categorization of included variables

For descriptive purposes we defined seven categories of regulatory interventions: direct healthcare professional communication (DHPC), black box warning, product information update, regulatory safety communication (e.g. guideline update, public health advisory communication, safety communication on websites), other additional risk minimization measures (e.g. medication guide, pregnancy prevention programme, controlled distribution), product suspension/withdrawal, and others (e.g. change in legal status, pack‐size restriction).

Analytical approaches for each study design were categorized as follows: descriptive analysis (with or without statistical significance tests), regression‐based approaches as described in the literature including Poisson and logistic regression 12, interrupted time series (ITS) regression 13, Joinpoint regression 14, and others.

Outcome measures were categorized into three groups: i) drug utilization; ii) health outcomes; and iii) knowledge, behaviour and clinical practice. A descriptive analysis of included studies was undertaken based on the extracted study information.

Results

The systematic review identified 1246 articles of which 229 were eligible for full‐text review, and 153 articles met the inclusion criteria and were retained in the descriptive analysis (Figure 1).

Overview of studies

Out of 153 studies included in our analysis, 70 (45.8%) assessed the impact of regulatory interventions in the USA, 69 (45.0%) in Europe and 14 (9.2%) in the rest of the world. Analgesics and antidepressants were the most common therapeutic classes, each being evaluated in 27 (17.6%) studies, followed by blood glucose lowering drugs with 14 (9.2%), antipsychotics with 13 (8.5%), and retinoids for systemic use with 12 (7.8%) studies (Table 1). The most frequently evaluated single regulatory interventions related to the risk associated with paracetamol poisoning and overdose, the risk of suicide in children and adolescents treated with selective serotonin reuptake inhibitors (SSRIs) and the cardiovascular risks with thiazolidinediones.

Table 1

Proportion of impact research articles (n = 153) by anatomical therapeutic chemical (ATC) classes and geographic regions (left). The right side shows the evaluated regulatory intervention(s)

ATC class and region	Articles n (%)		Regulatory intervention evaluated
			DHPC	Black box warning	Product information update	Regulatory safety communication	Additional risk minimisation	Suspension/ withdrawal	Othere
Analgesics	27 (17.6)		1	‐	1	4	3	4	18
Europe	24 (15.7)		‐	‐	1	3	2	4	18
USA	3 (2.0)		1	‐	‐	1	1	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Antidepressants	27 (17.6)		1	17	7	22	2	‐	‐
Europe	8 (5.2)		1		3	6	‐	‐	‐
USA	15 (9.8)		‐	14	3	13	1	‐	‐
Rest of the World	4 (2.6)		‐	3	1	3	1	‐	‐
Blood glucose lowering drugs a	14 (9.2)		4	5	8	7	‐	2	‐
Europe	5 (3.3)		2	‐	4	3		2
USA	8 (5.2)		2	5	3	3	‐	‐	‐
Rest of the World	1 (0.7)		‐	‐	1	1	‐	‐	‐
Antipsychotics	13 (8.5)		3	5	3	7	‐	‐	‐
Europe	5 (3.3)		‐	‐	1	5	‐	‐	‐
USA	7 (4.6)		2	5	2	1	‐	‐	‐
Rest of the World	1 (0.7)		1	‐	‐	1	‐	‐	‐
Retinoids for systemic use	12 (7.8)		‐	‐	1	‐	11	‐	1
Europe	7 (4.6)		‐	‐	1	‐	6	‐	1
USA	3 (2.0)		‐	‐	‐	‐	3	‐	‐
Rest of the World	2 (1.3)		‐	‐	‐	‐	2	‐	‐
Hormonal contraceptives	5 (3.3)		‐	3	‐	2	‐	‐	‐
Europe	2 (1.3)		‐	‐	‐	2	‐	‐	‐
USA	3 (2.0)		‐	3	‐	‐	‐	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
NSAIDs	5 (3.3)		‐	‐	2	4	1	3	‐
Europe	5 (3.3)		‐	‐	2	4	1	3	‐
USA	‐		‐	‐	‐	‐	‐	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Propulsives b	5 (3.3)		4	1	3	‐	1	‐	‐
Europe	‐		‐	‐	‐	‐	‐	‐	‐
USA	5 (3.3)		4	1	3	‐	1	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Antihistamines	4 (2.6)		2	4	‐	‐	‐	‐	‐
Europe	‐		‐	‐	‐	‐	‐	‐	‐
USA	4 (2.6)		2	4	‐	‐	‐	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Cough and cold preparations	4 (2.6)		‐	‐	2	2	‐	2	‐
Europe	‐		‐	‐	‐	‐	‐	‐	‐
USA	4 (2.6)		‐	‐	2	2	‐	2	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Hormone replacement therapy	4 (2.6)		‐	‐	1	4	‐	‐	‐
Europe	4 (2.6)		‐	‐	1	4	‐	‐	‐
USA	‐		‐	‐	‐	‐	‐	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Antiasthmatics	3 (2.0)		1	1	3	2	1	‐	‐
Europe	‐		‐	‐	‐	‐	‐	‐	‐
USA	3 (2.0)		1	1	3	2	1		‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Psychostimulants c	3 (2.0)		‐	3	2	2	1	‐	‐
Europe	‐		‐	‐	‐	‐	‐	‐	‐
USA	3 (2.0)		‐	3	2	2	1	‐	‐
Rest of the World	‐		‐	‐	‐	‐	‐	‐	‐
Other drugs d	27 (17.6)		6	5	7	12	6	‐	1
Europe	9 (5.9)		3	‐	1	7	2	‐	‐
USA	12 (7.8)		1	5	2	4	2	‐	1
Rest of the World	6 (3.9)		2	‐	4	1	2	‐	‐
Total	153 (100)		22	44	40	68	26	11	20

Thiazolidinediones

Cisapride

Agents used for attention deficit hyperactivity disorder (ADHD)

Therapeutic classes with less than three studies identified were grouped together.

Other regulatory interventions include studies evaluating the impact of paracetamol pack size restrictions, the impact of a healthcare reminder system for patient monitoring, the impact of advice on the clinical management of drug poisoning and compliance with national guidelines for isotretinoin.

DHPC, Direct healthcare professional communication; NSAID, nonsteroidal anti‐inflammatory drug

The most commonly evaluated regulatory interventions included regulatory safety communications (28.8%) and black box warnings (23.5%, USA only). A quarter of studies evaluated DHPCs (10.5%) and other additional risk minimization measures (15.7%), including pregnancy prevention programmes. About 13% of studies evaluated the impact of pack‐size restrictions. Product withdrawals and individual product information updates were least frequently assessed (7.2% and 1.3% respectively). Seventy‐three studies (47.7%) evaluated the impact of a single regulatory intervention, whereas 80 studies (52.3%) looked at the impact of multiple interventions occurring simultaneously or over time.

Studied outcomes

Eighty‐four studies (54.9%) measured drug utilization patterns and only 42 studies (27.5%) evaluated health outcomes such as morbidity (e.g. reduction of disease or adverse reaction incidence), mortality (e.g. reduction in suicide rates), pregnancy related outcomes or changes in laboratory values as surrogate measure for health improvements as shown in Table 2. Among studies which evaluated health outcomes, a positive impact of the regulatory intervention was reported in 27 (64%) studies whereas 12 (29%) studies showed no or negligible effects and in three (7%) studies the results were inconclusive.

Table 2

Distribution of outcome measures evaluated in regulatory impact research (n = 153 articles)

Outcome measure	Articles n (%)	References
Drug utilization	84 (54.9)
Health outcomes	42 (27.5)
Mortality	20 (13.1)
‐Drug poisoning/overdose	12 (7.8)	15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
‐Suicide and self‐harm	7 (4.6)	27, 28, 29, 30, 31, 32, 33
‐Other	1 (0.7)	34
Hospitalization a	9 (5.9)	35, 36, 37, 38, 39, 40, 41, 42, 43
Risk incidence b	6 (3.9)	44, 45, 46, 47, 48, 49
Pregnancy related outcomes c	3 (2.0)	50, 51, 52
Adverse drug reaction(s) reporting	2 (1.3)	53, 54
Laboratory tests d	2 (1.3)	55, 56
Knowledge, behaviour or clinical practice	27 (17.6)

Hospital admission due to myocardial infarction, cancer, hip fracture, drug poisoning or overdose, pulmonary embolism, drug‐induced liver injury, child unsupervised ingestion;

Venous thromboembolism; breast cancer; opioid abuse, addiction or overdose; stroke; osteonecrosis of the jaw; depression;

Unplanned pregnancy, spontaneous or medically induced abortion, birth defect;

Serum glucose and lipid testing; change in mean HbA1c and fasting plasma glucose levels;

Twenty‐seven (17.6%) studies evaluated changes in patients' or healthcare professionals' knowledge and behaviour, or changes in clinical practice targeted by the regulatory intervention. Only a small number of studies examined unintended consequences of regulatory interventions such as switching of therapies or spill‐over effects (e.g. decrease of drug use in subpopulations not targeted by the regulatory action).

Study design, methodology and data sources

Over 80% of studies used a before–after design with 101 (66.0%) before–after time series analyses and 24 (15.7%) before–after cross‐sectional studies (Table 3). There was only one randomized controlled trial identified designed to evaluate the impact of interventions, and six cohort studies. Seven different analytical approaches were identified. The most commonly used analytical approach was ITS regression in 48 (31.4%) studies, with simple descriptive analysis in 44 (28.8%) and descriptive analysis with statistical significance tests in 36 (23.5%) studies as shown in Table 3.

Table 3

Overview of study designs and analytical approaches of the final list of articles (n = 153)

Design and analytical method	Articles n (%)	References
Before/after time series	101 (66.0)
Descriptive analysis only	21 (13.7)	17, 24, 26, 49, 54, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71
Descriptive statistics with significance test	12 (7.8)	19, 23, 50, 53, 72, 73, 74, 75, 76, 77, 78, 79
Interrupted time series regression	48 (31.4)	22, 29, 30, 31, 38, 44, 56, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120
Joinpoint regression	9 (5.9)	28, 37, 39, 121, 122, 123, 124, 125, 126
Poisson regression	5 (3.3)	18, 27, 32, 34, 46
Logistic regression	3 (2.0)	48, 127, 128
Other	3 (2.0)	21, 129, 130
Before/after cross‐sectional study	24 (15.7)
Descriptive analysis only	4 (2.6)	41, 42, 45, 131
Descriptive statistics with significance test	18 (11.8)	15, 16, 25, 33, 35, 36, 40, 43, 47, 132, 133, 134, 135, 136, 137, 138, 139, 140
Poisson regression	1 (0.7)	20
Logistic regression	1 (0.7)	141
Single time point cross‐sectional study	21 (13.7)
Descriptive analysis only	16 (10.5)	142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157
Descriptive statistics with significance test	4 (2.6)	158, 159, 160, 161
Other	1 (0.7)	162
Cohort study	6 (3.9)
Descriptive analysis only	3 (2.0)	52, 163, 164
Descriptive statistics with significance test	2 (1.3)	51, 165
Other	1 (0.7)	166
Randomized controlled trial	1 (0.7)
Logistic regression	1 (0.7)	167

Administrative claims databases and electronic health records databases were the main data sources used to measure impact (Figure 2). Among the research conducted in the USA, claims databases dominated the picture, being used in 26.1% of studies whereas, in Europe, claims databases and electronic healthcare records were used in similar proportions (13.7% and 15%). Other types of data sources relevant for impact research were questionnaires, medical charts, national registers (e.g. on birth, mortality, poisoning), national surveillance systems (e.g. USA Sentinel), national patient safety incident reporting systems or electronic prescribing systems. Figure 3 shows how study designs and analytical methods evolved over time with a significant trend of increasing use of ITS regression analysis (P = 0.003).

Figure 2

Types of data sources used for regulatory impact research (n = 153 articles)

Figure 3

Distribution of study designs (A) and analytical methods (B) in impact research over time (n = 153). *Includes randomized clinical trials and cohort studies. #P = 0.003 using chi‐squared test for trend

Discussion

Our systematic review aimed to describe studies measuring the impact of regulatory interventions with a focus on study designs, analytical methods, data sources and choice of outcome measures. We found a marked heterogeneity in published studies of regulatory interventions with variation by region, study design, analytical approach and main outcomes evaluated.

The published studies evaluated regulatory interventions in Europe and the USA in similar proportions, and both regions together accounted for the majority of the global literature in English language, potentially affecting the generalizability of results to other populations. This is also the case for studies conducted in the EU where the organization of healthcare systems varies markedly between countries and may affect results of impact research. An element of this variation may be the availability of large electronic data sources in some countries only where impact studies are feasible.

Although the number of identified studies was relatively large, the range of therapeutic classes subject to impact research was limited with several studies evaluating the same regulatory intervention (e.g. suicidality with SSRIs in paediatric patients, mortality risk of dementia patients treated with antipsychotics, cardiovascular risks with thiazolidinediones, mortality associated with paracetamol poisoning and overdose; Table 3). The latter is an early example of impact research evaluating the effects of legislation which reduced the maximum pack size of paracetamol containing medicines in the UK in 1998. Despite an apparent decrease in paracetamol‐associated mortality rates and hospital admissions, the public health impact of these observed changes remained unclear. The decline in mortality and hospital admissions had begun before the legislation and the variety of outcome measures and analytical approaches used made it difficult to determine whether the legislation has been a success 168.

Some articles focused on regulatory actions suggesting uncertain effects in several countries or that require adjustments in their implementation, as shown for isotretinoin pregnancy prevention programmes in Europe 169. It is unclear why only certain regulatory interventions have been evaluated. The choice of regulatory interventions evaluated might be driven by the higher public health importance (e.g. unintended pregnancy with teratogenic medicines), by the feasibility of studies using available data sources (e.g. availability of pharmacy dispensed prescribing data versus chemotherapy or biological agents) and by funding opportunities. To help assess the need for such studies, the PRAC has developed criteria for prioritizing impact research in areas where there is a need to generate additional data to monitor the impact of regulatory interventions, which are based on three pillars: the public health importance of the regulatory action; the potential impact on clinical practice; and whether the study will deliver decision relevant data 170.

The vast majority of studies included in our review were drug utilization studies and relatively few evaluated clinical outcomes. Whilst drug utilization provides proxy measures of impact, it is uncertain whether the changes in drug use translate into discernible clinical or public health benefits. In this regard, the actual consequences of changes in drug usage are often unknown or unintended consequences may occur. For example, research conducted in the USA and the Netherlands showed a decrease in SSRI prescriptions for children and adolescents after US and European warnings in 2003 about a suicide risk with antidepressant use in this age group. However, this decrease in use seemed to be associated with an unintended increase in suicide rates in children and adolescents due to untreated patients 27. In the same context, time‐series analyses of antidepressant prescribing in adults showed statistically and clinically significant spill‐over effects associated with the 2003 Food and Drug Administration public health advisory on antidepressant use in paediatric patients 44. In addition, the impact of pharmacovigilance on health outcomes is often more difficult to measure due to a lack of adequate data sources and the difficulty of proving a causal association between the observed changes and the regulatory intervention, particularly at product level.

The choice of study design and analytical approach varied. Guidelines describing the optimal study design and analytical approaches for evaluating the impact of pharmacovigilance are lacking. Whilst each situation may differ, studies estimating the net attributable impact of regulatory interventions require considering the target drug, clinical outcomes and the potential for switching therapies and unintended consequences. Few studies considered possible unintended consequences, such as the effect in groups not targeted by the intervention through age or disease risk, or measuring therapeutic alternatives that may be used as substitutes 131, 133.

Uncontrolled before–after cross‐sectional studies were used in 15.7% of studies and evaluated periods of time immediately before and after a regulatory intervention most commonly applying simple descriptive statistics (such as t test or chi‐squared test) to determine if changes were significant. Although this design requires less data collection, preintervention trends are ignored potentially leading to overestimate the effect of the intervention. Such tests also assume that data points are independent, which is often an incorrect assumption.

Before‐after time series was the most widely used study design to measure the impact of regulatory interventions. However, only 65 (42.5%) studies used statistical regression‐based approaches to determine significance. Although studies without regression modelling may be suitable for large immediate changes (e.g. product withdrawals) they risk producing spurious results when assessment is more subjective. ITS regression is a robust quasiexperimental design to evaluate longitudinal effects of time‐delimited interventions. Segmented regression analysis of interrupted time series data can be used to quantify the immediate change in outcome following an intervention, and changes in trend 171. However, ITS regression requires the date of the regulatory intervention to be prespecified, which can be difficult to define, particularly when implementation varies, and that autocorrelation is assessed. Furthermore, adequate power to conduct ITS regression requires sufficient data points as with all time series approaches, and changes may be influenced by other interventions occurring during the same time frame (e.g. media coverage). However, in most instances, these data were not fully reported.

In contrast, Joinpoint regression models plot trend lines at points where changes in prescribing or the incidence of an outcome have occurred. A potential advantage of such models is that the intervention date does not need to be prespecified, offering potential advantages if the implementation date varies or is unknown. All time series approaches measure associations rather than causation and due to the ecological nature of the study design are even more challenging to be applied to public health outcomes.

Approximately a third of the studies employed descriptive statistics providing only weak evidence to support a causal association, which in many cases will be considered inadequate.

Limitations

Not all impact research may have been published in scientific journals and a vast majority of studies is communicated within regulatory procedures but seldom published. Unpublished research was not captured by our search strategy and therefore not included in the review. There is also a risk of publication bias, reflected by the higher percentage of published articles that reported positive outcomes. Some articles might have been missed due to a lack of common definitions and consistent terminology to describe such studies. A previous review of the use of ITS methods in drug utilization research showed a large variation in the reporting of analytical methods 172, confirming our findings and the need for standardized reporting. Therefore, our search strategy was supplemented with references from published review articles and known in‐house literature. Although different study designs and analytical methods are described, there has been no assessment of the risk of bias and of the quality, which requires further review. The challenge of evaluating multiple coinciding interventions remains to be addressed and the effectiveness of individual regulatory measures may not be discernible other than by interventional study designs.

Conclusion

Despite their potential global impact, the effects of pharmacovigilance regulatory interventions remain largely unquantified. A collaborative effort is required among regulators, health technology assessment bodies, academia and industry to help define measurable public health outcomes including intended and unintended consequences of regulatory decisions at the population level. Guidelines on the reporting of such studies and research to establish the best methods to evaluate such interventions are required. Results of impact research should be systematically disseminated to increase knowledge on the effectiveness of regulatory interventions. The EU PAS Register®, a publicly accessible platform for observational post‐authorization research, could be used for this purpose.

Competing Interests

There are no competing interests to declare.

The authors thank Carla Alonso Olmo and Oscar Francos at the European Medicines Agency for the administrative support and help with reviewing the literature.

The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.