A framework for monitoring of new drugs in Sweden.

In order to monitor the net public health benefit of new drugs, especially in the light of recent stepwise approval approaches, there is a need to optimize real-time post-marketing evaluation of new drugs using data collected in routine care. Sweden, with its unique possibilities for observational research, can provide these data. We herein propose a framework for continuous monitoring of the effectiveness, safety, and cost-effectiveness of new drugs, using prospectively determined protocols designed in collaboration between all relevant stakeholders. We believe that this framework can be a useful tool for healthcare authorities and reimbursement agencies in the introduction of new drugs.

Introduction

Before the marketing of a new drug, its efficacy and safety are evaluated using randomized controlled trials (RCTs). RCTs are invaluable for establishing the relative treatment benefit of a new drug. However, the population eventually treated with a new drug often differs in many respects from the sample studied in the RCTs. In routine care, drugs are often used by elderly patients with more comorbidities and with multiple concomitant drug use. RCTs are also typically powered to detect only the most common adverse events. Consequently, rare or late adverse effects are not expected to be identified in RCTs. Hence, understanding of the realizable net public health benefit from a new drug requires other study designs than the RCT. Some of the data needed for such estimations may be available before the new drug is marketed, but other data are not available until the drug is actually used. Society needs these estimations as soon as possible, for determination of long-term safety, net public health benefit, and regulatory decisions. We herein propose a framework for such studies, using data collected in routine care.

Observational studies of comparative effectiveness and safety

Comparative effectiveness research (CER) involves the comparison of healthcare interventions, aiming to produce evidence regarding the effectiveness and safety of medical products (1). A drug’s efficacy is defined as ‘the extent to which a specific health intervention produces a beneficial result under ideal conditions’ (i.e. a response to the question: ‘Can it work?’), whereas effectiveness can be defined as ‘the extent to which a specific health intervention produced a beneficial result when deployed in the field under routine conditions’ (i.e. a response to the question: ‘Does it work in practice?’) (2). The ultimate goal of comparative effectiveness research is to improve health by developing and disseminating evidence to patients, healthcare professionals, and policy-makers regarding the effectiveness of specific interventions. The exponential developments in the quantity, quality, and availability of digital healthcare data generated in routine care hold great promise for development of observational CER in digitally mature countries. This development parallels a trend towards more use of adaptive licensing of new drugs (3), with healthcare authorities and regulators relying increasingly on observational data for evaluation of drug effectiveness and safety (4–6).

Unique possibilities in Sweden

Sweden provides unique opportunities for observational CER. The advantages lie in the country’s civic registration system involving a 12-digit personal identity number, unique to all Swedish citizens (7), and the fact that all residents have universal access to healthcare with a negligible co-payment for healthcare visits, hospitalizations, and drugs (8). Using the personal identity number to link healthcare data to a variety of nationwide health registers (classifying diagnoses using the International Classification of Diseases [ICD] system (9), surgical procedures using the Nordic Medico-Statistical Committee Classification of Surgical Procedures [NCSP] system (10), and filled drug prescriptions using the Anatomical Therapeutic Chemical Classification [ATC] system (11)), quality registers, and registers on sociodemographics and socioeconomics allows for research on large populations with near-zero loss to follow-up. Although many countries have registry data on prescription drugs at the individual level in ambulatory care, such data are often missing for drugs administered in hospitals (12, 13). Lack of individual-level data for hospital-based drugs is a large and increasing problem, since we are currently observing a trend towards more hospital-administered drugs.

Electronic health records (EHRs) provide an opportunity to include hospital drugs in observational CER. Sweden started to implement EHRs in the 1990s, and all regions had implemented EHRs in all healthcare areas by 2012 (14). Today, eight different EHR systems account for 97% of all EHR usage in Sweden, and most regions in Sweden have chosen to implement one single EHR system in their region (15). EHRs also contain more detailed clinical information than data from national health registries. Since the medical records are recorded as part of patient care, EHRs are instantly updated. This opens up for new and better opportunities to monitor treatments continuously with regard to its utility, safety, and cost-effectiveness.

In a PhD thesis, ‘Real-Time Monitoring of Healthcare Interventions in Routine Care. Effectiveness and Safety of Newly Introduced Medicines’ (16), we developed and validated a ‘sequentially evaluated non-randomized comparative effectiveness (SENCE)’ framework for continuous follow-up of new treatments using data from routine care in Sweden. This model is based on EHR data but can also utilize data from other sources. The model is built to collect and analyze data continuously as soon as new information is generated in the clinical data sources, making it possible to evaluate drugs sequentially and provide observational effectiveness evidence in as timely a manner as possible. The following is a summary of the proposed framework.

Proposal for a framework for continuous post-marketing monitoring of new drugs in Sweden

We have developed a generic sequential cohort model for real-time head-to-head (drug A versus drug B) comparisons of new drugs when used in routine care. We propose that this model is set up prospectively before a new drug enters the market, and sequential monitoring is launched when the drug is marketed. Other interventions than drugs may also be analysed. To maximize the credibility and utility of the results produced by the model, we propose that this framework is carried out in collaboration between all relevant stakeholders (regulators, payers, and manufacturers).

Process

Before the launch of a new therapy, regulators and authorities set the requirements for the post-marketing monitoring of the new drug. A project steering group and a scientific project group are formed. The steering group includes representatives from regulators, payers, and the drug manufacturer. The scientific project group includes experts in the actual therapeutical area, epidemiologists, and statisticians.

Initiation phase. In the initiation phase, a study protocol, statistical analysis plan (SAP), and an ethics committee application are developed by the project group. Permission to extract and include data is also obtained from data holders. A significant amount of work is dedicated to meticulously defining samples, data sources, exposures, best practice comparator drugs, covariates, and outcomes. A central task in this phase is the development of the causal models (17) and mimicking a target clinical trial (18). For development of causal models, we propose using the directed acyclic graphs (DAGs) approach (www.dagitty.net) in order to minimize potential bias (19). Statistical models used in sequential cohort design are defined (20–22). Before entering the data extraction and data management phase, all assumptions and the protocol are agreed upon between all stakeholders in the steering group, documented, and made available in the public domain (23).

The monitoring should begin when the drug is marketed. The data extraction and data management phase and the data analysis phase (see below) are repeated until stable results have been obtained.

Data extraction and data management phase. In this phase, data are extracted from EHRs and pseudonymized. Data from EHRs may also be linked to other data sources if necessary and defined in the initiation phase. All data should undergo quality checks including but not limited to logical checks, outlier detection, and investigation of missingness patterns. This step results in an analysis database and a data management report.

Data analysis phase. In the analysis phase, data are analysed according to the SAP. The model is sequentially updated in order to evaluate data as they are collected, at a frequency determined by the projected uptake of the drug on the market. In a proof-of-concept study, we updated the model every six months, but the model can be updated at any desired frequency. In each time period (recruitment cycle), new users of the new treatment (drug A) and new users of a comparator treatment (drug B) are included and added to the cohort to continuously increase the study sample size (Figure 1).

Figure 1.

In each recruitment cycle, new users of drug A and new users of drug B are included and added to the cohort to continuously increase the study sample size.

At the end of each recruitment cycle, a propensity score (the probability of treatment assignment conditional on observed baseline covariates) (24) is estimated for all patients, and all patients are assessed for study outcomes. At the end of each recruitment cycle, a comparative effectiveness and/or comparative safety analysis is carried out for the entire sample (all new users of drugs A and B over the whole time at risk) and plotted sequentially for each recruitment cycle (Figure 2). Results are continuously made available in a timely fashion in the public domain. In this observational monitoring framework, we have chosen not to account for the sequential nature of the analysis. The rationale for this is that the intention of the sequential analysis was never to point out a single significant estimate or to terminate the study when a satisfactory result was observed. As a result of this strategy, each confidence interval presented in Figure 2 is only valid one at a time.

Figure 2.

Effect estimates are published after each recruitment cycle.

The goal of using a propensity score is to achieve covariate balance between groups of treated and controls, which is a fundamental step in the proposed process. Several methods to assess balance have been proposed. One frequently used method is to estimate standardized differences and represent the number of standard deviations by which the two groups differ (25). We propose to visualize the standardized differences using the plot presented in Figure 3.

Figure 3.

Standardized differences in baseline covariates between new users of drug A and new users of drug B before and after adjusting on the propensity score. A standardized difference <0.1 indicates negligible imbalance.

The legitimacy of causal inference in observational studies is based on the assumption that no unmeasured confounding exists. This is a very strong assumption, and analyses should therefore be accompanied by sensitivity analyses investigating how the study findings may be affected by the presence of unmeasured confounding. We have in this framework included one commonly used approach proposed by Lin et al. (26) that involves evaluating how powerful an unmeasured confounder would have to be to change the observed results (Figure 4).

Figure 4.

Evaluation of how powerful an unmeasured confounder would have to be to change the observed results. For example, if the prevalence of a potential unmeasured confounder is 40% in the drug A group (x-axis) and 10% in the drug B group, then the unmeasured confounder must have a risk estimate (hazard ratio) of the outcome close to 3 to fully explain the advantage of drug A over drug B.

Conclusion

In order to ensure the effectiveness, safety, and cost-effectiveness of new drugs, especially in the light of recent stepwise approval approaches, there is a need for regulators, payers in healthcare, and the pharmaceutical industry to optimize real-time post-marketing evaluation of new drugs using data collected in routine care. Sweden, with its unique possibilities for observational research, can have a strong position in the post-marketing monitoring of new drugs. We suggest that the herein proposed framework can provide timely and comprehensive observational evidence of effectiveness, safety, and costs of new drugs, using a prospectively determined protocol designed in collaboration between all relevant stakeholders. We further suggest that analyses are repeated regularly as long as they provide value, and that all results are published without delay in the public domain. We believe that this framework can be a useful tool for healthcare authorities and reimbursement agencies in the introduction of new drugs.