Did we see it Coming? An Evaluation of the Swedish Early Awareness and Alert System.

BACKGROUND: Early awareness and alert systems have been established in many countries but evidence on their ability to accurately prioritize new medicines (for early assessment) is limited.
OBJECTIVE: The purpose of this study was to assess whether the Swedish Early Awareness and Alert System identified and prioritized (i.e., produced early assessment reports for) new medicines that would go on to have substantial economic impact.
METHODS: We adapted a study design commonly used in the assessment of diagnostic test accuracy. The prioritization made by the Swedish Early Awareness and Alert System prior to marketing authorization comprised the index test and the national drug sales data in the second year post-authorization served as the reference standard. All initial marketing authorization applications for medicinal products processed by the European Medicines Agency between 2010 and 2015 (study population) were classified using the index test and the reference standard.
RESULTS: Two hundred and fifty-three new medicinal products processed by the European Medicines Agency comprised the study population. Of these, 71 were prioritized by the Swedish Early Awareness and Alert System and 21 were classified as having substantial economic impact. The sensitivity and positive predictive value were 76.2% and 22.5%, respectively. Subgroup analyses showed that the accuracy of prioritization, in terms of sensitivity, was 100% for antineoplastic/immunomodulating agents.
CONCLUSIONS: The Swedish Early Awareness and Alert System identified all new medicines that would go on to have substantial economic impact and prioritized most of these medicines. Our findings provide reassurance to decision makers who rely on the outputs of the Swedish Early Awareness and Alert System to keep informed about new medicines. Moreover, this study also provides valuable insights to stakeholders willing to establish or evaluate their own early awareness and alert activities and systems.

Key Points for Decision Makers

Introduction

As already stretched healthcare budgets are put under additional pressure with the introduction of new expensive medicines [1-4], payers are increasingly seeking greater value for their money [5-10]. In addition, timely access to new and meaningfully better medicines to improve patient outcomes is considered a priority in many countries [11]. The early identification of new emerging technologies prior to their introduction can enable timely assessments to be undertaken and activities established to facilitate rational uptake and utilization. However, scrutinizing the research and development pipeline to identify medicines that may impact healthcare is notoriously difficult and requires considerable expertise. Horizon scanning systems [also known as early awareness and alert (EAA) systems] [12] can provide the necessary support to health technology assessment bodies and payers.

Early awareness and alert activities are now recognized as a key element in processes for the introduction of new medicines [7, 13]. In view of this, the structure, processes, outputs, and impact of an EAA system should be regularly evaluated and tailored to meet customer needs [14, 15]. It is reasonable to assume that an established EAA system carries out evaluations of its work as part of improvement initiatives. In Sweden, for example, feedback from stakeholder interviews was used to improve both the process and outputs of the Swedish EAA System [16]. However, such evaluations are typically internal and of a qualitative nature. Furthermore, they may not be readily accessible to wider audiences, thereby hampering the opportunity to learn from experiences across systems. To date, EAA systems with published evaluations include the UK’s National Institute for Health Research Horizon Scanning Research and Intelligence Centre [17-19], the Austrian Horizon Scanning Programme in Oncology [20], and the US Agency for Healthcare Research and Quality Healthcare Horizon Scanning System [21].

The Swedish EAA System has been performing EAA activities at the national level since 2010. It was established to support the long-term planning of the county councils and to optimize the readiness of the healthcare system to the introduction of new medicines. A detailed description of the Swedish EAA System is available elsewhere [22]. Both regional and national decision makers use the outputs of the system, including early assessment reports produced for prioritized medicines, to keep informed and prepared for the introduction of new medicines. The purpose of this study is to assess whether the Swedish EAA System identified and prioritized new medicines that would go on to have substantial economic impact.

Methods

Study Design

We adapted a study design commonly used in the assessment of diagnostic test accuracy to compare an index test’s classification of a target condition with a reference standard’s classification [17]. For the purpose of our assessment, the prioritization (i.e., selection for early assessment) made by the Swedish EAA System comprised the index test and the national sales data served as the reference standard. All initial marketing authorization applications for medicinal products processed by the European Medicines Agency (EMA) from 1 January, 2010, to 31 December, 2015 (study population) were classified using the index test and the reference standard. New indications of already authorized medicinal products (extensions of indications) were not included. The end date for selecting the study population was chosen to provide at least 2 years of follow-up sales data on all medicines following marketing authorization.

Study Population

To compile the study population, all European Public Assessment Reports on medicines, authorized or refused, were obtained from EMA website [23]. We also retrieved information on all withdrawals of initial marketing authorization applications. All medicines processed from 1 January, 2010, to 31 December, 2015, were included, regardless if they were authorized or refused by EMA, or withdrawn by the manufacturer.

We excluded generics, biosimilars, vaccines, as well as known active substances with a history of use in clinical practice for the applied indication [e.g., asparaginase (Spectrila®) for use in acute lymphoblastic leukemia; metformin hydrochloride/saxagliptin hydrochloride (Komboglyze®) for use in type 2 diabetes mellitus]. Finally, we removed duplicates, such as multiple and informed consent applications. The selection process for the study population is presented in Fig. 1.

Fig. 1

Study population and early assessment report selection flowchart. EMA European Medicines Agency, EPARs European Public Assessment Reports, KASs known active substances, MA marketing authorization, NASs new active substances

Index Test

The Swedish EAA System identifies and prioritizes new medicines and new indications of already approved medicines that may have an impact on the healthcare system [22]. The following criteria are considered in the filtration and prioritization steps: the patient population size; burden of disease; budget impact; anticipated clinical benefits; level of innovation; organizational impact; impact on treatment guidelines; safety aspects; level of interest from media and patient organizations; anticipated sub-optimal market uptake; and relevance from a legal, ethical and/or political aspect.

For a prioritized medicine, an early assessment report, the starting point for the national process for managed introduction and follow-up of new medicines, is produced prior to regulatory approval. In addition to early assessment reports, the Swedish EAA System produces the following outputs: a horizon scanning database of identified new and emerging medicines, a list of prioritized medicines (including already produced as well as planned early assessment reports), and quarterly newsletters. As per agreement with its customers (i.e., the New Therapies Council and individual counties), the Swedish EAA System seldom produces early assessment reports for medicines with existing mechanisms of action for indications or for orphan medicines.

The decision to prioritize a new medicine (i.e., to produce an early assessment report for a new medicine) comprised the index test. We retrieved all early assessment reports produced from 1 January, 2010, to 31 December, 2015. Reports on extensions of indications of already authorized medicinal products were excluded (e.g., apixaban [Eliquis®] was only included once with no distinction given to its multiple indications). Furthermore, we excluded reports for medicines processed by EMA before 1 January, 2010, or after 31 December, 2015.

Reference Standard

National drug sales data were used to derive the reference standard in our study. The eHealth Agency records complete pharmaceutical sales data from all pharmacies, retailers, and wholesalers in Sweden [24]. We obtained aggregate monthly sales data on hospital sales and dispensed prescription medicines in ambulatory care (reimbursed expenditure and patient co-payment) from 1 January, 2010, to 31 December, 2017.

For the main analyses, we calculated sales in the second year (i.e., the second 12-month period following the date of EMA marketing authorization). We also conducted sensitivity analyses based on sales in the third year, restricting the study population to those medicines with sufficient follow-up time.

As total annual pharmaceutical sales in Sweden have been approximately €4 billion (Euro:Swedish Krona = 9.5, average 2015–17) for the past few years, we chose 0.1% of that value (€4 million in sales per year) as a nominal threshold point (cutoff). A new medicine that exceeded this threshold in its second year on the market was classified as having substantial economic impact on the healthcare system (positive reference test). In addition, recognizing the arbitrary nature of the chosen threshold, we conducted sensitivity analyses exploring how varying the cutoff would change our results.

Statistical Analyses

Baseline characteristics were analyzed using descriptive statistics. Categorical data were reported as proportions and continuous data as medians with interquartile ranges. The accuracy of prioritizations was summarized in a contingency table. Outcome statistics were reported as sensitivity, specificity, positive predictive value, and negative predictive value with exact binomial 95% confidence intervals. Sensitivity was defined as the proportion of prioritized medicines among all medicines exceeding the sales threshold. Specificity was defined as the proportion of non-prioritized medicines among all medicines below the sales threshold. Positive predictive value was defined as the proportion of medicines exceeding the sales threshold among all prioritized medicines. Negative predictive value was defined as the proportion of medicines below the threshold among all non-prioritized medicines.

In addition to varying the cutoff for the reference standard (€2–8 million in annual sales) and timing of sales (sales in the third year on the market), we conducted additional analyses to restrict the study population to the following subgroups: new active substances; orphan medicines; and antineoplastic and immunomodulating agents (based on the Anatomical Therapeutic Chemical classification).

Results

During the study period (2010–2015), we identified 462 medicinal products that were authorized or refused by EMA and an additional 64 medicinal products whose initial authorization application was withdrawn by the pharmaceutical company. After applying our exclusion criteria, a study population of 253 medicinal products was yielded (Fig. 1). Most commonly, in almost three out of four cases, the excluded medicines were either generics or known active substances intended for use in an already approved indication. All 253 medicinal products were identified by the Swedish EAA System.

During the study period, 104 early assessment reports were published (Fig. 1). Of these, 33 reports were excluded from our analyses [see Electronic Supplementary Material (ESM)]; most frequently for covering extensions of indications of already authorized medicinal products. This left us with 71 prioritized medicines (positive index test).

Table 1 provides a description of the total study population and the prioritized medicines. The majority of the medicinal products processed by EMA were classified as new active substances (86%) and/or were granted marketing authorization (83%); among the medicines prioritized by the Swedish EAA System these proportions were slightly higher at 93% and 90%, respectively. Antineoplastic and immunomodulating agents were the most common group of medicines overall (29%) and among the prioritized medicines (51%). Early assessment reports were published around 3 months ahead of the date of marketing authorization [median (interquartile range) number of days: 87 (40–187)].

Table 1

Characteristics of the study population and the medicines prioritized by the Swedish Early Awareness and Alert (EAA) System

Characteristic	Study populationa		EAA System prioritized medicinal products
	n	%	n	%
Medicinal products
New active substanceb	217	86	66	93
Known active substance	36	14	5	7
Total	253	100	71	100
Marketing authorization status
Authorized	210	83	64	90
Refused	14	6	4	6
Withdrawn	29	11	3	4
Orphan
Yes	63	25	14	20
No	190	75	57	80
Year
2010	21	8	9	13
2011	37	15	10	14
2012	34	13	14	20
2013	53	21	13	18
2014	52	21	12	17
2015	56	22	13	18
Therapeutic area based on the ATC classification system
Alimentary tract and metabolism	31	12	6	8
Blood and blood-forming organs	20	8	2	3
Cardiovascular system	15	6	5	7
Dermatologicals	5	2	1	1
Genito-urinary system and sex hormones	8	3	1	1
Systemic hormonal preparations, excluding sex hormones and insulins	3	1	0	0
Anti-infectives for systemic use	27	11	4	6
Antineoplastic and immunomodulating agents	74	29	36	51
Musculoskeletal system	7	3	3	4
Nervous system	21	8	4	6
Anti-parasitic products, insecticides and repellents	1	0	0	0
Respiratory system	10	4	5	7
Sensory organs	6	2	1	1
Various	16	6	2	3
ATC missing	9	4	1	1

ATC Anatomical Therapeutic Chemical, EMA European Medicines Agency

aMedicinal products authorized or refused by EMA and withdrawals of initial marketing authorization applications

bA chemical substance not previously authorized as a medicinal product in the European Union; or an isomer, mixture of isomers, a complex or derivative or salt of a chemical substance previously authorized as a medicinal product in the European Union but significantly differing in properties with regard to safety and efficacy from that chemical substance previously authorized

In total, based on analyses of sales data, we classified 21 new medicinal products as having substantial economic impact on the healthcare system (positive reference test). An overview of these medicines is provided in Table 2.

Table 2

Medicinal products with substantial sales in Sweden in the second year on the market

Medicinal producta	Therapeutic area	ATC	Common name	EMA authorization date	Prioritization dateb
Elocta®; Swedish Orphan Biovitrum	Hemophilia A	B02BD02	Efmoroctocog alfa	2015-11-19	.
Opsumit®; Actelion	Hypertension, pulmonary	C02KX04	Macitentan	2013-12-20	.
Betmiga®; Astellas	Urinary bladder, overactive	G04BD12	Mirabegron	2012-12-20	2012-08-24
Triumeq®; ViiV Healthcare	HIV infections	J05AR13	Abacavir sulfate/dolutegravir sodium/lamivudine	2014-09-01	.
Daklinza®; Bristol-Myers Squibb	Hepatitis C, chronic	J05AX14	Daclatasvir	2014-08-22	.
Sovaldi®; Gilead	Hepatitis C, chronic	J05AX15	Sofosbuvir	2014-01-16	2014-01-24
Harvoni®; Gilead	Hepatitis C, chronic	J05AX65	Ledipasvir/sofosbuvir	2014-11-17	.
Yervoy®; Bristol-Myers Squibb	Melanoma	L01XC11	Ipilimumab	2011-07-13	2010-08-19
Opdivo®; Bristol-Myers Squibb	Melanoma	L01XC17	Nivolumab	2015-06-19	2015-05-06
Keytruda®; Merck Sharp & Dohme	Melanoma	L01XC18	Pembrolizumab	2015-07-17	2015-01-21
Zelboraf®; Roche	Melanoma	L01XE15	Vemurafenib	2012-02-17	2012-02-02
Imbruvica®; Janssen-Cilag	MCL; CLL	L01XE27	Ibrutinib	2014-10-21	2014-08-28
Kyprolis®; Amgen	Multiple myeloma	L01XX45	Carfilzomib	2015-11-19	2015-10-07
Xtandi®; Astellas	Prostatic neoplasms	L02BB04	Enzalutamide	2013-06-21	2013-01-29
Zytiga®; Janssen-Cilag	Prostatic neoplasms	L02BX03	Abiraterone	2011-09-05	2011-10-18
Gilenya®; Novartis	Multiple sclerosis	L04AA27	Fingolimod	2011-03-17	2010-12-22
Otezla®; Celgene	Arthritis, psoriatic; psoriasis	L04AA32	Apremilast	2015-01-15	2014-12-09
Entyvio®; Takeda	Colitis, ulcerative; Crohn's disease	L04AA33	Vedolizumab	2014-05-22	2014-02-20
Cosentyx®; Novartis	Psoriasis	L04AC10	Secukinumab	2015-01-15	2014-12-09
Tecfidera®; Biogen	Multiple sclerosis	N07XX09	Dimethyl fumarate	2014-01-30	2013-02-15
Eylea®; Bayer	Wet macular degeneration	S01LA05	Aflibercept	2012-11-22	2012-05-18

ATC Anatomical Therapeutic Chemical, CLL chronic lymphocytic leukemia, EAA early awareness and alert, EMA European Medicines Agency, HIV human immunodeficiency virus, MCL mantle cell lymphoma

aMedicinal products with sales >€4 million in the second year on the market

bDate of early assessment report publication

A tabular comparison of the prioritizations made by the Swedish EAA System and the economic impact is provided in Table 3 with corresponding measures of prioritization accuracy in Table 4. The Swedish EAA System prioritized 16 medicinal products for a sensitivity of 76%. The other prioritized medicines (n = 55) were among the 232 medicinal products that did not exceed €4 million in sales (corresponding to a specificity of 76%).

Table 3

Tabulation by prioritization made by the Swedish Early Awareness and Alert (EAA) System and the economic impact

EAA System prioritization	Economic impact
	> €4 million	≤ €4 million	Total
Prioritized	16	55	71
Not prioritized	5	177	182
Total	21	232	253

Table 4

Measures of prioritization accuracy

Measure	%	95% CI
Sensitivity	76.2	52.8–91.8
Specificity	76.3	70.3–81.6
PPV	22.5	13.5–34.0
NPV	97.3	93.7–99.1

CI confidence interval, NPV negative predictive value, PPV positive predictive value

Subgroup and sensitivity analyses were generally in line with the main analyses. The accuracy of prioritization, in terms of sensitivity, was highest for antineoplastic/immunomodulating agents (100%) and new active substances (76%). Conversely, the lowest accuracy of prioritization was observed when using the lower sales threshold (65%). Detailed data are presented in the ESM.

Discussion

We found that the Swedish EAA System identified all new medicines that would go on to have substantial economic impact and prioritized (i.e., produced early assessment reports for) most of these medicines. The few non-prioritized medicines whose sales exceeded the threshold were identified by the system but not selected for an early assessment report because similar medicines had been already prioritized and/or marketed earlier (e.g., Sovaldi® had already been selected for a report prior to Daklinza® and Harvoni®, thus the latter two were not prioritized). Overall, these findings indicate that decision makers in Sweden have been accurately informed about impending new medicines prior to their launch. This awareness allowed, for example, for special activities to be set up to facilitate a rational and cost-effective introduction of new medicines into clinical practice both at the regional [25, 26] and national [27] level. Notably, all new medicines comprising the study population were identified by the Swedish EAA System. For many of the non-prioritized medicines, for which no early assessment report was produced, information was summarized and disseminated in other ways (e.g., a list of all identified orphan medicines is regularly updated and provided to both regional and national stakeholders).

Potential economic impact is however only one of many criteria that the Swedish EAA System weighs in the decision to prioritize a medicine. For example, factors contributing to the prioritization of obesity medicines [e.g., phentermine/topiramate (Qsiva®)] included the patient population size, safety aspects, and anticipated media interest. New medicines that are seen as innovative methods of treating a disease are also prioritized. Bezlotoxumab (Zinplava®) as passive immunization against Clostridium difficile toxins, ipilimumab (Yervoy®) for the treatment of advanced melanoma, olaparib (Lynparza®) as maintenance treatment of platinum-sensitive recurrent BRCA-mutated ovarian cancer, and ranibizumab (Lucentis®) for the treatment of visual impairment due to diabetic macular edema were primarily prioritized for this reason [16]. The fact that the sales figures do not necessarily reflect all prioritization criteria explains the relatively low positive predictive value (range: 12–35%), i.e., fewer than one in three prioritized medicines went on to have a substantial economic impact. The prioritization of such medicines may however be equally important. For example, new innovative methods to treat diseases may have systemic effects on the delivery of healthcare services and require preparation ahead of the introduction. Consequently, given the holistic approach to prioritization employed by the Swedish EAA System combined with their efforts to improve efficiency (e.g., the Swedish EAA System typically does not prioritize medicines with existing mechanisms of action for indications or orphan medicines), it is difficult to propose revisions to their working methods that would lead to improved sensitivity and positive predictive value.

As a byproduct of our analyses, we obtained an overview of all new medicines assessed by EMA during the period 2010–15. We identified 217 new active substances that were submitted to EMA during these years. One fourth of these were in the oncology/immunology field, and a considerable number of medicinal products were also introduced for metabolic (diabetes and rare diseases) and infectious diseases. Correspondingly, over half of the medicines prioritized by the Swedish EAA System were in oncology or immunology, which is similar to outputs from other EAA systems [19, 20].

Apart from providing an assessment of how the Swedish EAA System has performed so far in identifying and prioritizing new medicines that would have substantial economic impact, this study also contributes to the scarce literature on the evaluation of EAA activities and systems. As outlined in the EuroSCAN methods toolkit, evaluation is an important element of an established EAA system [14]. However, few reports on such evaluations have been published to date [15, 17–21, 28]. This could possibly be explained by a lack of scientific interest and involvement, but also by methodological challenges, including the definition of a reference standard to be used for comparisons. The ideal reference standard should be able to reflect the prioritization criteria used by an EAA system but it may not be feasible to obtain a single measure that captures the criteria in their entirety (e.g., the Swedish EAA System’s prioritization criteria range from “potential to affect treatment costs” to “legally, ethically, or politically interesting”).

In addition to sales data, we also considered using other variables, including the size of the patient population, health technology assessment recommendation, and level of innovation. However, a reliable estimation of the size of the patient population proved to be difficult primarily owing to a lack of data on patient counts, particularly for narrow indications. Results of health technology assessments are only available for medicines that are included in the pharmaceutical benefits scheme, but many of those prioritized are intended for hospital use only for which, during the study period, no national appraisal was made. Finally, we also explored approaches to measure the level of innovation but found no universally accepted standard that could be used in our study.

Using sales data as a reference standard appears to capture both directly and indirectly some of the prioritization criteria used, including the size of the patient population, the potential to clinically improve patient outcomes, and the potential to affect treatment costs. In addition, sales objectively reflect which new medicines are actually adopted in clinical practice. However, we encountered a number of limitations with using sales data as a reference standard. We acknowledge that our choice of the timing of sales (i.e., second and third year) and the cutoff value is a pragmatic approach to operationalize our analyses.

Measuring sales early in the product life cycle may underestimate the number of impactful medicines. Reasons for this include medicines with slower uptake and medicines initially marketed in narrower indications (e.g., non-vitamin K antagonist oral anticoagulants). We believe, however, that early sales are well suited to capture medicines with a strong immediate impact that warrant preparedness by the healthcare system. The prioritization accuracy was also sensitive to the cutoff values used in our analyses. In the main analyses, we used a pre-specified threshold of approximately 0.1% of yearly total pharmaceutical sales. While this may not seem like a high level, in absolute terms the impact can be substantial, especially for specialist medicines used in a limited number of hospitals. Moreover, the aggregate nature of sales data does not allow us to attribute sales to a given indication for medicinal products that are used in multiple indications.

We also excluded all new vaccines from our analyses as no complete sales data were available because of separate distribution chains. While no vaccines were prioritized during the study period, more recently, in 2017, the Swedish EAA System prioritized the herpes zoster vaccine Shingrix® [29]. Recent progress in vaccine research and development may translate into an increase in breakthrough therapies [30] that warrants the inclusion of vaccines in future evaluations.

Finally, there is a possibility that an early assessment report could have influenced sales as the two measures are not independent (e.g., taking into account the information provided by the Swedish EAA System, the New Therapies Council or individual counties may have recommended not to use a given medicine). Analyses of  information on positive and negative recommendations could have been a possible way to address this; however, national-level recommendations were not provided until recently and, moreover, it was not possible to systematically collect data on all decisions taken at a county level.

This study was carried out as an independent evaluation of the Swedish EAA System. While all authors have knowledge of this particular EAA system and the role of EAA activities in the process of introduction of new medicines in general, none of the authors has been involved in decision making on which medicines to prioritize. Furthermore, our study is the first evaluation of an EAA system to use sales data as a reference standard to define which new medicines impacted the healthcare system. Despite the aforementioned limitations of such a one-dimensional measure, we believe it is nonetheless of value to explore its utility and share the results of this study with stakeholders both in Sweden and internationally.

Conclusions

To summarize, our findings provide reassurance to decision makers who rely on the outputs of the Swedish EAA System. Moreover, this study also contributes to the scarce literature on evaluations of EAA activities and systems and thus provides valuable insights to stakeholders willing to establish or evaluate their own EAA activities.

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 35 kb)

Supplementary material 2 (PDF 49 kb)

Of the new medicines authorized by the European Medicines Agency from 2010 to 2015, only 10% exceeded sales of €4 million (0.1% of total pharmaceutical expenditure per year in Sweden) in the second year on the market.

Over the first years of work since its establishment in 2010, the Swedish Early Awareness and Alert System identified all new medicines that would go on to have substantial economic impact in the first 2 years post-authorization. For most of these medicines, an early assessment report was provided to decision makers ahead of the marketing authorization.