Quality of evidence considered by Health Canada in granting full market authorisation to new drugs with a conditional approval: a retrospective cohort study.

OBJECTIVES: This study examines the characteristics of studies that Health Canada uses to grant full marketing authorisation for products given a conditional approval between 1 January 1998 and 30 June 2017.
DESIGN: Cohort study. DATA SOURCES: Journal articles listing drugs that fulfilled their conditions and received full marketing authorisation, Notice of Compliance database, Notice of Compliance with conditions website, Qualifying Notices listing required confirmatory studies, clinicaltrials.gov, PubMed, Embase, companies making products being analysed, journal articles resulting from confirmatory studies.
INTERVENTIONS: None. PRIMARY AND SECONDARY OUTCOME MEASURES: Characteristics of studies-study design (randomised controlled trials, observational), primary outcome used (clinical, surrogate), blinding, number of patients in studies, patient median age, number of men and women.
RESULTS: Eleven companies confirmed 36 publications for 19 products (21 indications). Twenty-nine out of the 36 studies were randomised controlled trials (RCTs) but only 10 stated if they were blinded. Twenty used surrogate outcomes. The median age of patients was 56 (IQR 44-61). The median number of men per study/trial was 184 (IQR 58-514) versus women 141 (IQR 46-263).
CONCLUSIONS: Postmarket studies required by Health Canada had more rigorous methodology than those required by either the Food and Drug Administration or the European Medicines Agency. There were still deficiencies in these studies. The absence of blinding in the majority of RCTs may introduce bias in their results. The use of surrogate outcomes especially in oncology trials means that improvements in survival are not available. The relatively young age of patients, even for products for cancer, means that predicting how the elderly will respond is often unknown. The almost universal finding that men outnumbered women may make it hard to differentiate responses by sex. These results raise potential concerns about the quality of evidence that Health Canada accepts. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Independent analysis of clinical studies required by Health Canada to approve new drugs and new indications for existing drugs.

Publications resulting from studies confirmed by companies.

Small number of drugs and indications limit generalisability.

Health Canada has access to more information than is contained in publications resulting from clinical studies.

Introduction

Health Canada is the arm of the Canadian government that is charged with drug regulation and operates in much the same fashion in this regard as the Food and Drug Administration (FDA) or the European Medicines Agency (EMA). The usual pathway to get a new medicine approved for marketing in Canada is for the pharmaceutical company involved to file a New Drug Submission (NDS) including preclinical and clinical scientific information about the product’s safety, efficacy and quality and information about its claimed therapeutic value, conditions for use and side effects.1 The key clinical evidence establishing the safety and efficacy of the new drug comes from the pivotal trials that Health Canada defines ‘as trials of high scientific quality, which provide the basic evidence to determine the efficacy, properties, and conditions of use of the drug’.2 Health Canada then reviews the NDS and makes a decision about whether or not to approve the drug or in the parlance of the agency issue a Notice of Compliance (NOC).

Being confident about the quality of evidence that Health Canada uses in approving new drugs and new indications for existing drugs helps to ensure that physicians are comfortable prescribing, and patients are comfortable using medicines. Health Canada has signalled its intent to proactively release industry-funded clinical trial information on efficacy and safety at some point in the future,3 but at present, it treats this type of information as confidential business information and will only release it with the consent of the company, even if an Access to Information request has been filed. The Summary Basis of Decision (SBD) project, started in January 2005, was meant to improve transparency in the drug review process, and to provide physicians and the public with access to unbiased information regarding all medicines authorised by Health Canada for marketing.4 However, an analysis of the 161 SBDs issued up to the end of April 2012 found that these documents were highly inconsistent in their quality and frequently omitted even basic information such as patient trial characteristics and the benefits and harms of tested treatments.5 Unlike the FDA, Health Canada does not release the redacted reports from its scientific reviewers that evaluate the evidence in the clinical trials. Therefore, there is no direct access to clinical trial data from Health Canada to be able to determine the quality of that data that Health Canada assesses in granting an NOC. (An NOC is Health Canada’s term for market authorisation.)

In the absence of any information from Health Canada, the one indirect source for gaining a partial understanding into the quality of the evidence that Health Canada considers regarding efficacy is the confirmatory studies required when a drug or a new indication is approved through Health Canada’s NOC with conditions (NOC/c) guidelines. The NOC/c approval pathway is similar to the accelerated approval process used by the FDA.6 It was developed in 1998 in an effort to ensure that promising therapies for serious illnesses can reach Canadians in a timely manner7 and allows drugs on the market based on limited evidence about efficacy.8 As part of the NOC/c process, companies are required to conduct confirmatory studies to validate the initial, incomplete evidence about efficacy for their products. Should these postmarket trials not provide sufficient evidence of clinical benefit, the NOC/c could be revoked and the product removed from the market.9 Once these studies are completed and accepted by Health Canada, the product is given a full NOC. This paper analyses the information that was contained in journal publications that result from the confirmatory studies for the cohort of drugs that fulfilled their conditions. Specifically, the paper assesses the evidence about study and patient characteristics that Health Canada evaluates when it approves new medicines.

Methods

Sources of data

A list of all drugs with an NOC/c and those that fulfilled the conditions from the time that the programme started, 1 January 1998 to 30 June 2017, was compiled from four sources: articles by Lexchin9 and Law10 that listed NOC/c and investigated whether they had been fulfilled, the Notice of Compliance database (http://webprod5.hc-sc.gc.ca/noc-ac/index-eng.jsp) and the Notice of Compliance with conditions website (http://www.hc-sc.gc.ca/dhp-mps/prodpharma/notices-avis/conditions/index-eng.php). In addition, the generic and brand-names of the drugs, the approved indication for the drug, the dates the drug received its NOC/c and the date it fulfilled its conditions and the names of the companies marketing them were recorded. Drugs could receive an NOC/c for more than one indication and each drug–indication combination was treated separately. The total number of new drugs approved in the same time period came from annual reports issued by the Therapeutic Products and the Biologics and Genetic Therapies Directorates of Health Canada (available from: publications@hc-sc.gc.ca).

The conditions that companies have to fulfil are outlined in a qualifying notice (QN). The QNs for products that had fulfilled their conditions were obtained from either the NOC/c website or by directly contacting Health Canada. The list and description of the confirmatory studies was abstracted from the QN. The QNs were independently screened by two people (the author and CO, a family physician) and disagreements resolved by consensus. Only confirmatory studies asking for information about efficacy/effectiveness were identified as these are the ones that healthcare practitioners would be most concerned about. There were no required confirmatory studies that focused solely on safety.

A web search was then undertaken in the final week of June 2017, to determine if a possible clinicaltrials.gov identifier and/or journal publication(s) could be identified for each study listed in the QN. Matches between studies required in the QN and journal publications or trials registered in clinicaltrials.gov were made on the basis of one or more of the following characteristics: generic name, number and particulars of trial participants (eg, women with breast cancer), primary outcomes and description of the treatment. If there was no journal publication given in clinicaltrials.gov then PubMed and Embase were searched to look for a journal publication. Terms used in the search depended on the level of detail in the QN about the required study. Descriptions of the required studies in the QNs were highly variable ranging from, for example, ‘data from the extension studies 105E2, 106E1’ to quite detailed, for example, ‘Study TMC435HPC3017 (planned n=300): A Phase III, multicenter, randomized, open label study to investigate the efficacy and safety of a 12- or 8 week treatment regimen of simeprevir in combination with sofosbuvir in treatment-naïve and -experienced subjects with chronic genotype 1 HCV infection without cirrhosis’. Only one QN included a clinicaltrials.gov ID for one study.

A letter was then sent to the companies making each product outlining the nature of the research, quoting a description of the confirmatory study or studies from the QNs, giving the possible clinicaltrials.gov ID (if one was found) and the possible publication (also if one was found) and asking the company to confirm that the publication corresponded to the study or if not, to provide a citation to a publication. After a month, a single reminder was sent to companies that did not respond. Only full journal publications that were confirmed by the companies were analysed in order to ensure that the publications accurately reflected the information in the confirmatory studies and that a full description of the study was available. Journal articles published up to 30 August 2017 were downloaded through the University of Toronto library website.

Analysis of journal articles

Information was extracted from the journal articles by the author and CO in two areas: (1) characteristics of the study—primary outcome (surrogate or clinical), number of patients in study arms (experimental therapy, active control, placebo control) and (2) study methodology (randomised, blinded, observational) and characteristics of the patients enrolled (median age and sex distribution). If studies for different drug-indication combinations resulted in the same journal article, the article was only counted once, however if there were more than one distinct publication for the same study, each publication was counted separately. Clinical outcomes were defined as ‘a characteristic or variable that reflects how a patient [or consumer] feels, functions, or survives’.11 Surrogate outcomes were defined as ‘a biomarker that is intended to substitute for a clinical endpoint. A surrogate endpoint is expected to predict clinical benefit (or harm or lack of benefit or harm) based on epidemiologic, therapeutic, pathophysiologic, or other scientific evidence’.11 The information from the articles was entered into an Excel spreadsheet, and descriptive statistics were reported using Prism V.7.0c (GraphPad Software).

Patient and public involvement

No patients were involved in any aspect of this study.

Ethics

No patients were involved in this study and only publicly available data were gathered. Therefore, ethics approval was not required.

Results

Between 1 January 1998 and 31 March 2017, a total of 65 distinct products received an NOC/c for 83 indications. Forty-four of those 65 drugs had never been marketed before with the result that 8.5% of the 517 drugs approved during this time were approved on the basis of an NOC/c. Thirty-four different products for 43 indications (1–2 indications per drug) made by 16 different companies fulfilled their conditions. Eleven companies responded and confirmed 37 unique publications for 40 studies listed in the Qualifying Notices. One company only confirmed one of three studies. In two cases, two studies were combined into a single publication. One publication was excluded because it was only available in abstract form, leaving 36 publications for 19 drugs (21 indications) for analysis (table 1).

Table 1

Drug-indication publications confirmed by companies

Generic name	Manufacturer	Indication(s)	Number of distinct studies required listed in the Qualify Notice	Number of distinct publications confirmed by company	Date drug received Notice of Compliance with conditions (year-month-day)	Date drug fulfilled conditions (year-month-day)
Abacavir	GlaxoSmithKline	HIV/AIDS	3	3	1999-04-06	2001-09-01
Alteplase	Hoffman-LaRoche	Stroke	2	2	1999-02-16	2005-01-26
Anastrozole	AstraZeneca	Breast cancer	3	1	2004-06-20	2008-12-02
Bortezomib	Janssen	Relapsed multiple myeloma	1	1	2005-01-27	2007-09-11
Capecitabine	Hoffman-LaRoche	Bowel cancer	1	1	2005-12-07	2008-10-23
Crizotinib	Pfizer	Lung cancer	2	2	2012-04-25	2015-11-18
Darunavir	Janssen	HIV/AIDS	2	2	2006-07-28	2009-02-11
Dasatinib	Bristol-Myers Squibb	Chronic myeloid leukaemia—two indications	3	3	2007-03-26 2011-07-19	2009-11-19 2015-11-26
Eculizumab	Alexion	Paroxysmal nocturnal haemoglobinuria	2	2	2013-03-01	2015-06-30
Exemestane	Pfizer	Breast cancer	3	3	2006-05-12	2008-08-06
Gefitinib	AstraZeneca	Lung cancer	4	3 (two studies published in one article)	2003-12-17	2009-12-18
Lenalidomide	Celgene	Myelodysplastic syndromes	1	1	2008-01-07	2013-06-06
Levodopa/carbidopa	AbbVie	Parkinson’s disease	2	2	2007-03-01	2014-03-12
Pregabalin	Pfizer	Central neuropathic pain	1	1	2007-11-09	2010-06-29
Recombinant factor VIIa	Novo Nordisk	Clotting disorders	1	1	1999-02-12	2005-06-19
Simeprevir	Janssen	HIV/AIDS	2	2	2015-01-30	2016-11-29
Sorafenib	Bayer	Kidney cancer	3	2 (two studies published in one article)	2006-07-28	2009-06-12
Sunitinib	Pfizer	Renal cancer—two indications	3	3	2006-08-17 2008-05-01	2010-04-23 2010-04-23
Zanamivir	GlaxoSmithKline	Influenza	1	1	1999-08-26	2003-08-26

(Online supplementary file 1 contains all of the data extracted from these 36 publications.) Eight companies that were required to undertake at least 48 confirmatory studies (the number of confirmatory studies for two products could not be determined) for 15 drugs (22 indications) either did not respond or publications could not be identified from the responses received (table 2). Some companies responded for one drug indication but not for another.

Table 2

No confirmation of publications for confirmatory studies

Generic name	Company	Drug indications where conditions fulfilled	Number of required confirmatory studies*	Date drug received Notice of Compliance with conditions (year-month-day)	Date drug fulfilled conditions (year-month-day)
Amprenavir	GlaxoSmithKline	HIV/AIDS—two indications	Unknown (all relevant material removed)	2001-03-01 2001-11-21	2004-07-05 2004-07-05
Anastrozole	AstraZeneca	Breast cancer	1 (company responded but did not confirm publication)	2004-06-20	2008-12-02
Aztreonam	Gilead Science	Cystic fibrosis	1	2009-07-17	2011-05-17
Dabrafenib	Novartis	Multiple myeloma	3	2015-03-06	2016-05-13
Deferasirox	Novartis	Transfusion-related iron overload	12	2016-02-14	2017-01-10
Deferasirox	Novartis	Thalassaemia	1	2006-10-18	2016-12-02
Delavirdine	Viiv Healthcare Canada	HIV/AIDS—two indications	6	1998-07-22 2000-04-25	2003-07-22 2003-07-22
Imatinib	Novartis	Gastrointestinal stromal tumour; chronic myeloid leukaemia—two indications	9	2001-09-20 2003-10-08 2007-04-24	2004-12-29 2010-06-17 2013-02-21
Letrozole	Novartis	Breast cancer; Breast cancer	3	2005-04-01 2006-10-06	2010-12-17 2010-12-17
Nevirapine	Boehinger Ingelheim	HIV/AIDS	Unknown (all relevant material removed)	1998-09-04	2004-09-13
Nilotinib	Novartis	Chronic myeloid leukaemia—three indications	3	2008-09-09 2010-07-22 2011-06-23	2011-11-30 2011-08-18 2015-08-19
Raltegravir	Merck Frosst	HIV/AIDS	2	2007-11-27	2009-03-04
Riluzole	Aventis	Amyotrophic lateral sclerosis	1 (Company letter said results of study disclosed in Product Monograph but letter and Product Monograph did not give any information about whether or not study was published)	2000-08-30	2007-11-29
Tenofovir	Gilead Science	HIV/AIDS	3	2003-03-18	2005-07-20
Trametinib	Novartis	Melanoma	3	2016-03-11	2016-05-13

*Based on information in Qualifying Notices (see text for definition of Qualifying Notice).

Twenty publications were for oncology products, 5 for products for HIV/AIDS and 11 for products with a variety of other indications (central neuropathic pain, clotting disorders, hepatitis C, influenza, Parkinson’s disease, paroxysmal nocturnal haemoglobinuria, stroke) (table 3). All of the publications for HIV/AIDS products came from observational studies, whereas 18 out of 20 for oncology products were RCTs. (Henceforth, unless specifically referring to RCTs, the generic term ‘studies’ will be used.) The split for products for other indications was almost even—five observational studies and six RCTs. Only 10 of the RCTs stated if they were blinded, 9 of the 19 RCTs for oncology products were not blinded, and five did not state if the trials were blinded. All of the five trials for HIV/AIDS products used active controls, as did 14 of 19 trials for oncology products and two of five for products for other indications. Studies were more likely to use surrogate outcomes compared with clinical outcomes (21–15) with all of the studies for HIV/AIDS products using the former. The use of the two different outcomes was mostly evenly split for oncology drugs: 11 surrogate versus 9 clinical, while for drugs for other indications it was four surrogate versus seven clinical. There were more patients enrolled in studies for oncology products (median 710 (IQR 270–1636)) than for HIV/AIDS products (median 562 (IQR 130–584)) or products with other indications (median 103 (IQR 41–671)). The lower number of patients in studies for products with other indications reflects the fact that some of these were rare indications—clotting disorders (36 patients) and paroxysmal nocturnal haemoglobinuria (22 and 41 patients).

Table 3

Characteristics of publications of confirmatory studies

Indication	All publications			Randomised trial publications		Outcome used in all publications		Number of patients per study/trial (median, IQR)
	Total number	Number observational (%)	Number randomised (%)	Number blinded (%)	Number with active controls (%)	Number surrogate (%)	Number clinical (%)
Oncology	20	2 (10)	18 (90)	5* (27.8)	14 (77.8)	11 (55)	9 (45)	710 (270–1636)
HIV/AIDS	5	0 (0)	5 (100)	2† (40)	5 (100)	5 (100)	0 (0)	562 (130–584)
Other‡	11	5 (45.5)	6 (54.5)	3 (50)	2§§ (33.3)	4 (36.4)	7 (63.6)	103 (41–671)
Total	36	7 (19.4)	29 (80.6)	10 (34.5)	21 (72.4)	20 (55.6)	16 (44.4)	541 (122–877)

*Five did not state if blinded.

†One stated ‘partially blinded’.

‡Central neuropathic pain, clotting disorders, hepatitis C, influenza, Parkinson’s disease, paroxysmal nocturnal haemoglobinuria, stroke.

§One randomised to two different durations of treatment.

The median age of patients was 56 (IQR 44–61) and greatest for oncology indications—median 59 (IQR 51–62) (table 4). There were only two studies where the median age was 65 or greater (stroke and myelodysplastic syndrome, median age 68). In all three indication groups, the number of men was greater than the number of women, and the median number of men per study/trial was 184 (IQR 58–524) versus women—median 141 (IQR 46–263). The above figures do not include three RCTs for exemestane and one for anastrozole for breast cancer that had only women (4724, 4724, 582 and 6186, respectively) and one RCT for recombinant factor VIIa that had only men (36). Out of the 28 studies that enrolled both men and women where the sex breakdown was given, only four had more women than men.

Table 4

Characteristics of patients in publications

Indication	Age of patients* (median, IQR)	Number of men (median, IQR)†	Number of women (median, IQR)†
Oncology	59 (51–62)	307 (131–681)	212 (100–281)
HIV/AIDS	35 (33–36)	470 (48–487)	75 (27–125)
Other‡	37 (27–58)	110 (36–311)	51 (20–243)
Total	56 (44–61)	184 (58–514)	141 (46–263)

*Excluded: no age given=9 studies, only mean age given=5 studies, not stated if age mean or median=2 studies, median age given for only part of study population=1 study.

†Excluded: women only=4 studies, men only=1 study, sex breakdown not given=3 studies.

‡Central neuropathic pain, clotting disorders, hepatitis C, influenza, Parkinson’s disease, paroxysmal nocturnal haemoglobinuria, stroke.

Discussion

Based on the results of the analysis of 36 publications resulting from confirmatory studies, the following characteristics of those studies were acceptable to Health Canada in granting full marketing authorisation: 29 (80.6%) were RCTs versus 7 (19.4%) that were observational studies, 21 (72.4%) of the RCTs had active controls, but only 10 (34.5%) of them definitely stated that they were blinded. Twenty (55.6%) of the 36 studies used surrogate outcomes, the median age of patients in all of the studies and in each of the three groups of indications was under 60 years of age and except for 4 (14%) out of 29 studies, men outnumbered women. These results compare favourably to those found by Naci et al12 in their examination of confirmatory studies required by the FDA where 10 (56%) out of 18 were RCTs, only 1 (6%) was blinded, 17 (94%) used surrogate outcomes and 7 (39%) had active comparators. Surrogate outcomes were used in 16 (27%) out of 59 postmarket studies that were requested for 21 conditionally authorised medicines by the EMA.13 The question remains as to whether the quality of the evidence that Health Canada accepts is rigorous enough.

Deficiencies in the data required by health Canada

Although the studies required by Health Canada had more rigorous methodology than those required by either the FDA or the EMA, there are still significant limitations in their design. Observational studies are useful for obtaining information about rare side effects but the confirmatory studies were meant to establish efficacy, not to search for rare side effects. Reliance on observational data to evaluate drug efficacy is problematic, given the inherent weaknesses of such studies14 and that the bias is, on average, often larger than the estimated effect.15

The lack of blinding or a statement about blinding in almost two-thirds makes it more difficult to eliminate or at least minimise potential biases. When researchers do not know to which group a subject has been assigned, they are less likely to influence the outcome of one group more than another. Blinding of study personnel assessing outcomes strengthens their objectivity, especially when the outcome is not a hard outcome such as death.16 Blinding of participants and research personnel is one of the criteria included in the Cochrane Handbook for assessing the risk of bias.17

All of the five studies on drugs for HIV/AIDS required by Health Canada used surrogate outcomes such as CD4 counts and suppression of viral load, and these have been accepted as valid markers for the effectiveness of pharmacological therapy,18 however the situation is more complicated for oncology where 11 out of 20 studies used surrogate markers. The FDA approved 54 oncology products between 1 January 2008 and 31 December 2012, with 36 made on the basis of surrogate markers. Eventually, five proved to have a survival benefit, the survival benefit was unknown for 13, and 18 had no survival benefit.19 Evaluation of surrogate markers shows that they are often biased, may overestimate drug benefits20 and correlate poorly with outcomes relevant to patients.21

The use of active controls in 21 out of 29 RCTs is encouraging but Health Canada’s position on the use of placebo controls is not clear. It has produced a guidance document for industry on the subject,22 but the document has no legal status as it is only intended to provide ‘assistance’ and is not part of the Food and Drugs Regulations.

The relatively young age of patients, even for conditions such as cancer and influenza where age is an important factor and where patients are likely to already be on multiple medications, means that predicting how the elderly will respond is often largely unknown when drugs are introduced. Health Canada issued a guidance document at the end of May 2013 on who should be included in clinical trials but it only had advisory status.23 The almost universal finding that men outnumbered women in the studies in some cases may reflect the preponderance of men with the diagnosis, for example, HIV, but in other cases, for example, bowel cancer, it may also be a result of the lack of a requirement for studies to include equal numbers of men and women and may make it harder to differentiate responses by sex.

Limitations

There are a number of limitations of this study. Only drugs approved under an NOC/c were considered. These were largely limited to a small number of indications, and companies responsible for drugs for 22 out of 43 indications did not respond. Had the Qualifying Notices issued by Health Canada contained more detailed information about the required postmarket studies, it may have been possible to identify more publications even without a response from the companies. All of these restrictions mean that the results about the study and patient characteristics cannot be generalised to drugs approved through either the standard or priority pathways or to drugs for other indications. Health Canada also has additional information available to it since publications only contain a small amount of the data that is in Clinical Study Reports (CSRs), the complete clinical trial dataset. Certain features of studies such as the median age of patients and whether or not RCTs were blinded that were not mentioned in publications may be in CSRs. However, this limitation would not apply to features such as whether the study was observational or an RCT or whether there was a surrogate or clinical outcome. Finally, the reports from Health Canada reviewers are not publicly available, so it is not possible to see whether they were concerned about the methodology employed in the studies and trials.

Conclusions

The results of this study pose potential concerns about the quality of evidence for efficacy for new drugs that Health Canada evaluates. Health Canada is currently developing regulations that will result in all of the clinical information regarding efficacy and safety data to be made publicly available with only minor redactions once products have been approved. The release of this type of information will either confirm the concerns about the quality of the data that Health Canada accepts or help to lay them to rest.