Health Canada's use of its priority review process for new drugs: a cohort study.

OBJECTIVES: Priority reviews of new drug applications are resource intensive and drugs approved through this process have a greater likelihood of acquiring a serious safety warning compared to drugs approved through the standard process. Therefore, when Health Canada uses priority reviews, it is important that it accurately identifies products that represent a significant therapeutic advance. The purpose of this study is to compare Health Canada's use of priority reviews to therapeutic ratings from two independent organisations, the Patented Medicine Prices Review Board (PMPRB) and the French drug bulletin Prescrire International, over the period 1 January 1997-31 December 2012.
DESIGN: Cohort study. DATA SOURCES: Annual reports of the Therapeutic Products Directorate, and the Biologics and Genetic Therapies Directorate; evaluations of therapeutic innovation from PMPRB and Prescrire International; WHO Collaborating Centre for Drug Statistics Methodology.
INTERVENTIONS: Assessments by PMPRB and Prescrire International treated as a gold standard for postmarket therapeutic value. PRIMARY AND SECONDARY OUTCOME MEASURES: Drug-by-drug comparison between the review status from Health Canada and the therapeutic status from PMPRB/Prescrire using κ values, and positive and negative predictive values. Analysis of the per cent of all new drug applications put into the priority review category over the 16-year period.
RESULTS: Health Canada approved 426 new drugs, and 345 were evaluated by PMPRB and/or Prescrire. 91 had a priority review and 52 were assessed as innovative (p=0.0003). Agreement between Health Canada and PMPRB/Prescrire was only fair (κ=0.330). The positive predictive value for Health Canada's review assignments was 36.3% and the negative predictive value was 92.5%.
CONCLUSIONS: Health Canada's assignment of a priority approval to a new drug submission is only a fair predictor of the drug's therapeutic value once it is marketed. Health Canada should review its criteria for using priority reviews. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Compares premarket priority review status from Health Canada to postmarket therapeutic status as assessed by two independent organisations.

Reviews a large number of drugs over a 16-year period.

Unclear exactly how Health Canada applies its criteria for a priority review.

Some drugs did not have a postmarket therapeutic assessment.

Postmarket therapeutic assessment is a gold standard.

Introduction

Health Canada's mission when it comes to medications is to “provide Canadians with…access to safe, high-quality, therapeutically effective…therapeutic products in a timely and cost-effective manner”.1 In part, to fulfil this mission, Health Canada has developed a priority review pathway to be used for drugs under two conditions: (1) for drugs that treat “a serious, life-threatening or severely debilitating disease or condition for which there is substantial evidence of clinical effectiveness that the drug provides…effective treatment…[and] for which no drug is presently marketed in Canada”, and (2) for drugs that represent “a significant increase in efficacy and/or significant decrease in risk such that the overall benefit/risk profile is improved over existing therapies…for a disease or condition that is not adequately managed by a drug marketed in Canada”.2 When companies file their new drug submissions they can apply for a priority review and Health Canada makes a decision about whether to grant this request within 30 days.

Medications that are given a priority review still need to complete all of the required clinical studies, but these are reviewed in 180 days rather than the standard 300 days. While these timelines are not always achieved in practice, there is still a substantial difference in review times for the two groups of drugs. For example, in 2012–2013, the median review time for new active substances (new molecules never before marketed in Canada) going through the standard approval process was 350 days compared to 210 days for priority review drugs.3

Previous work has shown that drugs that are approved after a priority review have a significantly greater chance of receiving a serious safety warning than those that underwent a standard review.4 The priority review pathway is also more resource intensive than a standard review since the same level of examination of material takes place, but in a significantly shorter period of time. To the extent that priority reviews are not properly assigned, then resources are being wasted and patient safety is potentially being endangered. Therefore, when Health Canada assigns a drug to a priority review, it is important that it accurately identifies products that represent a significant therapeutic advance.

The purpose of this study is to investigate whether granting a drug a priority review subsequently predicts its therapeutic value once it is marketed, by comparing Health Canada's assignment of priority reviews with the assessment of the therapeutic value of drugs produced by two independent sources. The two sources, the Human Drug Advisory Panel (HDAP) of the Canadian Patented Medicine Prices Review Board (PMPRB) and the French drug bulletin Prescrire International, were chosen as gold standards for determining the therapeutic value of new drugs for a number of reasons, including their extensive search for information about new drugs, their independence from industry and government, and their rigorous methodology.5 6 Second, the per cent of all new drug applications put into the priority review category, and the level of agreement between Health Canada and PMPRB/Prescrirer were examined over the 16-year time period to see if there were any temporal changes.

Methods

Data sources

Starting in 1997, the annual reports from the Therapeutic Products Directorate (TPD), and the Biologics and Genetic Therapies Directorate (BGTD) (available by directly contacting the directorates at ), the parts of Health Canada that regulate drugs and henceforth, collectively referred to as the TPD, began listing priority and standard review drugs separately. A list of all drugs approved from 1 January 1997–31 December 2012, their dates of approval and their review status was compiled from the annual TPD reports.

The PMPRB is a federal agency that is responsible for calculating the maximum introductory price for all new patented medications introduced into the Canadian market. It is important to note that the PMPRB is not a payer and therefore, its decisions about therapeutic value are not influenced by how much it might have to pay for the product. As part of the process of determining the price, the PMPRB's independent HDAP determines the therapeutic value of each product it reviews and these evaluations are published in its annual reports available online from 2003 to 2012 at and for previous years by directly contacting the PMPRB at . HDAP determines the ratings for the drugs before the maximum price is established. For the purpose of this study, products that were deemed as breakthrough and with substantial improvement were termed as ‘innovative’ and products in other categories were termed ‘not innovative’. In deciding on the level of therapeutic innovation, HDAP considers two primary factors: increased efficacy and reduction in incidence or grade of important adverse reactions, and 9 secondary factors: route of administration, patient convenience, compliance improvements leading to improved therapeutic efficacy, caregiver convenience, time required to achieve the optimal therapeutic effect, duration of usual treatment course, success rate, percentage of affected population treated effectively and disability avoidance/savings. The primary factors are given the greatest weight, followed by an assessment of any additional improvement as a result of the secondary factors.6 In some cases, the PMPRB annual reports indicated that the therapeutic value of the product was still being determined and for these cases, the PMPRB was contacted directly to determine the final classification.

Prescrire assesses the therapeutic value of medicines through a multistep process. First, it “examines the condition or clinical setting for which the drug is proposed; then the natural course of the disease, the efficacy and safety of existing treatments and the most relevant outcome measures. This is followed by a systematic search for clinical data on the efficacy and adverse effects of the new drug, and an assessment of the level of evidence. Based on [its] independent analysis of clinical data, [it] form[s] a judgement as to whether or not the new drug is beneficial for patients or whether or not its harmful effects outweigh the benefit”.7 Based on its analysis, it rates products using the following categories: bravo (major therapeutic innovation in an area where previously no treatment was available); a real advance (important therapeutic innovation but has limitations); offers an advantage (some value but does not fundamentally change the present therapeutic practice); possibly helpful (minimal additional value and should not change prescribing habits except in rare circumstances); nothing new (may be new molecule but is superfluous because it does not add to clinical possibilities offered by previously available products); not acceptable (without evident benefit but with potential or real disadvantages); judgment reserved (decision postponed until better data and more is available for thorough evaluation).8 The first three Prescrire categories were defined as ‘innovative’ and the other Prescrire categories (except judgment reserved) were defined as ‘not innovative’.

Health Canada does not give any details about the criteria it uses to assign a priority review, but as table 1 shows, the definitions used by Health Canada, the PMPRB and Prescrire are closely aligned.

Table 1

Criteria used by Health Canada in determination of priority review and by Human Drug Advisory Panel and Prescrire International in determining innovation status

Health Canada—criteria for priority review1 2	Human Drug Advisory Panel of Patented Medicine Prices Review Board—criteria for breakthrough and substantial improvement6	Prescrire International—criteria for bravo, a real advance and offers an advantage8
A serious, life-threatening or severely debilitating illness or condition for which there is substantial evidence of clinical effectiveness that the drug provides: effective treatment, prevention or diagnosis of a disease or condition for which no drug is presently marketed in Canada	Breakthrough=first drug product to treat effectively a particular illness	Bravo=major therapeutic innovation in an area where previously no treatment was availableA real advance=product is an important therapeutic innovation but has certain limitationsOffers an advantage=product has some value but does not fundamentally change the present therapeutic practice
A serious, life-threatening or severely debilitating illness or condition for which there is substantial evidence of clinical effectiveness that the drug provides: significant increase in efficacy and/or significant decrease in risk such that the overall benefit/risk profile is improved over existing therapies	Substantial improvement=provides a substantial improvement over existing drug products

The PMPRB does not review its decisions about therapeutic status whereas Prescrire will review its decisions in light of new evidence. Only the initial determination of innovation status by Prescrire was used in this study, but subsequent reviews by Prescrire of products were examined to see if the initial determination was revised.

Data analyses

For each year, the number of drugs receiving a priority review was calculated as a per cent of all drugs approved and additionally, the same calculation was done for the entire 16-year period. If a drug was judged innovative by either the PMPRB and/or Prescrire it was rated as innovative. If both organisations evaluated the drug and the ratings were discordant, that is, one said it was not innovative and one said it was, the drug was still considered innovative.

Two metrics were used to compare the review assignment by Health Canada with the evaluations by the PMPRB/Prescrire. First, all of the drugs evaluated by the PMPRB/Prescrire were considered and the per cent of these drugs judged as innovative was compared to the per cent of these drugs given a priority review by Health Canada and a χ2 test was used to see if there was any significant difference.

Second, the review status from Health Canada was compared to the assessments for the same drug by the PMPRB/Prescrire, on a drug-by-drug level in two separate ways. κ Values were calculated on a yearly basis and for the entire 16-year period. κ Scores measure whether there is more or less agreement between different evaluations than would be expected by chance. Levels of agreement were graded in accordance with the recommendations of Landis and Koch.9

Next, if there is a limited level of knowledge about the true effectiveness of a drug at the time that Health Canada makes its assignment of the review status then one would expect this relative ignorance should apply equally to drugs given a priority review and to those given a standard review; that is, Health Canada should be equally likely to correctly assign drugs to a priority review and to a standard review, and the positive and negative predictive values should be nearly equal. In this case, the positive predictive value measures the number of drugs evaluated as innovative by the PMPRB/Prescrire as a per cent of all drugs given a priority review by Health Canada. The negative predictive value is the number of drugs rated as not therapeutically innovative as a per cent of all drugs given a standard review by Health Canada. This hypothesis was tested by comparing the review status with PMPRB/Prescrire ratings on a drug-by-drug basis for the entire sample of drugs and then for a subgroup of drugs that were first in class, also on a drug-by-drug basis.

The rationale for the subgroup comparison is that when drugs enter an existing therapeutic class, the assumption may be that they are not therapeutically innovative and therefore, the negative predictive value will be higher. This concern is much less likely to apply to drugs that are first in class. First in class status was determined using the Anatomical Therapeutic Chemical (ATC) classification system from the WHO.10 The fourth level ATC group for each drug was determined by searching the web site of the WHO's Collaborating Centre for Drug Statistics Methodology. The fourth level is the chemical subgroup that the product belongs to. Drugs in the fourth level are listed in the order in which they appeared on the market and thus, the first drug will have the numeral ‘01’ at the end of its coding.

As a sensitivity analysis, when the ratings from PMPRB and Prescrire were discordant, the product was considered not innovative. The κ score for level of agreement between Health Canada and PMPRB/Prescrire was recalculated for the entire 16-year period, as were the positive and negative predictive values.

Second, the per cent of drugs given a priority review in each of the 16 years was calculated and graphed as was the κ value for each of the 16 years.

Calculations were done using Prism V.6.0 (GraphPad Software).

Results

A total of 426 drugs were approved by Health Canada between 1997 and 2012. Three hundred and forty-five of these drugs were evaluated by PMPRB/Prescrire and 52 (15.1%, 95% CI 11.68% to 19.23%) were rated as innovative. Health Canada gave a priority review to 91 of these 345 drugs (26.4%, 95% CI 22.01% to 31.27%) (table 2). There was a statistically significant difference at p=0.0003.

Table 2

Number of drugs approved by Health Canada and evaluated by the Patented Medicine Prices Review Board and/or Prescrire International, 1997–2012

	Health Canada		Patented Medicine Prices Review Board/Prescrire International			Agreement between Health Canada and Patented Medicine Prices Review Board/Prescrire International(κ)
Year	Number of approved	N (%) with priority review	Number of evaluated	N (%) assessed as innovative	N (%) given priority review
1997	42	8 (19.0)	24	4 (16.7)	4 (16.7)	0.400
1998	30	2 (6.7)	24	2 (8.3)	2 (8.3)	−0.091
1999	36	12 (33.3)	31	4 (9.7)	10 (32.3)	0.006
2000	26	11 (42.3)	20	4 (20.0)	9 (45.0)	0.468
2001	27	9 (33.3)	23	4 (17.4)	7 (30.4)	0.495
2002	24	4 (16.7)	20	1 (5.0)	4 (20.0)	−0.087
2003	20	7 (35.0)	19	2 (10.5)	8 (42.1)	0.336
2004	29	9 (31.0)	27	4 (14.8)	8 (29.6)	0.585
2005	24	10 (41.7)	21	4 (19.0)	8 (38.1)	0.106
2006	23	9 (39.1)	21	6 (28.6)	8 (38.1)	0.152
2007	24	7 (29.2)	22	4 (18.2)	7 (31.8)	0.488
2008	17	2 (11.8)	15	2 (13.3)	2 (13.3)	0.423
2009	27	6 (22.2)	26	3 (11.5)	6 (23.1)	0.081
2010	22	2 (9.1)	18	2 (11.1)	2 (11.1)	1.000
2011	35	7 (20.0)	20	3 (15.0)	4 (20.0)	0.828
2012	20	6 (30.0)	14	3 (21.4)	2 (14.3)	0.276
Total	426	111 (26.1)	345	52 (15.1)	91 (26.4)	0.330

Yearly κ values comparing Health Canada and the PMRPB/Prescrire on an individual drug level ranged from a low of −0.091 (95% CI −0.180 to −0.002) in 1998 to a high of 1.000 (95% CI 1.000 to 1.000) in 2010. κ Values were at or below 0.400 in 9 of the 16 years, that is, a level of agreement of fair or less in those years. The overall κ for the 16 years was 0.334 (95% CI 0.220 to 0.447) or fair (table 2).

Based on a drug-by-drug comparison for all drugs evaluated by the PMPRB/Prescrire, the positive predictive value of Health Canada's ratings was 36.3% (95% CI 26.6% to 47.1%); in other words, it gave a priority review to 91 drugs, but only 33 were evaluated as innovative. The negative predictive value of Health Canada's ratings was 92.5% (95% CI 88%.4 to 95.3%), that is, it gave a standard review to 254 drugs while 235 were assessed as not innovative (table 3).

Table 3

Comparison of Health Canada and the Patented Medicine Prices Review Board and/or Prescrire International on a drug-by-drug level, all drugs

	Patented Medicine Prices Review Board/Prescrire International
	Innovative	Not innovative
Health Canada
Priority review	33	58
Standard review	19	235

Positive predictive value=36.3% (95% CI 26.6% to 47.1%).

Negative predictive value=92.5% (95% CI 88.4% to 95.3%).

There were 33 drugs that were first in class. Comparing Health Canada's rating to the evaluations by the PMPRB/Prescrire, the negative predictive value was 89.5% (95% CI 66.8% to 98.4%) and the positive predictive value was 57.1% (95% CI 28.9% to 82.2%) (table 4).

Table 4

Comparison of Health Canada and the Patented Medicine Prices Review Board and/or Prescrire International on a drug-by-drug level, first in class drugs

	Patented Medicine Prices Review Board/Prescrire International
	Innovative	Not innovative
Health Canada
Priority review	8	6
Standard review	2	17

Positive predictive value=57.1% (95% CI 28.9% to 82.2%).

Negative predictive value=89.5% (95% CI 66.8% to 98.4%).

Over the 16-year period, Health Canada gave a priority review to 111 (26.1% 95% CI 0.2217% to 0.305%) of the 426 drugs it approved, going from a low of 6.7% (2 of 30) in 1998 to a high of 42.3% (11/26) in 2000 (table 1). Based on a visual analysis of figures 1 and 2, there were no discernable trends, either positive or negative, in Health Canada's use of priority reviews or in its level of agreement with PMPRB/Prescrire, respectively.

Figure 1

Priority reviews as a per cent of all drugs approved.

Figure 2

Level of agreement between Health Canada and PMPRB*/Prescrire.

The ratings by PMPRB and Prescrire were discordant in 31 cases. When these drugs were evaluated as not innovative, that left 21 products where both organisations rated them as innovative (8 cases) where one assigned that rating and the other did not evaluate the product (PMPRB=4, Prescrire=9). In this case, the κ value comparing Health Canada and PMPRB/Prescrire for the entire 16-year period was 0.187 (95% CI 0.088 to 0.287) or poor. The positive and negative predictive ratios for Health Canada's rating assignments were 16.5% (95% CI 9.8% to 26.1%) and 97.6 (95% CI 94.7% to 99.0%), respectively.

Although there were multiple articles in Prescrire International about a number of products where new indications or new safety issues were discussed, Prescrire's therapeutic rating for the initial indication was not altered for any drug.

Discussion

The results of this study suggest that Health Canada's use of the priority review process was not a good predictor of the therapeutic value of a drug once it is on the market. Of the 345 drugs that were evaluated by the PMPRB/Prescrire, Health Canada gave a priority review to 91 whereas only 52 were judged to be innovative. The level of agreement between Health Canada and the PMPRB/Prescrire on a drug-by-drug basis as measured by the κ value was only fair. Health Canada was also much more accurate in assigning drugs to a standard review than it was to a priority review, as shown by a negative predictive value of 92.5% versus a positive predictive value of 36.3% on drug-by-drug basis for all drugs. When a more conservative measure of innovation was used (products assigned not innovative where the PMPRB/Prescrire ratings were discordant), Health Canada's predictive ability was even less accurate.

After drugs that were second or later entries into a class were removed, there was still a large difference between Health Canada's negative predictive value (89.5%) and its positive predictive value (57.1%). Therefore, the difference between the predictive values does not seem to be due to Health Canada's determination that follow-on drugs in a class are unlikely to be therapeutically innovative. Nor does the difference seem to be explained by different evaluation criteria used by Health Canada, PMPRB and Prescire. Although the three organisations are evaluating drugs for different purposes, review status in the case of Health Canada and therapeutic innovation in the case of PMPRB/Prescrire, table 1, shows that the criteria that they are using are quite closely aligned.

The number of drugs assigned a priority review by Health Canada may be due to pressure from the pharmaceutical industry, since user fees from the industry account for about 50% of the operating budget for the drug submission review process.11 However, if that explanation was true, then the number of priority reviews as a per cent of all drugs approved on a yearly basis might be expected to be increasing or to be at least stable, and this is clearly not the case (see figure 1).

Figure 2 shows that the level of agreement between Health Canada and PMPRB/Prescrire has been highly variable over the 16-year period. Reasons for the fluctuation in κ values are not apparent, but clearly Health Canada's predictive ability has not been improving.

One way to help determine in more detail how Health Canada makes its decisions to use priority review status would be to examine both the clinical trial reports that companies submit to Health Canada at the time that they apply for marketing approval and the analyses that Health Canada reviewers produce based on the clinical trial reports. At present, none of these documents are available as Health Canada claims that they contain confidential business information and so can only be released with the approval of the company in question.12

Another explanation for the poor predictive value of priority review status could be because the PMPRB/Prescrire assess drugs at a later stage in their lifecycle whereas Health Canada assigns the review status on the basis of a preliminary evaluation of just the premarket clinical trials. Another possibility is that the PMPRB/Prescrire are too conservative in their assessments of therapeutic innovation. In contrast to the discrepancy between the evaluations from Health Canada and Prescrire, a comparison of the 1997–1999 postmarket therapeutic ratings by Prescrire and the premarket ratings by Swedish Medical Products Agency shows that it is possible for there to be good agreement between drug regulatory agencies and organisations such as Prescrire. According to the conclusion of the study, the contents of reviews published by the two were ‘broadly similar’ with complete agreement on 40 of 54 (74%) of the drugs, although no κ value was given.13

Between 1997 and 2012, 111 of 426 (26.1%) new drugs approved by Health Canada had a priority review. During the same period, the US Food and Drug Administration (FDA) gave a priority review to 188 of the 429 new drugs (43.8%) it approved.14 15 This difference could be accounted for by a number of mutually non-exclusive causes. The FDA might be more willing to grant a priority review, while Health Canada might be more conservative or the difference might be because applications for approval are submitted to Health Canada, on average, 540 days later than they are to the FDA or the European Medicines Agency16 and therefore, Health Canada has more information about the drugs at its disposal.

This study has a couple of limitations. The major one is the assumption that the evaluations by PMPRB/Prescrire represent a gold standard in the evaluation of a drug's therapeutic value. While there is always a legitimate debate about therapeutic value, the rigorous processes that these organisations use to arrive at their conclusions and their independence give strong face validity to their assessments. Second, neither PMPRB nor Prescrire evaluated 81 of the 426 drugs approved by Health Canada. Had these drugs been evaluated the comparison between Health Canada and PMPRB/Prescrire may have been either better or worse.

Priority reviews should continue to be used even in the face of significant uncertainty about the ultimate therapeutic value of drugs under certain circumstances, for example, for diseases that are rapidly fatal or that quickly lead to irreversible morbidity. In these cases, there is a need to get drugs on the market as soon as possible. However, based on this study, Health Canada's assignment of a priority review is only a fair predictor that a drug will offer a significant therapeutic advantage once it is on the market. Priority reviews are more resource intensive and drugs with a priority review are more likely to acquire a serious safety warning than those with a standard review. For both of these reasons, Health Canada should review its criteria for awarding a priority review.