An analysis of redactions in Canada's Common Drug Review Clinical Review Reports and how they relate to the patients' voice.

IMPORTANCE: Canada's Common Drug Review (CDR) evaluates drug data from published and unpublished research, as well as input from patient groups, to recommend provincial coverage. Currently, the CDR process gives manufacturers the opportunity to redact information in the final publicly available report. Patients often have strong feelings regarding the efficacy, harms, health-related quality of life (HRQL), and cost associated with the drugs under review and their redacted data. Highlighting Canada's approach will hopefully build on the growing international concern regarding transparency of clinical study data.
OBJECTIVE: The purpose was to objectively examine and classify completed, publicly available CDR-Clinical Review Reports (CRR) for redactions, and compare them to the patients' reported interests as patient-centred outcomes.
METHODS: Two independent reviewers searched for and examined publicly available CDR-CRR from November 2013-September 2016 through the Canadian Agency for Drugs and Technologies in Health (CADTH) on-line database. Both reviewers separately classified the redactions and patient-reported interests into the following categories: efficacy, harms, HRQL and costs. All discrepancies were rectified by consensus involving a third reviewer.
RESULTS: Fifty-two completed CDR-CRR were reviewed. 48 (92%) included patient-reported interests and 40 (77%) had redactions classified in the following categories: efficacy (75%), costs (48%), harms (38%), HRQL (23%). 89% of redactions were outcomes identified as patient-reported interests (69% efficacy, 42% harms, 36% cost, 33% HRQL). When examining drug characteristics, biological agents were statistically associated with increased odds of redactions with respect to either efficacy (OR 3.4, 95% CI 1.0 to 11.6) or harms (OR 3.5, 95% CI 1.02 to 12.4) compared with non-biological agents.
CONCLUSIONS: Whether data from the CDR-CRR used in the decision-making should be fully disclosed to the public is controversial. Our findings suggest clinical data (efficacy, harms, HRQL) matters to patients and should be publicly available within the CDR-CRR. Canada trails Europe and the USA regarding the transparency of clinical study data. This lack of transparency relates to the patient voice, and limits movement towards patient-centred care and patient-engaged research, restricting real-world value measurement. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

To our knowledge, this is the only analysis of redactions present in Canada’s Common Drug Review Clinical Review Reports and provides empiric evidence for the growing international concern regarding transparency of clinical study data.

Our methodology accurately reflects the publicly available reports, with two independent reviewers having accessed and assessed the reports.

Our methods are simple and easy to replicate; it would be valuable to reproduce our study using data from other international drug authorisation agencies.

A limitation is we do not know the exact content redacted, nor the rationale behind the redactions.

Introduction

The approval of new drugs and technologies in health varies internationally; the three biggest agencies being the US Food and Drug Administration (FDA), European Medicines Agency (EMA) and Health Canada. They all strive to be objective, using both published and unpublished research in decision-making.

The FDA (USA) is a single legal authorising body, supervising both clinical trials and eventual market approval of drugs; no state-level input is considered in the decision-making process.1 In Europe, the EMA acts as a centralised regulatory authority of the European Union. It provides independent recommendations on drugs and the basis for market approval; however, the legal decision and final drug approval falls under the remit of the European Commission (for centrally authorised products) or the individual national competent authorities of the EU Member States (for nationally authorised products).2 In Canada, the system shares elements with both the FDA and EMA. Like the FDA, a centralised body, Health Canada, legally provides national market approval. Once a drug is approved, the system is more similar to the EMA. A centralised Common Drug Review (CDR), through the Canadian Agency for Drugs and Technologies in Health (CADTH), is completed to provide recommendations on whether the drug should be reimbursed through provincial public funding plans. The CDR aims to standardise drug evaluations for public coverage across the country,3 but the final decision to reimburse the cost of the drug remains provincial, irrespective of the recommendation of the CDR. Moreover, the provincial jurisdiction can request additional information or conduct further analyses of data through their own local advisory committees.

To facilitate public funding approval, CADTH prepares CDR Clinical Review Reports (CDR-CRR) with experts nationally (or internationally if needed). These evidence reports largely comprised unpublished clinical study reports (CSR) from the manufacturers, published clinical studies (when available), and cost and cost-effectiveness data relative to current accepted therapy.3 4 In addition, since 2010, patient input has been included,5 demonstrating CADTH’s commitment to patient interests and patient-engaged research. Their input shows that patients value the efficacy, harms, health-related quality of life (HRQL) and cost of these drugs.

The public availability of the CDR-CRR data, particularly the unpublished CSR data from the manufacturers, is controversial. Currently, the CDR process provides manufacturers the opportunity to redact information before the final report is made publicly available.3 Redactions are blatant, blacked out words and data. Although provincial decision-makers have full access to the unredacted report, the data used in the review (both published and unpublished) may be of interest to patients and the public to guide clinical care choices. This lack of transparency limits advancements in patient-centred care and patient-engaged research.

Transparency of drug approval data is an increasingly international issue. The EMA (Europe) and the FDA (USA) also face the challenge of disclosing data and results, while protecting commercially sensitive information. In 2016, the EMA granted public access to CSRs for new drugs, making it the first regulatory body worldwide to provide such broad access to clinical data. Public access to CSRs submitted by pharmaceutical companies will include information on the methods used and unpublished results of clinical trials.6 The FDA allows the review of non-published data included in new drug applications by request.7 Canada currently does not provide any such mechanisms to review unpublished data.

Thus, we aimed to objectively examine and classify completed, publicly available CDRs for redactions and compare them to the patients’ reported interests as patient-centred outcomes. By highlighting Canada’s approach we aimed to build on the growing international concern regarding transparency of clinical study data.

Methods

We searched the CADTH on-line database for completed CDR-CRR between November 2013 and September 2016. All related submission documents, that is, recommendations, and clinical and pharmacoeconomic reports, were examined. Two independent reviewers initially assessed the reports for redactions and patient-reported interests, and HRQL measurements. Both reviewers separately classified the redactions and patient-reported interests into the following categories: efficacy, harms, HRQL and costs (table 1). As the CDR-CRRs are highly structured into specific sections, utilisation of table and figure labels, report sub-headings and surrounding text enabled the identification and classification of the redactions. Any discrepancies were discussed with a third investigator, and mutual consensus was reached.

Table 1

Examples of the classification of redactions and patient-reported interests content

Classification	Redactions	Patient-reported interests
Efficacy	Key efficacy outcomes Study elements ultimately affecting efficacy interpretations (ie, internal or external validity, outcome data, methodology and statistical analysis, etc)	Treatment success (eg, ‘Good control of chronic spontaneous urticaria with great reduction or eradication of symptoms’(p.3)12)
Harms	Withdrawal Due to Adverse Events, Adverse Events, Serious Adverse Events	Side effects (eg, ‘Several respondents noted that side effects seemed ‘much less severe’,…’(p.5)13)
HRQL	Health-Related Quality of Life measurement	Physical, Emotional and Social concerns (eg,‘… Patients endure severe inflammation, chronic pain, and fatigue, which affect every aspect of their day-to-day life (physical, social, and emotional), including concentration and cognitive abilities in class and a reduced ability to perform tasks such as tying shoe laces, pulling zippers, or completing basic household chores.’ (p.34)14)
Costs	Pharmacoeconomic data/analysis Price	Direct cost of treatment (eg, ‘Patients want access to affordable treatments…’(p.38)15)

The CDR-CRR documents were further investigated to describe the drugs containing redactions and those which did not contain redactions. We used the following categories to describe the drugs, study designs and/or populations included in the CDR-CRR: specialty product, active comparator trials, inclusion of paediatric populations and biological agent. These were chosen as they were elements common to all CSR-CRR reports. Specialty status was defined as high cost (>$500/dose or $6000/year), high complexity (physician specialist involvement and/or administration) and/or high touch (cold chain maintenance). Those reviews that included at least one active comparator trial were coded as active comparator. Those reviews that included at least one paediatric population sample (<16 years) were coded as paediatric inclusion. Any product with biological components was coded as biological agent. We also described the drugs when the CDR-CRR final recommendation noted cost as a concern.

Analysis

Basic descriptive analysis was completed using Microsoft Excel and STATA V.14 (StataCorp). The total number of redactions and patient-reported interests were reported in each category. Redactions were also evaluated according to the different characteristics associated with the drug (specialty product, biological agent, study designs and/or populations) as well as when the CDR-CRR final recommendation noted cost as a concern. For these analyses, univariate logistic regression was completed to evaluate the statistical association between the different characteristics and the odds of redactions being present in the CDR-CRR overall, as well as within individual categories (ie, efficacy, harms, HRQL, costs). A multivariate model, which included all characteristics, was also constructed to evaluate the independent effects of the different characteristics and their association with redactions. All data from the logistic regression models are reported as ORs and 95% CI. Finally, each category was further examined for parallels between redactions and patient-reported interests.

Results

We found 52 CDR-CRR completed between 1 November 2013 and 1 September 2016. Twelve reports did not have any redactions present; four reports did not have any patient-reported content. Of the 52, six were duplicates either due to resubmission or multiple clinical indication submissions; they remained included, as information presented in each CDR-CRR was different between submissions. Seventy-three total redactions were found within 40 of the 52 (77%) CDR-CRR reviewed. Of those with redactions, efficacy redactions (30, 75%) were most common followed by cost (19, 48%), harms (15, 38%) and HRQL (9, 23%). The majority (48, 92%) of the CDR-CRR had patient-reported interests with 164 total patient-reported interests documented—40 (85%) efficacy, 45 (94%) harms, 46 (96%) HRQL and 33 (69%) cost. Twenty-nine (56%) of the CDR-CRR had all four patient-reported interest categories present.

Redactions were similar across all our drug description categories, with efficacy occurring most often (specialty product (59%), active comparator trials (48%), paediatric population (67%) and biological agent (75%)) (table 2). With respect to costs, the CDR committee noted cost as a concern in their final report in half our reviews (26). Of those, 17 (65%) had efficacy and 11 (42%) had cost redactions present.

Table 2

Number (%) of redactions by drug descriptions

Drug descriptions	Any redaction	Cost redaction	Efficacy redaction	HRQL† redaction (n/N (%)	Harms redaction
Specialty product (n=29)	23 (79)	11 (38)	17 (59)	5/23 (22)	9 (31)
Non-specialty product (n=23)	17 (74)	8 (35)	13 (57)	4/11 (36)	6 (26)
OR (95% CI)	1.4 (0.37 to 4.9)	1.1 (0.37 to 3.6)	1.1 (0.4 to 3.3)	0.49 (0.1 to 2.4)	1.3 (0.4 to 4.3)
Active comparator in RCT (n=27)	21 (78)	12 (44)	13 (48)	3/13 (23)	8 (30)
Non-active comparator in RCT (n=25)	19 (76)	7 (28)	17 (68)	6/21 (29)	7 (28)
OR (95% CI)	1.1 (0.3 to 4.0)	2.1 (0.7 to 6.5)	0.4 (0.1 to 1.4)	0.75 (0.2 to 3.7)	1.1 (0.3 to 3.6)
Paediatric population (n=12)	10 (83)	6 (50)	8 (67)	3/9 (33)	3 (25)
Non-paediatric population (n=40)	30 (75)	13 (33)	22 (55)	6/25 (24)	12 (30)
OR (95% CI)	1.7 (0.31 to 8.9)	2.1 (0.6 to 7.7)	1.6 (0.4 to 63)	1.6 (0.3 to 8.3)	0.8 (0.2 to 3.4)
Biological agent (n=20)	18 (90)	8 (40)	15 (75)	4/15 (27)	9 (45)
Non-biological agent (n=32)	22 (69)	11 (34)	15 (47)	5/19 (26)	6 (19)
OR (95% CI)	4.1 (0.79 to 21.1)	1.3 (0.4 to 4.0)	3.4 (1.0 to 11.6)*	1.0 (0.2 to 4.7)	3.5 (1.02 to 12.4)*
CDR cost concerns noted (n=26)	22 (85)	11 (42)	17 (65)	5/20 (25)	5 (19)
Non-CDR cost concerns noted (n=26)	18 (69)	8 (31)	13 (50)	4/14 (29)	10 (38)
OR (95% CI)	2.4 (0.63 to 9.5)	1.7 (0.5 to 5.2)	1.9 (0.6 to 5.8)	0.8 (0.2 to 3.9)	0.4 (0.1 to 1.3)

*p<0.05.

†Restricted to only those reviews that stated HRQL was measured.

CDR, Common Drug Review; HRQL, health-related quality of life; RCT, randomised controlled trial.

In univariate analyses, only biological agents were statistically associated with increased odds of redactions with respect to either efficacy (OR 3.4, 95% CI 1.0 to 11.6) or harms (OR 3.5, 95% CI 1.02 to 12.4) compared with non-biological agents. This relationship for biologics persisted even after simultaneously adjusting for the different characteristics (ie, specialty, biologic, paediatric populations and active comparator trials) with statistically significant increased odds of redactions related to either efficacy (adjusted OR (aOR) 5.5, 95% CI 1.15 to 26.1) and harms (aOR 6.3, 95% CI 1.11 to 35.4) observed. No other characteristics were statistically associated with redactions in either univariate or multivariate analyses (table 2).

Thirty-six (69%) CDR-CRR had both redactions and patient-reported interests. When a redaction was present, 89% of the time it shared the same patient-reported interest category. There were 62 total parallels among the categories: 25 (69%) efficacy, 15 (42%) harms, 9 (25%) HRQL and 13 (36%) cost (figure 1).

Figure 1

Frequency and parallels of patient-reported interests and redactions. HRQL, health-related quality of life.

Discussion

In an era of increased emphasis on transparency and patient-centred care, it is surprising that redactions are so prominent (77%) regarding topics of patient interest in publicly available drug review reports. Patients clearly have a vested interest in drugs submitted for public funding and their associated use, including efficacy, harms and HRQL. In this review, we found redactions related to efficacy were most common (75%), particularly when the drug under review was a biological agent (over a fivefold increase in redactions compared with non-biological agents). Overall, redactions (89%) matched the same patient-reported interest category with efficacy again being the most common (69%).

It is documented that unpublished CSRs are vital to clinical decision-making, especially when directly comparing treatments.8 Wieseler et al found that a substantial amount of information on patient-relevant outcomes that was collected during trials was unavailable publicly. Their review showed that although trial publication and registry report rates are increasing, the rate of completeness of information on patient-relevant outcomes in these sources is not.8 They push for CSRs to be public, to enable anyone to fully evaluate the drug.8 They even suggest two cases in which decisions on benefits and harms may have changed if access to all clinical trials were made available to independent researchers, and subsequently to clinicians and patients.8 As health research moves towards more comparative effectiveness research, CSRs and their data will become essential sources to inform meaningful comparisons for researchers, clinicians and patients. The goal being not only to assess a single drug, but its position in a given therapeutic area.8 Bonini et al believe that the access and use of clinical trial data will help researchers perform unbiased reanalyses of data, in turn advancing science and help with policy and front-line clinical decision-making.9

A significant number of redactions matched patient-reported interest categories (89%). Why are the manufactures redacting information about efficacy, harms and HRQL? What are they protecting? One company claimed that information about harms ‘is confidential… because if released, other companies could use it to help them get products approved’.10 However, the flip side is should patients and their clinicians not be fully informed on potential harms? Since the marketing of the very first biological agents, there has always been concerns regarding the safety of these agents. Yet, in our review we observed that biologics are over six times more likely to have harms redacted which is concerning. There is considerable publication and outcome reporting bias present in clinical research,8 which consequently limits our revered evidence-based medicine. Christmas concludes that evidence-based medicine is not perfect and current systems do not offer enough protections from the influence of industry.10 Seeding trials, publication planning, messaging, ghost writing and selective publications and reporting of trial outcomes, distort the publicly available information.11 Evidence-based medical practice requires objective, unbiased research be accessible, not only to inform clinical decisions, but also to be used in systematic reviews, meta-analyses and guideline recommendations.11 CADTH and other agencies do use unpublished research to evaluate drugs, however, the redactions and unavailability of CSRs limits further independent research regarding the efficacy, harms and HRQL effects of drugs. All of these categories matter to patients.

It would substantially increase transparency to have all CSR data and reports (published and unpublished) made public. There are a number of initiatives worldwide to increase trial reporting, including AllTrials. The AllTrials campaign is supported by thousands of individual patients, clinicians, researchers and organisations working to report clinical trial data (http://www.alltrials.net). However, this type of initiative is voluntary. CADTH is in a unique position to be a world leader in creating policies that promote transparency, such as restricting manufacturers’ redactions to cost only. To better support informed decision-making in healthcare, whether by an individual patient, a healthcare professional or administrator, CSRs should be accessible to help make the best treatment choice.

Our unique review is not without limitations. Our sample size is 52, and not all had redactions and patient-reported interests. As a result, we had low power to observe statistical differences in our analyses related to drug characteristics and redactions. As well, CADTH releases documents in ‘batches’, so although our review examined completed reports between November 2013 and September 2016, possibly some reports were not yet publicly released. Perhaps most importantly, we do not know the reason behind the redaction. In some cases, lack of statistical or clinical important benefits or increase in harms cannot be ruled out. However, in other cases, redactions of publicly available data may have been required as authors pursue peer-reviewed publication of the data, which often limits prior disclosure of results.

Conclusion

Whether data from the CDR-CRR used in decision-making should be public is controversial. Our findings suggest clinical data (efficacy, harms, HRQL) matter to most patients and should be publicly available within the CDR-CRR. Canada trails Europe and the USA regarding the transparency of clinical study data. This lack of transparency relates to the patient voice, and limits movement towards patient-centred care and patient-engaged research, restricting real-world value measurement.