Clinical trial registration, reporting, publication and FDAAA compliance: a cross-sectional analysis and ranking of new drugs approved by the FDA in 2012.

OBJECTIVE: To evaluate clinical trial registration, reporting and publication rates for new drugs by: (1) legal requirements and (2) the ethical standard that all human subjects research should be publicly accessible to contribute to generalisable knowledge.
DESIGN: Cross-sectional analysis of all clinical trials submitted to the Food and Drug Administration (FDA) for drugs approved in 2012, sponsored by large biopharmaceutical companies. DATA SOURCES: Information from Drugs@FDA, ClinicalTrials.gov, MEDLINE-indexed journals and drug company communications. MAIN OUTCOME MEASURES: Clinical trial registration and results reporting in ClinicalTrials.gov, publication in the medical literature, and compliance with the 2007 FDA Amendments Acts (FDAAA), analysed on the drug level.
RESULTS: The FDA approved 15 drugs sponsored by 10 large companies in 2012. We identified 318 relevant trials involving 99 599 research participants. Per drug, a median of 57% (IQR 32-83%) of trials were registered, 20% (IQR 12-28%) reported results in ClinicalTrials.gov, 56% (IQR 41-83%) were published, and 65% (IQR 41-83%) were either published or reported results. Almost half of all reviewed drugs had at least one undisclosed phase II or III trial. Per drug, a median of 17% (IQR 8-20%) of trials supporting FDA approvals were subject to FDAAA mandated public disclosure; of these, a median of 67% (IQR 0-100%) were FDAAA-compliant. 68% of research participants (67,629 of 99,599) participated in FDAAA-subject trials, with 51% (33,405 of 67,629) enrolled in non-compliant trials. Transparency varied widely among companies.
CONCLUSIONS: Trial disclosures for new drugs remain below legal and ethics standards, with wide variation in practices among drugs and their sponsors. Best practices are emerging. 2 of our 10 reviewed companies disclosed all trials and complied with legal disclosure requirements for their 2012 approved drugs. Ranking new drugs on transparency criteria may improve compliance with legal and ethics standards and the quality of medical knowledge. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This study uniquely analyses the transparency of clinical trial information for new drugs, whereas other studies analyse transparency on the trial level. It also debuts an innovative strategy for reforming areas of low transparency.

This study uses Food and Drug Administration (FDA) databases as a key data source, because they characterise all clinical trials supporting new drug approvals. Prior studies evaluate the transparency of already registered trials in ClinicalTrials.gov, providing limited insights into the many unregistered studies.

This study takes a uniquely comprehensive approach, analysing five critical elements of transparency for trials of new drugs: (1) registration, (2) results reporting, (3) publication in a medical journal, (4) compliance with legal disclosure requirements and (5) adherence with the ethics standards enshrined in the Common Rule, Helsinki Declaration and elsewhere which state that all trials should be ‘designed to develop or contribute to generalisable knowledge’, that is, should be publicly accessible.

A main limitation for this study is the need to extend and repeat the analysis beyond the 15 drugs approved by the FDA in 2012 that were manufactured by large companies, to include drugs approved in other years and sponsored by other institutions. We are in the process of expanding the rankings to include drugs approved in other years as well as additional trial sponsors.

Introduction

For decades, many clinical trials have been publicly inaccessible, raising ethics, medical practice and population health concerns. While recent transparency efforts have improved practices, a significant portion of both commercially and publicly funded trials and trial results still remain inaccessible, because they are unregistered and their results are unreported in trial registries,1 2 or they are never published in the medical literature.3 4

Studies have shown that roughly 30–50% of clinical trials remain unpublished, often years after their completion,5 6 and most fail to meet baseline legal disclosure requirements, such as those established in the 2007 US Food and Drug Administration Amendments Act (FDAAA).7 Moreover, studies that are published by sponsors, journals and researchers tend to show favourable or statistically significant results.8–11 This selective trial dissemination can distort the medical evidence and challenge physicians, prescription guideline writers, payers and formulary decision-makers’ abilities to recommend and provide the right drugs for the right patients. It also represents a violation of the rights of human research subjects, as experimenting on humans is largely justified by its potential to contribute to generalisable knowledge (as stated in the 1981 US Common Rule). Furthermore, transparency may be essential to ensuring the integrity and trustworthiness of the clinical research enterprise.

Despite numerous major reform strategies, the transparency problem persists, raising questions of what more can be done. Efforts include the 1997 US Food and Drug Administration Modernization Act requiring the registration of drug trials for serious or life-threatening conditions, FDAAA requiring that select trials be registered and publicly report results, and the 2008 World Medical Association guidelines identifying trial registration and results reporting as an ethical obligation in the Declaration of Helsinki. The International Committee of Medical Journal Editors, Institute of Medicine, individual drug companies and their trade associations, the European Medicines Agency, WHO and Bill and Melinda Gates Foundation have also made efforts to improve transparency in clinical research. Recently, both the Department of Health and Human Services and National Institutes of Health (NIH) called for public comment on two new proposals to further expand access to clinical trial information. The DHHS proposal would substantially expand the scope of registration and results reporting requirements under FDAAA.12 13 The NIH proposal would require registration and results reporting for all NIH funded clinical trials, including phase I trials.14

To help understand the efficacy of these transparency efforts for new drugs and how to improve them, this paper examines whether clinical trials for drugs approved by the FDA in 2012, which were sponsored by large companies, were registered, reported, published in the medical literature and complied with legal transparency requirements established in FDAAA.

This study and approach are novel for at least five reasons. First, we evaluated the transparency around individual new drugs. Previous studies generally evaluate transparency on the trial level. We thought evaluating on the drug level could help make the transparency problem more understandable and proximate for stakeholders who consume, prescribe, reimburse, stockpile or otherwise regulate medicines and vaccines. Moreover, drug level transparency evaluations are critical to improving clinical practice. When a new drug enters the market, the trials we evaluated in our rating system contain the safety and efficacy profile for that drug, and all, or nearly all, available evidence to inform clinical practice.

Second, we used FDA approval packages as a key data source, because they characterise all clinical trials supporting new drug approvals. Prior studies evaluate the transparency of already registered trials in ClinicalTrials.gov, which provide limited insights into the many unregistered studies. Third, we focused on large companies because, as a group, they sponsor a significant portion of the trials conducted annually and the majority of new drug applications (NDAs) submitted to the FDA. Also, they were expected to have the infrastructure to comply with regulatory and ethics standards. Thus, we likely captured a best-case scenario. Fourth, we evaluated transparency on both legal and ethics standards, providing a uniquely comprehensive overview. Lastly, we introduce an innovative strategy to improve the state of transparency for drugs: an annual transparency scorecard that audits and ranks all new medicines and vaccines.

Methods

Data sources

We used data collected from Drugs@FDA, a publicly accessible database containing records of FDA drug regulatory decisions, including drug approvals and medical and scientific reviews of approved drugs; ClinicalTrials.gov, a clinical trial registry and database maintained by the National Library of Medicine (NLM) at the NIH; MEDLINE-indexed journals (accessed through PubMed); information from the NLM to identify certificates of delay (provided by Tse T to Anderson M: personal communication); information from large companies that had new drugs approved by the FDA in 2012; and pharmaceutical company press releases. The databases were accessed several times between October 2013 and April 2014.

Study samples

Drugs

New drugs approved by the FDA in the calendar and fiscal year of 2012 were identified from FDA reports,15 16 and included innovative and novel drugs and new molecular entities (NMEs), henceforth referred to simply as ‘drugs’. We restricted the total number of drugs to those that were sponsored by large biotechnology and pharmaceutical companies, defined as the 20 institutions with the highest market capitalisations in 2012.17

Clinical trials

All trials conducted to gain FDA approval in 2012 for each drug

Each drug's 2012 FDA approval package was located in the Drugs@FDA database. We reviewed all pages of a drug's summary review, Medical Review(s), Chemistry Review(s), Pharmacology Review(s), Statistical Review(s), Clinical Pharmacology Biopharmaceutics Review(s), Risk Assessment and Risk Mitigation Review(s), and other reviews to create a list of every clinical trial reviewed by the FDA to approve each drug. Where possible, the basic characteristics of each trial were catalogued, including the organisational identification number, phase, study population, number of research participants, primary end point(s), study start and completion date(s), location, and description of the treatment (eg, dosage and comparators), participants received in the various arms, and whether the trial was controlled and/or interventional. We excluded any trials that were terminated without enrolment of participants, still ongoing or not at least 1 year past their primary completion date by our study cut-off date of 1 February 2014.

FDAAA applicable trials for each drug

We narrowed the ‘all trials’ sample to only those subject to mandatory registration and reporting requirements under FDAAA, that is, generally, ‘controlled clinical investigation(s), other than a phase I clinical investigation, of a drug subject to section 505 of the Federal Food, Drug, and Cosmetic Act or to section 351 of this Act’.18 These trials should have ‘either initiated after 27 September 2007, or initiated on or before that date and were still ongoing as of 26 December 2007’, and meet one of the following conditions:

The trial has one or more sites in the USA,

The trial is conducted under an FDA investigational new drug application (IND), or

The trial involves a drug or biological that is manufactured in the USA or its territories and is exported for research.19

Because of conflicting understandings on the reach of FDAAA, we created two sample pools of FDAAA-subject trials, one for controlled and one for interventional trials. Trials with unknown phases or that were listed as phase I/II in at least two primary data sources were excluded from the FDAAA analysis (n=1).

Main outcome measures

Determining transparency according to the ethics standard that all trial results should be publicly accessible

We ascertained in ClinicalTrials.gov whether each identified trial from the FDA approval packages of each drug (the ‘all trials’ study sample) was registered and reported results. Search terms to locate and match trials included the trial's organisational identification number (org ID), product name, number of trial participants and other trial characteristics captured from the approval packages. Once identified, we abstracted the National Clinical Trial number (NCT number), number of research participants enrolled in the various treatment arms, description of the treatment (eg, dosage and comparators), whether the trial was controlled and/or interventional, primary outcome measurements, trial start date, registration date, primary completion date (date the last participant was examined and data for the primary outcome measure collected) and any links to clinical study reports. Any clinical trial(s) with results received by ClinicalTrials.gov on or before 1 February 2014 was deemed to have results publicly available. This study cut-off date was chosen to provide at least 13 months for trial results disclosure post-FDA approval of a drug.

Second, using search terms that included the branded drug name or active ingredient and trial indication, we determined from PubMed whether individual trials for each drug were published in a MEDLINE-indexed journal, on or before our cut-off date. We then matched at least two of the following characteristics—the NCT number or organisational trial identification number, number of enrolled research participants, descriptions of the treatment (eg, dosage and comparators) and/or primary outcome measurements—in the publication with what was in ClinicalTrials.gov or the Drugs@FDA databases. We also reviewed papers listed on ClinicalTrials.gov for registered trials and used the same matching criteria. Papers summarising and reviewing multiple phase I trial results in a single publication, although rare, were counted.

Determining transparency according to FDAAA legal requirements

We reviewed whether FDAAA applicable trials (for both the ‘controlled’ and ‘interventional’ samples) had timely registration and reporting as defined by FDAAA. Registration (which in our case are trials for approved drugs), in general, should occur within 21 days after enrolling the first participant. Results should be reported, generally, no later than 12 months after the trial's primary completion date, in ClinicalTrials.gov, although results submissions can be delayed by submitting certificates to the NIH (see online supplementary appendices 1 and 2). If a trial met FDAAA requirements for both registration and disclosure of results, it was counted as compliant with legal requirements.

Validation

Data were extracted by at least two research assistants (working independently and blinded to each other's work), with discrepancies resolved through discussion and consensus (see Acknowledgments for a list of research assistants). Our final data sets for each drug were sent to each NDA company sponsor to verify the accuracy and completeness of our extracted information. Data and input received from companies (response rate was 100%) were verified by public data sources.

Statistical analysis

We used descriptive statistics to calculate the median number of clinical trials per drug that were registered and reported results in ClinicalTrials.gov, were published in a MEDLINE-indexed journal, and were publicly accessible. Public accessibility of a trial was defined as being either reported in Clinicaltrials.gov or published in a MEDLINE-indexed journal. We also used descriptive statistics to calculate the median number of clinical trials per drug subject to FDAAA that were in compliance with the statute. All data were collected and analysed in Microsoft Excel V.2013 (Redmond, Washington, USA).

Results

In 2012, the FDA approved 39 novel new medicines, known as NMEs, and 35 novel drugs. Combining these lists, the FDA approved a total of 48 new drug entities, 15 of which were sponsored by 10 large pharmaceutical or biotechnology companies with market capitalisations valued over $19 billion. A total of 342 trials were conducted to gain regulatory approval of the 15 drugs, 24 of which were excluded from our analysis, leaving 318 trials involving 99 599 participants relevant to our study, a median of 17 trials per drug (see table 1).

Table 1

Transparency Index: ranking of new drugs according to the ethics standard that all trial results should be publicly available to contribute to generalisable knowledge

Drug*	Company	Indication	Number of trials analysed from FDA* approval package	Percentage of trials registered	Percentage of trials reported	Percentage of trials published	Percentage of trial results that are publicly available (reported or published)
Stribild	Gilead	HIV	34	24	9	21	21
Aubagio	Sanofi	Multiple sclerosis	32	34	19	16	22
Elelyso	Pfizer & Protalix	Gaucher disease	5	100	20	40	40
Zaltrap	Sanofi	Colorectal cancer	30	40	30	37	40
Stivarga	Bayer	Colorectal cancer	12	75	17	42	42
Eliquis	BMS	Anticoagulant	39	26	10	44	44
Zioptan	Merck & Santen	Eye pressure, glaucoma	16	25	13	44	44
Xeljanz	Pfizer	Rheumatoid arthritis	34	82	53	56	65
Bosulif	Pfizer	Leukaemia	17	100	24	71	71
Perjeta	Genentech/Roche	Breast cancer	12	50	8	75	75
Signifor	Novartis	Cushing’s disease	17	29	12	82	82
Erivedge	Genentech/Roche	Basal cell carcinoma	12	83	25	83	83
Inlyta	Pfizer	Renal cell carcinoma	28	61	46	100	100
Sirturo	Janssen (J&J)	Tuberculosis	14	57	21	93	100
MenHibrix	GSK	Meningitis vaccine, children	16	100	100	100	100
Median			17	57	20	56	65
IQR			13–31	32–83	12–28	41–83	41–83

*For a list of the active ingredients for these drugs (see online supplementary appendix 2).

Transparency evaluated by the ethics standard that all trial results should be publicly accessible

The median proportion, per drug, of publicly registered trials was 57% (IQR 32–83%), of trials reporting results in ClinicalTrials.gov, 20% (IQR 12–28%), and of published trials, 56% (IQR 41–83%; see table 1). A median of 65% (IQR 41–83%) of clinical trial results were publicly available, that is, the results were either reported in ClinicalTrials.gov or published in the medical literature, but with considerable variation (see table 1). Importantly, among trials that reported results in ClinicalTrials.gov, a median of 100% (IQR 86–100%) were also published.

Among the 35% of trials, per drug, with results unavailable in either the medical literature or ClinicalTrials.gov, a median of 91% (IQR 60–100%) were phase I, 0% were phase II (IQR 0–15%), 0% were phase III trials (IQR 0–2%) and 0% were of unknown phase. Among the 15 drugs, 20% had at least one publicly unavailable phase III trial, 27% had at least one undisclosed phase II trial and 47% one of either. In total, 5566 research participants (of the 99 599 total participants) participated in publicly undisclosed trials for these 15 drugs.

Public availability of clinical trial information varied widely by company, and sometimes within companies for those with multiple drugs approved in 2012 (table 1). For example, 3 of the 10 companies (GSK, J&J and Pfizer) publicly disclosed all clinical trial results for at least one of their reviewed drugs, whereas the lowest scoring company, Gilead, disclosed 21% (7 of 34) of the trial results for its HIV medicine Stribild.

Transparency evaluated by FDAAA legal requirements

The legal requirements enshrined in FDAAA (table 2) offer at least two potential interpretations for what constitutes an applicable clinical trial: controlled and interventional trials. Applying the ‘controlled’ definition, a median of 17% (IQR 8–20%) of trials per drug were subject to legal disclosure requirements, hereafter referred to as ‘applicable trials’ .A median of 100% (IQR 93–100%) of these trials met registration requirements, whereas 67% (IQR 0–100%) met reporting requirements. Overall, per drug, a median of 67% (IQR 0–100%) of applicable trials fully complied with the law (see table 2), with considerable variation. Sixty-eight per cent of research participants (67 629 of 99 599) participated in FDAAA-subject trials, with 51% of them (33 405 of 67 629) enrolled in non-compliant trials. Six of the 10 reviewed companies showed 100% compliance with the law for at least one drug. However, an almost equal number, 5 of 10, had at least one drug that was 0% compliant.

Table 2

Legal Compliance Index: ranking of new drugs according to their compliance with disclosure requirements under the Food and Drug Administration Amendments Acts (FDAAA)

Drug*	Company	Indication	FDAAA definition 1: ‘controlled’ trials				FDAAA definition 2: ‘interventional’ trials
			No. trials subject to FDAAA	Timely registration (%)	Timely reporting (%)	FDAAA compliance (%)	No. trials subject to FDAAA	Timely registration (%)	Timely reporting (%)	FDAAA compliance (%)
Elelyso	Pfizer/Protalix	Gaucher disease	1	100	0	0	3	100	0	0
Stivarga	Bayer	Colorectal cancer	1	100	0	0	2	100	0	0
Perjeta	Genentech/Roche	Breast cancer	2	50	0	0	2	50	0	0
Signifor	Novartis	Cushing’s disease	1	100	0	0	2	100	0	0
Erivedge	Genentech/Roche	Basal cell carcinoma	2	100	0	0	3	100	0	0
Zioptan	Merck/Santen	Eye-pressure, glaucoma	7	17	17	17	7	29	14	14
Eliquis	BMS	Anticoagulant	6	83	33	33	6	83	33	33
Aubagio	Sanofi	Multiple sclerosis	7	86	71	71	7	86	71	71
Zaltrap	Sanofi	Colorectal cancer	6	100	67	67	9	100	78	78
Inlyta	Pfizer	Renal cell carcinoma	2	100	100	100	7	100	86	86
Stribild	Gilead	HIV	3	100	100	100	3	100	100	100
Xeljanz	Pfizer	Rheumatoid arthritis	11	100	100	100	11	100	100	100
Bosulif	Pfizer	Leukaemia	1	100	100	100	2	100	100	100
MenHibrix	GSK	Meningitis vaccine, children	3	100	100	100	3	100	100	100
Sirturo	Janssen (J&J)	Tuberculosis	1	100	100	100	2	100	100	100
Median			2	100	67	67	3	100	71	71
IQR			1–6	93–100	0–100	0–100	2–7	93–100	0–100	0–100

*For a list of the active ingredients for these drugs (see online supplementary appendix 2).

Applying the ‘interventional’ definition, a median of 19% (IQR 15–29%) of trials, per drug, were subject to legal disclosure requirements under FDAAA. A median of 100% (IQR 93–100%) of these trials met registration requirements, whereas 71% (IQR 0–100%) met reporting requirements. Overall, a median, per drug, of 71% (IQR 0–100%) of applicable trials complied with FDAAA (see table 2). Sixty-nine per cent of research participants (68 703 of 99 599) participated in FDAAA-subject trials, with 51% of them (33 786 of 68 703) enrolled in non-compliant trials. Five of the 10 reviewed companies had at least one drug that showed 100% compliance with FDAAA. The same number of companies (5 of 10) had at least one drug that was 0% compliant.

Discussion

Medical practice remains largely an empirical discipline, highly dependent for its advancement on the complete and accurate sharing of information. Nowhere is this truer than in the reporting of clinical trials, in particular those that support the efficacy and safety of new medicines. The purpose of this study was to review all new drugs approved by the FDA in 2012 that were sponsored by large companies, identifying all clinical trials that supported their approval and determining whether the trials were publicly registered and had trial results reported in ClinicalTrials.gov, were published in the medical literature within at least 13 months of FDA approval, and complied with federal disclosure laws. While nearly two-thirds of clinical trials, per drug, were publicly disclosed, there was wide variation among drugs and companies. At first approximation, it may seem difficult to understand failures to comply with federal law, now 8 years old, whose origins track back to 1997, and even more difficult to understand failures to meet the over-riding ethics obligation that human research be designed to contribute to generalisable knowledge.

Transparency by legal standards

There are at least three reasons why compliance with current disclosure laws might be suboptimal. First, legal requirements are perceived to be unclear or ambiguous, as a spectrum of interpretations of FDAAA has emerged. Some companies believe only controlled trials are subject to mandatory disclosure, others interventional trials. Some believe that results are due 1 year after a trial's primary completion date regardless of whether the drug has been approved, while others believe results are not due until 30 days post-FDA approval of a trial's investigated indication. There is also disagreement about the role of certificates of delay. These varying interpretations for FDAAA came to light during our discussions with the ranked companies. Recall that we sent all data to the companies whose products we scored (with a 100% response rate).

Second, mergers, acquisitions, collaborations and licensing agreements may complicate compliance. Two companies in our sample acquired or licensed drugs initially developed by smaller companies, and another used a partner company for some trials, raising questions about whose responsibility it was to ensure trials complied with FDAAA.

Finally, compliance may be affected by a perceived lack of enforcement. FDAAA empowers the FDA to impose a $10 000 a day penalty for non-compliance. To date, this penalty has never been imposed.

Transparency by ethics principles

In contrast to legal requirements, that applied to roughly only one-fifth of all clinical trials supporting a new drug approval, ethics standards enshrined in the Common Rule, Helsinki Declaration and elsewhere, apply to all clinical trials. Ethically, all research involving human participants should be ‘designed to develop or contribute to generalisable knowledge’,20 that is, should be publicly accessible. Surprisingly, adherence to this ethics standard was similar to that to legal standards: the results of approximately two-thirds of the studied trials, per drug, were publicly accessible, either through results reporting in ClinicalTrials.gov or publication. Adherence may be less than complete because companies may not act without an authoritative body promulgating an organising policy. Perhaps the new WHO guidelines calling for all trial results to be publicly disclosed, including phase I trials, may serve this purpose.21 Our proposed pharmaceutical transparency scorecard may also help.

Motivating transparency

Given the wide variation in compliance with both legal and ethical standards across drugs and companies, we propose continuing our clinical trials transparency monitoring, evaluations and scoring of new drugs approved by the FDA, along with their sponsors. These ongoing rankings—developed initially with support from Harvard University, Duke University, Susan G. Komen Foundation and the Raskob Foundation (for a full list of sponsors, see the Acknowledgements section)—will be conducted annually under the auspicious of Bioethics International, with grant support from the Laura and John Arnold Foundation.22–24

This system will help identify best practices, incent better behaviours and standardise the industry's practices and thereby contribute importantly to an enrichment of medical knowledge. Moreover, the scorecard and rankings have the potential to benefit consumers of clinical trial information by helping to assure them of the integrity and completeness of their data. Not least, full transparency of clinical trials would also strengthen the protection of human research participants by avoiding their unknowing recruitment into already failed experiments.

Limitations

Several limitations deserve further consideration. We limited our study to 1 year of FDA approvals: 15 drugs sponsored by 10 large companies. Further measurements are needed of trials for drugs approved in other years, of additional sponsors (such as smaller companies and academic centres), and of the quality of reported information. We are in the process of expanding the rankings to include drugs approved in other years as well as additional trial sponsors. At times, mergers, acquisitions, subsidiaries, partnerships and licensing practices complicated determining the sponsors and responsible parties for trial transparency. Lastly, there is some disagreement on whether the scores and index we presented should include a weighting mechanism to account for the variation in the number of trials conducted per drug to gain FDA regulatory approval.

Conclusion

Nearly two-thirds of clinical trials supporting new drugs approved by the FDA in 2012 were publicly disclosed, perhaps encouraging but below both legal and ethics standards. While several large companies’ drugs were superseding legal requirements, others had low rates of transparency. Implementing a transparency scorecard and ranking system for all newly approved drugs could motivate and increase transparency, thereby supporting existing transparency initiatives, advancing clinical innovation, promoting a trustworthy innovation sector and strengthening protection of human research subjects globally.