How transparent are migraine clinical trials? Repository of Registered Migraine Trials (RReMiT).

Transparency in research requires public access to unbiased information prior to trial initiation and openly available results upon study completion. The Repository of Registered Migraine Trials is a global snapshot of registered migraine clinical trials and scorecard of results availability via the peer-reviewed literature, registry databases, and gray literature. The 295 unique clinical trials identified employed 447 investigational agents, with 30% of 154 acute migraine trials and 11% of 141 migraine prophylaxis trials testing combinations of agents. The most frequently studied categories in acute migraine trials were triptans, nonsteroidal anti-inflammatory drugs, antiemetics, calcitonin gene-related peptide antagonists, and acetaminophen. Migraine prophylaxis trials frequently studied anticonvulsants, β-blockers, complementary/alternative therapies, antidepressants, and botulinum toxin. Overall, 237 trials were eligible for a results search. Of 163 trials completed at least 12 months earlier, 57% had peer-reviewed literature results, and registries/gray literature added another 13%. Using logistic regression analysis, studies with a sample size below the median of 141 subjects were significantly less likely to have results, but the dominant factor associated with availability of results was time since study completion. In unadjusted models, trials registered on ClinicalTrials.gov and trials with industry primary sponsorship were significantly more likely to have results. Recently completed trials rarely have publicly available results; 2 years after completion, the peer-reviewed literature contains results for fewer than 60% of completed migraine trials. To avoid bias, evidence-based therapy algorithms should consider factors affecting results availability. As negative trials are less likely to be published, special caution should be exercised before recommending a therapy with a high proportion of missing trial results.

Migraine is a common neurologic disorder that produces significant disability and reduced health-related quality of life. Population-based surveys and longitudinal studies report migraine prevalence ranging from 16% to 23% (17% in women and 6% in men).[1-3] The WHO ranks migraine 19th in terms of years lived with disability.[4] Epidemiologic studies suggest that episodes are frequent and severe enough to justify preventive therapy in 39% of migraineurs, but only 3%–13% use it.[1] Migraine therapy guidelines distinguish between treatment of acute migraine and migraine prevention.[5-11] Some acute therapy guidelines focus on specific clinical situations, such as treatment in the emergency department or use of opioid analgesics.[12,13] Evidence-based guidelines typically combine systematic literature reviews, grading of the available research data, and expert opinion.

In this project, the science behind migraine therapy is approached from a different perspective. A snapshot of the entire landscape of migraine clinical trials and a scorecard of publicly available results for completed trials reveals how much progress has been made toward transparency in research, i.e., the public can learn of the study before initiation and find results once the study is completed. The snapshot and scorecard approach assists guideline developers by estimating how much data are unavailable. The potential for bias in guidelines is reduced by determining study characteristics most strongly predicting availability of results.

The 1997 Food and Drug Administration (FDA) Modernization Act resulted in the creation of the ClinicalTrials.gov (CTG) registry,[14,15] and the subsequent FDA Amendments Act of 2007 (FDAAA) requires all FDA phase 2–4 biologic drug and device trials be registered on CTG within 21 days of study start date.[14,16] However, registration before subject enrollment commences is inconsistent in many areas of clinical research,[17-20] and registry fields are often missing or incomplete.[21-24]

Accessing results of completed trials is more difficult than accessing basic trial information. In 2005, the International Committee of Medical Journal Editors (ICMJE) set clinical trial registration as a precondition for publication in an ICMJE journal.[25] ICMJE guidelines are not always followed, and manuscripts reporting on unregistered studies can publish in one of the many journals not requiring preregistration.[26,27] Concerns about low publication rates and the lag time between study completion and final journal publication persist.[20,21,28-32] Publication bias, selective outcome reporting, and other types of reporting bias remain prevalent.[14,17,18,33-39] Discrepancies between the registry trial record, conference abstract, and final published article are disturbingly common, and often favor statistically significant results.[22,40,41] In the few instances where unpublished results are publicly available on the FDA's Web site, studies have demonstrated significantly altered efficacy estimates and risk/benefit ratios compared to the peer-reviewed literature.[35,39,42-44]

In 2008, the CTG Web site was reconfigured to allow basic trial results to be uploaded in a tabulated format.[31] No other large trial registry allows direct posting of study results. For certain categories of industry-sponsored trials, posting summarized results on CTG within 1 year of study completion, or within 30 days of FDA approval of the drug being studied, is a legal requirement.[16,45] Compliance is poor; recent studies have found that only 22% of clinical trials required to post study results on CTG met their reporting deadline,[46,47] and only 28%–45% had results that could be found via CTG or PubMed.[19,30,48] Fewer than 10% of completed trials provided both a linked article and basic results.[30] For those trials not required by the FDAAA mandate to post results, only 10% posted results within a year.[47]

An important but often overlooked source of information about trial results is the gray literature of trial-specific press releases or company statements and information found on the public Web sites of pharmaceutical companies.[49-51] Trials presented at scientific meetings may remain unpublished years after presentation, but meeting program abstracts can sometimes be found by searching the Internet. However, results in the gray literature are not permanent, are not indexed on PubMed, may differ greatly from the actual meeting posters, and are not likely to have been fully peer-reviewed.[50,52] Overall, results in the gray literature have smaller treatment effects than results in the published literature, suggesting that some negative trials reported in the gray literature do not progress to full journal publication.[39,51,53]

METHODS

Developed as part of the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) public–private partnership, the Repository of Registered Migraine Trials (RReMiT) provides a snapshot of registered clinical trials for migraine and a scorecard for public availability of results. The WHO International Clinical Trials Registry Platform (ICTRP) provides a single public-access search portal to 15 primary registries, which all follow the international standards for clinical trial registries based on ICMJE requirements.[25,52,54,55]

RReMiT follows methodology initially developed for the Repository of Registered Analgesic Clinical Trials (RReACT), which comprises all analgesic clinical trials for postherpetic neuralgia, fibromyalgia, and diabetic peripheral neuropathy.[52,56] Both RReMiT and RReACT are freely accessible via the ACTTION Web site (http://www.acttion.org/).

We searched all ICTRP primary registries to identify trials of acute migraine treatment and migraine prophylaxis. Trials in RReMiT are randomized, and have a primary (or key secondary) outcome measure assessing drug efficacy. Trials of nutritional supplements and nontraditional medications are included, but device studies are excluded. RReMiT provides registry information on investigational drug names, drug route and mechanism of action, secondary identifiers, study sponsor, study phase, start and completion dates, countries of enrollment, number of subjects, design summary, comparison groups, and primary/key secondary outcomes. Trial status (as of August 2013) is listed as actively recruiting, active but not recruiting, terminated, completed, unknown, or other.

All trials found were examined manually. If the same trial was listed on 2 or more registries, it was considered to be multiply registered and only analyzed once, as in RReACT.[52] Results were sought except for those trials shown as actively recruiting, withdrawn prior to subject enrollment, or not yet open for recruitment. A comprehensive search algorithm was followed. If links or citations of journal publications were provided on the registry record, they were manually checked to confirm correct pairing with the registered trial. If none was available, a manual search of PubMed using the trial name, drug name, keywords from the study title, registry identifiers, and other trial information was conducted. The gray literature was searched using Google, Google Scholar, and sponsor-related Web sites. To ensure accurate registry–results pairings, we relied on all available trial information, including registry data on comparison groups, sample size, principal investigator, and study dates. Available trial-specific efficacy endpoint results are categorized as peer-reviewed journal article, data entered on registry, or gray literature. Results from the highest-level source are summarized, with peer-reviewed articles ranking highest and gray literature lowest.

We first categorized each trial as testing acute migraine therapy or migraine prophylaxis. We then determined whether the treatment consisted of a single therapeutic agent or a combination of agents (defined as treatments consisting of multiple agents administered separately in one or more study arms, as well as single pills/capsules containing multiple active ingredients such as the Treximet combination of sumatriptan and naproxen). We also determined whether the trial design included a true placebo group. Active comparators were not considered true placebos and were included as investigational agents. Placebo arms providing rescue medications to patients as needed were considered true placebo groups.

Each investigational agent was categorized into only 1 of 22 predetermined classes (table 1). Drugs with multiple effects or uses were classified based on their most relevant effect. Agents not falling into any of the predetermined classes were categorized as other. Some categories were broadly defined to accommodate investigational agents with multiple effects, those under study for off-label use in migraine, and those with unknown mechanisms of action. For example, the anticonvulsant category included AMPA/kainate antagonists, not yet approved for sale as anticonvulsants, and tonabersat, a benzopyran derivative with anticonvulsant properties (table 1). The other category included agents ranging from barbiturates to renin-angiotensin system inhibitors and nontriptan drugs acting on 5HT receptor subtypes other than 1B/1D. Cannabinoids were categorized as complementary/alternative therapy because preparations containing the active ingredients vary across states and countries in type, availability, and legal status.

Table 1

Categories of investigational agents in RReMiT migraine trials

Logistic regression analysis was used to identify factors associated with availability of trial results. Initially, univariate models examined the following variables: trial registration on CTG (yes vs no), industry primary sponsor (yes vs no), time since study completion (2 years or more vs less than 2 years from August 2013 based on the registry-listed completion date), type of therapy (acute vs prophylaxis), treatment type (single investigational agent vs combination), presence of a true placebo group (yes vs no), and estimated sample size (dichotomized above vs below the median). Availability of any results (peer-reviewed, registry, or gray literature), and availability of results in the peer-reviewed literature, were analyzed separately.

The variables that showed significant, or nearly significant (p < 0.1), relationships in univariate analyses were included in the final logistic regression models, and adjusted odds ratios (ORs) were generated for 2 outcomes: availability of any results and availability of peer-reviewed results.

RESULTS

Trial registration.

As of August 1, 2013, there were 295 unique clinical trials meeting criteria for inclusion in the RReMiT database on the 15 primary ICTRP registries. A total of 215 trials were registered on CTG. The other 80 trials were listed exclusively on one or more of ICTRP's 14 other registries. Forty-one trials were multiply registered, with 32 trials on both CTG and EU Clinical Trials Register, 2 on 3 different registries, and 7 on other pairs of registries. The median estimated sample size was 141 subjects.

The RReMiT database contains 154 (52%) acute migraine treatment trials and 141 (48%) migraine prophylaxis trials. The figure, A, shows the number of registered trials initiated each year for the past 15 years. The year 2009 had the most trial initiations. More migraine prophylaxis trials were initiated in the last 5 years (2009–2013) than in the previous 5 years (2004–2008), a temporal association not present in acute migraine treatment trials (Fisher exact test; p = 0.014).

Figure

Number of trials initiated over time, with trial status and results availability

(A) The number of new trials initiated each year for both acute migraine trials and migraine prophylaxis trials. (B) The trends over time for the cumulative total number of trials, the cumulative number of trials eligible for a results search (total number of trials minus those actively recruiting, those withdrawn prior to subject enrollment, and those not yet open for recruitment), the cumulative number of trials with results available of any type (peer-reviewed literature, results entered on registry, and gray literature), and the cumulative number of trials with results available in the peer-reviewed literature.

Investigational agents.

A total of 447 investigational agents were used in the 295 RReMiT trials. The most frequently studied categories were triptans (34% of all 295 trials), anticonvulsants (22%), nonsteroidal anti-inflammatory drugs (NSAID) (15%), other (10%), complementary/alternative therapies (8%), β-blockers (7%), and antiemetics (7%). Combination treatments were used in 61 trials (21%). Thirteen trials compared different triptans, and 9 trials compared agents within another single category. The design of 177 trials (60%) included a true placebo group.

The most frequently studied categories of investigational agents in the 154 acute migraine treatment trials were triptans (58%), NSAID (26%), antiemetics (12%), calcitonin gene-related peptide antagonists (8%), other (8%), and acetaminophen (6%) (table 2). Thirty percent of all acute migraine trials (46/154) tested combinations of agents and 57% (88/154) included a true placebo group.

Table 2

Frequency of investigational agents studied for acute migraine trials and migraine prophylaxis trials

The most frequently studied categories in the 141 migraine prophylaxis trials were anticonvulsants (40%), β-blockers (13%), complementary/alternative therapies (13%), other (12%), antidepressants (10%), and botulinum toxin (8%) (table 2). Eleven percent of all migraine prophylaxis trials (15/141) evaluated combinations of agents and 63% (89/141) included a true placebo group.

Trial status and availability of trial results.

Forty-three of the 295 total trials were actively recruiting participants, 7 were withdrawn prior to subject enrollment, and 8 were not yet open for recruitment. Results were sought for the remaining 237 trials (200 trials listed as completed, 13 trials terminated after beginning enrollment, 10 trials listed as active but not recruiting, and 14 trials of unknown status). Table 3 and the figure, B, describe trials by status and trends over time. Except for 2 studies of unknown status, all studies initiated in 2006 or earlier have been completed or terminated. All studies listed as actively recruiting were initiated in 2008 or later.

Table 3

Trial status, availability of results, and results type

Fifty-five percent of trials (131/237) had available results. By manually searching PubMed, 45% (106/237) had results in a peer-reviewed journal. Another 15 had results available via direct posting on the registry (all on CTG), and 10 had results available only in the gray literature. Focusing only on the 200 trials listed as completed, results could be found for 127 (64%), 103 (52%) of which were in peer-reviewed journals, 15 were posted directly onto the registry, and 9 were in the gray literature.

Most, but not all, studies listed as completed provide a completion date. Of 163 trials with a listed completion date of August 2012 or before, 57% had peer-reviewed literature results, and registries/gray literature added another 13%. In contrast, 14 trials in RReMiT listed a completion date between August 2012 and August 2013, and none had available results. For the 147 trials with a listed completion date of August 2011 or before, 59% had peer-reviewed literature results (71% with any available results). Only 20% of the 30 trials with listed completion dates between August 2011 and August 2013 had peer-reviewed publications (30% with any available results). For those trials with article links on the registry trial record, the average number of linked publications was 1.35.

Correlates of results availability.

Factors in the univariate analyses predicting results availability (peer-reviewed literature, results deposited on the registry, and gray literature) included trial registration on CTG (p = 0.004), industry primary sponsor (p < 0.001), and time since study completion of 2 years or more (p < 0.001) (table 4). Study enrollment below the median sample size negatively predicted results availability in the unadjusted model (p < 0.001).

Table 4

Factors related to the availability of peer-reviewed results and results of any type (peer-reviewed literature, results deposited on registry, and gray literature) in unadjusted models

When considering the availability of peer-reviewed publications, the same variables significantly, or nearly significantly, predicted results availability in unadjusted models: trial registration on CTG (p = 0.055), industry primary sponsor (p = 0.011), and time since study completion of 2 years or more (p < 0.001) (table 4). Examining the data by specific time epochs (≥2 years but <4 years, ≥4 years but <6 years, ≥6 years but <8 years, and ≥8 years) showed steadily increasing unadjusted ORs of 4.6, 7.5, 8.6, and 14.6 relative to completion less than 2 years ago. Study enrollment below the median sample size negatively predicted availability of peer-reviewed results (p = 0.007).

Variables not predicting results availability, including in peer-reviewed publications, were the type of therapy (acute vs prophylaxis), treatment type (single investigational agent vs combination), and presence of a true placebo group.

In the multivariate analysis, 2 factors predicted results availability (table 5). Studies completed 2 or more years ago were significantly more likely to have available results (adjusted OR 4.5, 95% confidence interval [CI] 2.0–10.5; for peer-reviewed publications, adjusted OR 6.9, 95% CI 2.6–18.2). Trials with a sample size below the median of 141 subjects were significantly less likely to have available results (adjusted OR 0.4, 95% CI 0.2–0.8).

Table 5

Adjusted likelihood for availability of peer-reviewed results and results of any type (peer-reviewed literature, results deposited on registry, and gray literature)a

DISCUSSION

RReMiT presents a complete and global picture of all registered acute migraine treatment trials and migraine prophylaxis trials contained in the 15 ICTRP registries, including the largest, ClinicalTrials.gov. As with RReACT,[52] successfully building RReMiT required extensive manual searching and verification at several points. First, the registries had to be searched with a variety of techniques to find all relevant trials. Second, each registry record had to be examined individually to ensure that only unique trials were analyzed; registries are not consistently cross-linked and some trials were listed on up to 3 different registries. Third, a complex search strategy was employed to find results outside the peer-reviewed literature. Although the gray literature contributed a relatively small number of results in RReMiT compared to RReACT (4.2% vs 8.7% of trials had results only in the gray literature), and gray literature results are not necessarily permanently archived or peer-reviewed, this is an important resource that cannot be overlooked.

Eleven ICTRP registries allow linking of the registry record to a publication. However, links are often not provided, even on CTG. A study by Huser and Cimino[57] found that 44% of trials without linked results have published articles retrievable by manually searching PubMed. A different group reported that only 14% of a sample of trials on CTG had a published article linked to the registry record, and manually searching PubMed increased the proportion to 52%.[31] Only 20% of trials in the RReMiT database had one or more published articles linked to the trial record, but this proportion increased to 45% when PubMed was manually searched. For those trials with article links, the average number of publications was 1.35, which is comparable to the value of 1.46 linked articles per trial calculated by Huser and Cimino[30] when analyzing a sample of trials registered on CTG. Accuracy and relevance of links cannot be assumed. Articles may not provide any results from the trial they are linked to, may reveal major protocol and subject enrollment differences from the registry description, may provide only preliminary/interim results, or may present only a pooled analysis of multiple trials.

The RReMiT database has limitations. First, unregistered trials are not included, but as adoption of trial registration has become progressively more widespread, the number of newly published results from unregistered trials should steadily decline.[26] Confirming that a published study is registered usually requires a manual search because 40%–45% of journal publications fail to include trial registration numbers.[58] Many journals do not ask for specific trial registration numbers, and not all editors and publishers are convinced they should follow ICMJE guidelines.[27,59] Finding relevant publications would be facilitated if journal publications always included trial registration numbers.

Furthermore, RReMiT does not calculate the exact time from study completion to availability of results for each trial, and therefore cannot assess compliance with the 2007 FDAAA mandate about reporting results within specified time periods. In other studies, as few as 12% of all trials had reported results within 1 year of study completion.[20] Ross et al.[32] calculated 23 months as the median time to publication. In a random sample of 150 trials registered on CTG with results posted on the registry, Zarin et al.[31] found the proportion with an associated peer-reviewed journal article had risen from 25% to 52% 1 year later. In RReMiT, 59% of trials completed at least 2 years ago had peer-reviewed publications available, compared to 20% of trials completed less than 2 years ago. There was no clear temporal break point, as the unadjusted OR steadily increased in each subsequent 2-year epoch since completion.

Does the responsibility for failing to publish fall on the authors or the system? Currently, for studies without available results, there are no methods short of individually asking every investigator if a results paper was ever submitted. It is unfortunate but well-known that negative trials are less likely to be published.[60] Competing articles whose findings are deemed more relevant for the readership are chosen instead to fill the limited pages of a journal. A funder may want to delay or prevent publication, or a manuscript may cycle through multiple journals over the course of a year before acceptance. For small studies with negative results, concerns over methods and power loom especially large since there is a difference between failing to reject the null hypothesis that there is no difference between groups and accepting the null hypothesis that there is no difference between groups. Although RReMiT does not interpret studies as positive or negative, it does show that studies with small sample sizes are less likely to have available results.

Perhaps contrary to expectations, industry primary sponsorship was positively associated with availability of results in univariate analyses. Other recent analyses of results and outcome reporting have also found that industry-funded trials were significantly more likely to have results deposited in the CTG registry.[19,48] Law et al.[20] and Kuehn[46] reported that industry-sponsored clinical trials were more than 3 times more likely to post results on CTG than NIH-funded trials. A possible explanation is that industry-sponsored trials are in fact subject to higher public scrutiny and more institutional regulations, and also may have more resources available to comply with reporting mandates.

By searching all the major clinical trial registries and analyzing nearly 300 migraine studies, the RReMiT database is an important contribution to the literature on trial registration and transparency in reporting of results. Compared with other disorders, are migraine trials more or less likely to have available results? In a recent update of the RReACT database for fibromyalgia, postherpetic neuralgia, and painful diabetic neuropathy trials, results of any kind could be found for 46% of the 391 trials eligible for a results search, and peer-reviewed publications could be found for 30% of trials, irrespective of time since completion.[52] Migraine trials appear to be significantly better reported (55% with any results; 45% with peer-reviewed results; χ2 test p < 0.001 for peer-reviewed results).

The peer-reviewed literature is an imperfect and incomplete resource for constructing evidence-based therapy guidelines. Biased recommendations could result if the published literature does not accurately reflect the full range of potentially available results. Even for large, well-designed drug registration trials, studies with negative results are much less likely to be published.[42] In the case where a guidelines group was intending to recommend a drug as beneficial, many unavailable results from completed trials should reduce confidence in the recommendation on the grounds that the missing publications are more likely negative than positive. In the context of a global focus on clinical trials, complete transparency remains elusive. Forty percent of migraine trials remain unpublished more than 2 years following study completion. RReMiT demonstrates that recently completed studies, small studies, and studies without industry as the primary sponsor are all less likely to have results available in the peer-reviewed literature.