Primary outcome switching among drug trials with and without principal investigator financial ties to industry: a cross-sectional study.

OBJECTIVES: To determine the relationship between manufacturer-related financial ties among investigators of published drug trials and rates of discrepant registered and published primary trial outcomes.
DESIGN: Cross-sectional study.
SETTING: Human subjects drug trials published in 'core clinical' MEDLINE journals in 2013. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary study endpoint was the presence of a prospectively registered, clearly defined primary outcome that matched the published primary outcome for each included trial. Secondary outcomes included assessments of registration timing and quality, and the impact of outcome discrepancies between registration and publication on the statistical significance of the included trials.
RESULTS: Of 192 included trials, 134 (70%) were positive and 58 (30%) were negative. Financial ties were present between first or last authors and drug manufacturers for 130 trials (68%), of which 78% were positive, versus 53% of trials with no financial ties that were positive. Clearly defined, prospectively registered outcomes that matched the published outcomes were present in just 76 of the 192 trials (40%). After adjusting for study start date and sample size, the observed relationship between investigator financial ties and the presence of a match between prospectively registered and published primary outcomes was of borderline statistical significance (OR 2.12, 95% CI 0.998 to 4.50). Studies with financial ties present were more likely than studies without ties to have been prospectively registered (78%vs48%, P<0.001) and were more likely to have prospectively registered a clearly defined primary outcome(62%vs35%, P<0.001).
CONCLUSIONS: Less than half of the trials in this cohort were prospectively registered with a clear primary outcome that was consistent with the primary outcome reported in the published manuscript. The presence of investigator financial ties was associated with higher quality registration practices, though this association diminished after adjusting for factors that impact registration quality. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

The relationship between investigator–manufacturer financial ties and registration quality has not been previously assessed.

Multiple reviewers independently assessed study endpoints.

Trial registration patterns have evolved over time, and these results may not reflect current registration and publication practices.

Introduction

Randomised controlled trials (RCTs) are a critical means of assessing the efficacy of medical interventions. A core principal in the ethical conduct of RCTs is the presence of clinical equipoise, which mandates the existence of genuine uncertainty within the medical community with respect to the best choice between the various treatment options that a trial is investigating.1 A recent cross-sectional study investigating the relationship between the financial ties of principal investigators and RCT outcomes found that trials with financial ties between principal investigators and drug manufacturers were substantially more likely to report positive results than trials without financial ties.2

There are several plausible explanations for the higher proportion of positive results among trials in which the investigators have financial ties to drug manufacturers. First, these investigators may make design decisions such as selection of a comparator or enrolment of a specific population that favour the intervention.3–5 Second, investigators with financial ties may be less likely to publish the results of unfavourable trials, leading to publication bias.6 7 Third, investigators with financial ties may be more likely than others to change primary study outcomes after trial completion in order to highlight those outcomes that are statistically significant or otherwise favourable.8 In evaluating this third possibility, clinical trial registries may allow for the assessment of outcome switching by facilitating comparisons between prospectively registered primary outcomes and published primary outcomes.9 10

The goal of this investigation was to determine the relationship between the presence of manufacturer-related financial ties among principal investigators of published drug trials and rates of discrepant registered and published primary trial outcomes.

Methods

Sample identification

We analysed a group of clinical trials that had been identified for a recent cross-sectional study assessing the association between the financial ties of principal study investigators and study outcomes. Trials were randomly selected for inclusion in the original study from among English-language, human subjects drug trials that had been published between 1 January 2013 and 31 December 2013 in ‘core clinical’ MEDLINE journals.2 The original study included manuscripts describing 195 trials. We excluded three of these manuscripts because they reported the results of secondary or follow-up analyses, leaving 192 included trials.

Investigator financial ties

We defined principal investigators as the first and last author, along with any other authors specified as sharing first or last author responsibilities for each included manuscript. A comprehensive search was performed to identify financial ties between these investigators and manufacturers of the study drug. A financial tie was defined as direct compensation to the investigator in the form of advisor/consultancy payments, honorariums, speaker’s fees, stock ownership, travel/meal reimbursement, a patent for which the investigator was the inventor or an employer–employee relationship. The search for financial ties has been previously described in detail and included a search of the trial publication, MEDLINE (to allow review of other publications by the investigator), Google, ProPublica’s Dollars for Doctors database and the US Patent Office.2 Financial ties were only included if they occurred within 2 years before article publication. All financial ties were independently verified by at least two reviewers, and discrepancies were resolved by consensus. If any of the principal investigators of a manuscript was found to have a financial tie to a manufacturer of the study drug, the manuscript was considered to have a financial tie.

Registry search

For each trial, one of two reviewers initially inspected the published manuscript for evidence of trial registration. If no registration information was provided, the reviewers searched ClinicalTrials.gov by keyword to identify registry entries corresponding to the included trials. When this initial search failed to identify a trial registration entry matching the published trial report, a third reviewer (CJ) with expertise in trial registration searched ClinicalTrials.gov, the International Standard Registered Clinical/soCial sTudy Number database, the WHO International Clinical Trials Registry Platform search portal and any national registries corresponding to the principal investigators’ countries of origin (eg, Australian New Zealand Clinical Trials Registry and European Union Clinical Trials Register). The final registry search occurred in February 2017. Searches were performed by keyword, title, principal investigator, and funding source. In addition to these characteristics, potential matches between published trials and registry entries were assessed based on study interventions, planned sample sizes, enrolment dates and trial locations. We considered trials unregistered if neither independent search identified a corresponding registry entry.

Assessment of outcomes from the clinical trials

Standardised data collection forms were used to record information from published manuscripts and registry entries corresponding to each included trial. We recorded the primary and secondary outcome(s) reported within each published manuscript. If no primary outcome was explicitly defined within the manuscript or abstract, we considered the outcome used for the power calculation to be the primary published outcome. If no primary outcome was defined and there was no power calculation, we considered the published primary outcome to be undefined.

Trial outcomes were classified as positive if the study hypothesis was supported for the primary efficacy outcome and negative if it was not. Superiority trials were considered positive if the drug in the intervention arm was statistically superior to the control (P<0.05) and non-inferiority trials were considered positive if the intervention arm was not significantly worse than the control. Trials with multiple published primary outcomes were positive if at least one efficacy outcome was either positive (superiority studies) or not significantly different from the control (non-inferiority studies).

For each registered trial, we recorded the date of initial trial registration, the date on which a primary outcome was first registered, the registered primary outcome(s) and registered secondary outcomes. If changes to the registered primary outcome were made, we recorded the primary outcome that was listed at the time trial enrolment began. Two reviewers (CJ and BSM) then independently determined whether each registered outcome was clearly defined. In order to be considered a clearly defined primary outcome, the registered outcome needed to provide sufficient detail to allow a reader to reasonably design a study measuring the same outcome measure. In most cases, in order to meet this standard, the registered outcome needed to describe a specifically defined variable of interest and a specific time-point for assessment. Discrepancies were resolved by consensus.

We considered trials that had a clearly defined primary outcome registered within 90 days of the start of enrolment to be prospectively registered; trials registered more than 90 days after enrolment began were classified as being retrospectively registered.

Primary outcome

For those trials with prospectively registered, clearly defined primary outcomes, we compared the registered and published primary outcomes to assess for primary outcome switching. Two reviewers (CJ and BSM) who were blinded to the presence or absence of principal investigator financial ties independently assessed outcome consistency. Discrepancies were resolved by consensus following a review of the full manuscript and the full registry entry. Registered and published outcomes were considered consistent with one another if every registered primary outcome for a specific study was reported as a primary outcome in the published manuscript and if every primary outcome described in the manuscript was registered clearly and prospectively. We then categorised primary outcome discrepancies according to the classification used by Mathieu et al.11

The primary outcome for our study was the presence of a prospectively registered, clearly defined primary outcome that was consistent with the published outcome. We report the proportion of all included trials meeting this primary outcome measure, and also compare the primary outcome results among trials having investigator financial ties to those trials without investigator financial ties.

Secondary outcomes

Secondary outcomes for the present study were study registration, prospective study registration and prospective registration with a clearly defined primary outcome. When possible, we also assessed the impact of any outcome discrepancies on each trial’s statistical significance. An outcome discrepancy was defined as favouring statistical significance if the discrepancy resulted in publication of a statistically significant published outcome or if it resulted in demotion of a non-significant registered primary outcome. We considered a trial’s registration to be high quality if it was prospectively registered with matching registered and published primary outcomes.

Statistical methods

We compared registration outcomes between trials having principal investigators with financial ties and those without financial ties using χ2 testing and Fisher’s exact test. Registration outcomes were also compared between trials with and without industry funding using χ2 testing. Logistic regression was used to assess the relationship between the presence of financial ties and a clearly defined registered primary outcome that matched the published primary outcome while controlling for the study start date and sample size, as these factors are both associated with registration rate and quality.9 12 13 Analyses were performed using PASW Statistics V.18.0 (IBM Corp, Armonk, New York, USA). Missing data were excluded in pair-wise fashion.

Results

A total of 192 trials met inclusion criteria and were included in the analysis. Most were phase III trials (53%), and 70% were funded by industry (table 1). The study cohort included trials across a broad range of sample sizes, including 21% with fewer than 100 participants and 27% with at least 500 participants. Principal investigators had financial ties for 130 of the 192 included trials (68%). These ties were most common among trials that were large (88% of trials with ≥500 participants) and among trials sponsored by industry (84%). In total, 134 of the 192 included trials (70%) were positive and 58 (30%) were negative. Trials with financial ties were more likely to be positive than trials with no financial ties (78% vs 53%, P=0.001).

Table 1

Baseline characteristics of included trials

Trial characteristic	N (%)
	All trials n=192	Financial ties present* n=130	Financial ties absent n=62
Study phase
Phase II	50 (26)	38 (29)	12 (19)
Phase III	102 (53)	81 (62)	21 (34)
Phase IV	16 (8)	7 (5)	9 (15)
Other	24 (13)	4 (3)	20 (32)
Blinding
Double blind	146 (76)	101 (78)	45 (73)
Single blind	7 (4)	5 (4)	2 (3)
Open label	37 (19)	22 (17)	15 (24)
Unknown	2 (1)	2 (2)	0 (0)
Sample size
<100 participants	40 (21)	17 (13)	23 (37)
100–299 participants	55 (29)	36 (28)	19 (31)
300–499 participants	45 (23)	31 (24)	14 (23)
≥500 participants	52 (27)	46 (35)	6 (10)
Funding source
Any industry funding	134 (70)	113 (87)	21 (34)
Any government funding	56 (29)	23 (18)	33 (53)
Non-profit	32 (17)	14 (11)	18 (29)
No funding reported	6 (3)	1 (1)	5 (8)
First author continent
North America	87 (45)	69 (53)	18 (29)
Europe	68 (35)	46 (35)	22 (35)
Asia	28 (15)	8 (6)	20 (32)
Beginning of enrolment†
Prior to 2006	37 (19)	17 (13)	20 (32)
2006–2007	45 (23)	31 (24)	14 (23)
2008–2009	63 (33)	49 (38)	14 (23)
2010 or later	46 (24)	33 (25)	13 (21)

*Direct financial relationship present between first or last study author and drug manufacturer.

†Start date not reported for one trial.

Almost all trials were registered (97%), and the majority of trials were registered with ClinicalTrials.gov (84%). Sixty-eight per cent (131/192) of the included trials were prospectively registered, with a median time from study start to registration of 15 days (IQR for registration timing was from 26 days before enrolment to 116 days after enrolment; range 906 days before enrolment to 5225 days after enrolment). Fifty-five trials (29%) were retrospectively registered, and six trials (3%) were unregistered (table 2). Among the prospectively registered trials, 103 (79%) had clearly defined primary outcomes, and 28 (21%) had unclear outcomes.

Table 2

Registration characteristics of included trials

Study characteristic	N (%)
	All trials n=192	Financial ties present n=130	Financial ties absent n=62
Registered at any time	186 (97)	130 (100)	56 (90)*
Registered prospectively	131 (68)	101 (78)	30 (48)*
Prospectively registered with clearly defined primary outcome	103 (54)	81 (62)	22 (35)*
Prospectively registered and published primary outcomes are consistent	76 (40)	62 (48)	14 (23)*
Prospectively registered primary outcome inconsistent with published primary outcome	27 (14)	19 (15)	8 (13)
Registered primary outcome reported as secondary outcome in published manuscript	11 (6)	7 (5)	4 (6)
Registered primary outcome not reported in published manuscript	5 (3)	5 (4)	0 (0)
Published manuscript includes new primary outcome	12 (6)	10 (8)	2 (3)
Published primary outcome described as secondary in registry	4 (2)	3 (2)	1 (2)
Timing of assessment of primary outcome variable differs between registry and manuscript	7 (4)	6 (5)	1 (2)
Does registration discrepancy favour statistically significant published results?	n=27	n=19	n=8
Yes	14 (52)	12 (63)	2 (25)
No	8 (30)	3 (16)	5 (63)
Unknown	5 (19)	4 (21)	1 (13)

*P≤0.001.

In analysing our primary outcome, clearly defined, prospectively registered outcomes that matched the published outcomes were present in just 76 of the 192 included trials (40%). In the unadjusted primary outcome analysis, trials with investigator financial ties were more likely than those without financial ties to be prospectively registered with clear primary outcomes that matched the published primary outcomes (48% vs 23%, P<0.001). More recent trials were more likely than earlier trials to follow high-quality registration practices (ie, prospectively registered with a matching published outcome) and the proportion of trials with financial ties was also greater among more recently initiated trials. When controlling for study start date and sample size, the presence of investigator financial ties showed a borderline statistically significant association with registration quality (OR 2.12, 95% CI 0.998 to 4.51) (table 3). Within this model, study start prior to 2006 (OR 0.05, 95% CI 0.01 to 0.24 relative to start after 2009) and study start in 2006/2007 (OR 0.42, 95% CI 0.18-.996) were both associated with lower rates of high-quality trial registration. As compared with trials with no financial ties, those with the presence of financial ties were also more likely to be prospectively registered (P<0.01) and more likely to have clearly registered primary outcomes (P<0.01).

Table 3

Odds of a prospectively registered outcome consistent with the published outcome on unadjusted analysis and after adjusting for study characteristics

Study characteristic	N	Unadjusted OR (95% CI)	Adjusted OR (95% CI)
Date enrolment began
Prior to 2006	37	0.04 (0.01 to 0.21)	0.05 (0.01 to 0.24)
2006–2007	45	0.42 (0.18 to 0.99)	0.41 (0.18 to 1.00)
2008–2009	63	0.79 (0.37 to 1.71)	0.73 (0.33 to 1.60)
After 2009	46	–	–
Study size (per 100 participants)		1.03 (1.00 to 1.06)	1.0 (0.99 to 1.05)
Investigator financial ties present
Yes	130	3.13 (1.57 to 6.22)	2.12 (1.00 to 4.51)
No	62	–	–

Among the 28 trials with an unclear registered primary outcome, positive results in the published manuscript were reported for 21 (75%), and 73 of the 103 (71%) trials with clearly registered primary outcomes reported positive results. Among the 27 trials with unmatched registered and published primary outcomes, 22 (81%) reported positive results, as compared with 51/76 (67%) of those with matching outcomes.

Trials with industry funding were more likely than those without to be registered with a clear primary outcome that was consistent with the published outcome (46% vs 24%, P<0.01), and trials with either investigator financial ties or industry funding were also more likely than those without any industry ties to be registered with a clear, consistent primary outcome (44% vs 22%, P<0.01).

Discrepancies between registered and published primary outcomes were present in 27 trials. The most commonly observed discrepancies were the addition of a new primary outcome in the published manuscript (n=12) and reporting a registered primary outcome as a secondary outcome in the manuscript (n=11). Among trials with financial ties present, 12 of 19 (63%) discrepancies favoured the publication of a statistically significant result, 3 (16%) did not and 4 (21%) could not be classified. Among trials without financial ties, 2 of 8 (25%) discrepancies favoured statistical significance, 5 (63%) did not and 1 (13%) could not be classified.

Rates of retrospective registration, unclear outcome registration and inconsistencies between registered and published primary outcomes were all similar between positive and negative trials (table 4). The impact of primary outcome inconsistencies on the statistical significance of the published outcomes is described in table 5.

Table 4

Presence of investigator financial ties and evaluation of registration quality among positive and negative trials

Study characteristic	N (%)		P value
	Positive trials n=134	Negative trials n=58
Financial ties present	101 (75)	29 (50)*	0.001
Financial ties absent	33 (25)	29 (50)*
Registered at any time	130 (97)	56 (97)	1.0
Registered prospectively	94 (70)	37 (64)	0.39
Prospectively registered with clearly defined primary outcome	73 (54)	30 (52)	0.67
Prospectively registered and published primary outcomes are consistent	51 (38)	25 (43)	0.51

*P=0.001.

Table 5

Statistical significance of primary outcome changes among prospectively registered trials with inconsistent registered and published outcomes

Study characteristic	N (%)		P value
	Positive trials n=22	Negative trials n=5
Impact of outcome discrepancy on statistical significance
Newly significant published primary outcome	12 (55)	2 (40)	0.82
No newly significant published primary outcome	6 (27)	2 (40)
Unable to determine significance of outcome change	4 (18)	1 (20)

*P=0.001.

Discussion

We assessed the consistency between prospectively registered and published primary outcomes among 192 trials that were characterised according to the presence or absence of financial ties among the principal study investigators. We found that approximately half of the trials with financial ties were registered appropriately, with a clear, prospectively defined primary outcome that matched the published outcome, and fewer than one quarter of trials without financial ties met this standard. However, after controlling for other relevant trial characteristics, these differences reached only borderline significance. Retrospective registration, registration with unclear primary outcomes and inconsistencies between registered and published outcomes were common within this cohort of clinical trials regardless of the presence of industry-related financial ties among principal investigators.

This study represents an attempt to understand why trials in which principal investigators have financial interests at issue are more likely to produce positive outcomes as compared with trials without direct investigator financial interests at stake. This pattern is troublesome as it suggests that clinical equipoise, and therefore the ethical justification to randomise participants to different treatment groups, may be violated.14

Several primary mechanisms exist that might result in higher rates of positive trial results among those with investigator financial ties. First, it is possible that investigators with financial ties make study design decisions that increase the chances of producing a trial with positive results. These decisions potentially include choosing comparators or selecting dosages that favour the intervention, selecting outcomes that favour the intervention or choosing data analysis techniques that are more likely to produce favourable results.3–5 15 16 Our study design was unable to address this possibility. However, given the existing evidence showing that study design decisions sometimes introduce bias into industry-sponsored trials, it is likely that similar problems may affect trials with investigator financial ties. Second, it is possible that investigators with financial interests are more likely than others to change primary published outcomes in order to favour statistically significant results. We observed some evidence of outcome switching favouring positive results among trials in which financial ties are present, but this does not appear to be more common than among trials in which investigators do not have financial ties.

Another possibility is that trials with financial ties may be at higher risk of non-publication due to publication bias.6 7 In other words, investigators with financial ties may be less likely to publish trials with results that they perceive as being unfavourable. Previous studies have shown that registered trials sponsored by industry are less likely to be published than those without industry sponsorship.17–19 However, the relationship between investigator financial ties and publication has not been thoroughly assessed. While our data do not directly address the relationship between investigator financial ties and non-publication, we did not observe increased rates of retrospective registration or failure to register among trials with investigator financial ties, which could create the opportunity for bias. Importantly, the absence of this pattern does not rule out publication bias as an explanation for the high observed rate of positive published trials. Additionally, almost 90% of the trials in this sample with investigator financial ties received industry funding, which is associated with high rates of favourable reported study outcomes.16 Industry-funded trials are often performed for regulatory purposes; as such, they often differ from non-funded trials with respect to the population studied, study design and trial implementation. Therefore, the presence of an investigator financial tie may serve as a marker for these other fundamental trial differences.

This study has several limitations that are important to consider when interpreting these results. The majority of the included trials were started between 2006 and 2010. Trial registration patterns have evolved over time, and observations based on this cohort may not reflect current practice.12 The process of assessing registered and published outcomes necessarily involves making somewhat subjective judgements regarding the consistency of these outcomes. We addressed this issue by using multiple independent raters, and in most cases the observed outcome inconsistencies were not subtle. Importantly, our results are consistent with findings from other similar studies assessing registration quality.10 11 20–24 A further study limitation is that we considered the presence or absence of financial ties among principal investigators, which we defined as the first and last study authors. It is possible that for some trials other members of the research team did have financial ties and exerted significant influence over the conduct and reporting of these trials. However, in most cases, the first and last study authors are primarily responsible for study conduct and reporting, and inclusion of other authors in this analysis would have potentially diluted any observed relationship between financial ties and study outcomes or registration practices. Finally, we defined retrospective registration as registration that occurred more than 90 days after the start of trial enrolment. While the International Committee of Medical Journal Editors (ICMJE) requires that registration occur prior to beginning enrolment, previous studies have used this 90-day definition.25 In this case, we considered it unlikely that registration within 90 days after the start of enrolment would have been performed in response to an analysis of the trial data and that registration within this timeframe is likely more reflective of poor familiarity with registration requirements rather than an attempt to impact the trial outcome by changing outcome definitions.

Conclusions

We observe that regardless of the presence of financial ties, less than half of the trials in this cohort were prospectively registered with a clear primary outcome, which was consistent with the primary outcome reported in the published manuscript. Registration practices within this cohort were consistently poor, and after adjusting for relevant factors, registration quality did not differ substantially between trials with and without investigator financial ties. These results do not support our hypothesis that post hoc outcome switching explains the high rate of positive results among trials with investigators that have financial ties to industry. If the findings of RCTS are to appropriately inform patient care, the primary findings reported must reflect the study’s prespecified outcome measures. Trial investigators, sponsors, peer reviewers and editors all have a responsibility to ensure that published manuscripts consistently report the prespecified primary outcomes.