What Constitutes an Independent Statistical Analysis?

BACKGROUND

Potential bias in clinical trials related to relationships with the pharmaceutical industry is a longstanding concern.1 Between 2005 and 2013, JAMA journals required industry-sponsored studies to conduct independent statistical analysis (ISA), defined as data analysis by an “independent statistician at an academic institution” using the raw data set.2 While no journals currently require ISA, the term may be used to denote impartiality and robustness in data analysis.2, 3 However, its meaning, frequency of use, and association with study characteristics are not clear. Our study’s purpose was to investigate the prevalence and characteristics of ISA in published RCTs focused on drug efficacy and their adherence to JAMA’s definition.

METHODS

We searched MEDLINE and randomly selected 646 drug efficacy RCTs from 2013, as described previously;4 190 met inclusion criteria. Two of four reviewers (AA, RA, AW, SS) independently abstracted data regarding trial characteristics, clinical area, results, funding source, investigator/manufacturer financial ties, and description of ISA or independent statistician. Among papers reporting ISA, we abstracted in duplicate information concerning the analysis the sponsor’s relationship to data and analyses, and statistician(s) identity. Disagreements were resolved by consensus. When ISA was described, we determined conformity with its definitional components (academic statistician affiliation and use of the full dataset) and the relationship between ISA and study characteristics and outcome. We used the Mann–Whitney test for continuous variables and Chi-squared for categorical variables (SAS, V9).

RESULTS

Statistical Analysis Characteristics

Among the 190 trials, 17 (8.9%) reported ISA; the majority (15, 88%) were industry-funded and published in high impact journals (IF > 10) (12, 71%) (Table 1). Most identified the independent statistician(s) by name (11, 65%). Roles of independent statisticians varied; they led the analysis in eight trials (47%), validated the sponsors analysis in four (24%), provided statistical assistance in three (18%), and had an unspecified role in two (12%). ISA adhered to both components of the definition in seven trials (41%); independent analysts had academic affiliation in 13 trials (76%) and full dataset access in 11 (65%).

Table 1

Prevalence of Independent Statistical Analysis by Trial Characteristics (N = 195)

	N	Independent statistical analysis present, N (%)	Independent statistical analysis absent, N (%)	p value
Outcome
Positive	136	10 (7.4)	126 (92.6)	0.31
Negative	59	7 (11.9)	52 (88.1)
Funding source
Any industry funding	134	15 (11.2)	119 (88.8)	0.069
No industry funding	61	2 (3.3)	59 (96.7)
Financial ties
Financial ties present	132	13 (9.8)	119 (90.2)	0.42
Financial ties absent	63	4 (6.3)	59 (93.7)
Impact factor
≥ 10	100	13 (13.0)	87 (87.0)	0.03*
< 10	95	4 (4.2)	91 (95.8)
RCT phase
Phase 3	102	10 (9.8)	92 (90.2)	0.57
Other	93	7 (7.5)	86 (92.5)
RCT type
Double-blinded	147	12 (8.2)	135 (91.8)	0.63
Other	48	5 (10.4)	43 (89.6)
Sample size
Q1 (13–118)	49	1 (2.0)	48 (98.0)	< 0.001
Q2 (119–315)	49	2 (4.1)	47 (95.9)
Q3 (316–615)	49	3 (6.1)	46 (93.9)
Q4 (616–21,105)	48	11 (22.9)	37 (77.1)
Clinical area
Cardiology	31	9 (29.0)	22 (71.0)	< 0.001
Oncology	22	2 (9.1)	20 (90.9)
Other specialties	142	6 (4.2)	136 (95.8)
Trial registration
Yes	184	17 (9.2)	167 (93.8)	0.29
No	11	0 (0)	11 (100.0)
Type of analysis
Superiority	174	14 (8.0)	160 (92.0)	0.34
Non-inferiority	21	3 (14.3)	18 (85.7)
Comparator
Placebo	146	13 (8.9)	133 (91.1)	0.87
Active	49	4 (8.2)	45 (91.8)
Outcome measure
Surrogate	65	4 (6.2)	61 (93.8)	0.37
Clinical	130	13 (10.0)	117 (90.0)
First author
US	74	4 (5.4)	70 (94.6)	0.20
Other	121	13 (10.7)	108 (89.3)

*p value based on continuous pooled variance test: impact factor, 0.0029; sample size, < 0.0001

Papers which described an independent statistical analysis

1. Oettle H, Neuhaus P, Hochhaus A et al. Adjuvant Chemotherapy With Gemcitabine and Long-term Outcomes Among Patients With Resected Pancreatic CancerThe CONKO-001 Randomized Trial. JAMA.2013;310(14):1473–1481. doi:10.1001/jama.2013.279201

2. Nicholls SJ, Bakris GL, Kastelein JJP et al. Effect of Aliskiren on Progression of Coronary Disease in Patients With PrehypertensionThe AQUARIUS Randomized Clinical Trial. JAMA. 2013;310(11):1135–1144. doi:10.1001/jama.2013.277169

3. Opal SM, Laterre P, Francois B et al. Effect of Eritoran, an Antagonist of MD2-TLR4, on Mortality in Patients With Severe SepsisThe ACCESS Randomized Trial. JAMA. 2013;309(11):1154–1162. doi:10.1001/jama.2013.2194

4. Steg PG, Mehta SR, Pollack CV et al. Anticoagulation With Otamixaban and Ischemic Events in Non–ST-Segment Elevation Acute Coronary SyndromesThe TAO Randomized Clinical Trial. JAMA. 2013;310(11):1145–1155. doi:10.1001/jama.2013.277165

5. Steg PG, van ‘t Hof A, Hamm CW, et al. Bivalirudin started during emergency transport for primary PCI. N Engl J Med 2013;369:2207–2217

6. de Zeeuw D, Akizawa T, Audhya P, et al. Bardoxolone methyl in type 2 diabetes and stage 4 chronic kidney disease. N Engl J Med 2013;369:2492–2503

7. Armstrong PW, Gershlick AH, Goldstein P, et al. Fibrinolysis or primary PCI in ST-segment elevation myocardial infarction. N Engl J Med 2013;368:1379–1387

8. Kosiborod M, Arnold SV, Spertus JA, et al. Evaluation of Ranolazine in Patients With Type 2 Diabetes Mellitus and Chronic Stable Angina. Journal of the American College of Cardiology. 2013;61(20):2038–2045. doi:10.1016/j.jacc.2013.02.011

9. Burton LA, Sumukadas D, Witham MD, Struthers AD, Mcmurdo ME. Effect of Spironolactone on Physical Performance in Older People with Self-reported Physical Disability. The American Journal of Medicine. 2013;126(7):590–597. doi:10.1016/j.amjmed.2012.11.032

10. Gisslinger H, Gotic M, Holowiecki J, et al. Anagrelide compared with hydroxyurea in WHO-classified essential thrombocythemia: the ANAHYDRET Study, a randomized controlled trial. Blood. 2013;121(10):1720–1728. doi:10.1182/blood-2012-07-443,770

11. Parfrey PS, Chertow GM, Block GA, et al. The Clinical Course of Treated Hyperparathyroidism Among Patients Receiving Hemodialysis and the Effect of Cinacalcet: The EVOLVE Trial. The Journal of Clinical Endocrinology & Metabolism. 2013;98(12):4834–4844. doi:10.1210/jc.2013-2975

12. Banai S, Finkelstein A, Almagor Y, et al. Targeted anti-inflammatory systemic therapy for restenosis: The Biorest Liposomal Alendronate with Stenting sTudy (BLAST)—a double blind, randomized clinical trial. American Heart Journal. 2012;165(2). doi:10.1016/j.ahj.2012.10.023

13. Youn YJ, Lee J-W, Ahn SG, et al. Multicenter randomized trial of 3-month cilostazol use in addition to dual antiplatelet therapy after biolimus-eluting stent implantation for long or multivessel coronary artery disease. American Heart Journal. 2013;167(2). doi:10.1016/j.ahj.2013.08.028

14. Zhu F-C, Liang Z-L, Li X-L, et al. Immunogenicity and safety of an enterovirus 71 vaccine in healthy Chinese children and infants: a randomised, double-blind, placebo-controlled phase 2 clinical trial. The Lancet. 2013;381(9871):1037–1045. doi:10.1016/s0140-6736(12)61764-4

15. Giugliano RP, Ruff CT, Braunwald E et al. Edoxaban versus warfarin in patients with atrial fibrillation. The New England Journal of Medicine. 2013; 369(22):2093–104. doi: 10.1056/NEJMoa1310907

16. Ryan CJ, Smith MR, de Bono JS et al. Abiraterone in Metastatic Prostate Cancer without Previous Chemotherapy. New England Journal of Medicine. 2013;368(6):584–584. doi:10.1056/nejmx130004

19. Pulido T, Adzerikho I, Channick RN et al. Macitentan and morbidity and mortality in pulmonary arterial hypertension. New England Journal of Medicine. 2013; 369(9):809–18. doi: 10.1056/NEJMoa1213917

Relationship to Study Characteristics

ISA was not associated with industry funding (p value = 0.07), positive study outcome (p value = 0.31), or financial ties to the manufacturer (p value = 0.42). ISA was strongly associated with sample size (p value < 0.0001) and clinical area (p value < 0.001), notably cardiology. ISA was not associated with trial registration, analysis type, phase, comparator, outcome measure, or first author country (Table 1).

DISCUSSION

We found that drug efficacy RCTs rarely self-reported ISA, though the term was used more commonly in large, industry-funded studies published in high impact journals. The meaning of ISA varied among trials with some statisticians controlling the analysis and others serving as collaborators or consultants.

In the past, JAMA clearly defined ISA and required it to ensure integrity and minimize bias,3 but this requirement resulted in fewer manuscript submissions by industry and was dropped.5, 6 Regardless, the term remains in use. Our findings demonstrate ambiguity around its meaning, possibly resulting in an unwarranted implication of rigor and integrity. Given this ambiguity, readers of the literature should not assume that ISA represents methodological rigor. Instead, readers concerned about the integrity of data analysis should note details of the identity, role, and affiliation of authors or statisticians performing analyses and their involvement in the protocol and access to data. Similarly, editors with concerns about the integrity of data analysis should ask authors for transparency regarding these issues.

Our study was limited by the low prevalence of ISA and may not have been powered to detect significant differences in study characteristics. In particular, there was a trend toward an association between ISA and industry funding that we may have been underpowered to detect.

In conclusion, while the term independent statistical analysis is used in scientific literature, its meaning varies across studies and it may be incorrectly associated with data integrity. Given the lack of consensus around its meaning, transparency regarding statisticians’ roles and access to the primary data may be better options for ensuring the integrity of the literature.