Brennan C Kahan1, Tahania Ahmad2, Gordon Forbes3, Suzie Cro4. 1. MRC Clinical Trials Unit at UCL, London, UK. Electronic address: b.kahan@ucl.ac.uk. 2. Pragmatic Clinical Trials Unit, Queen Mary University of London, London, UK. 3. Department of Biostatistics and health informatics, Institute of Psychiatry, Psychology & Neuroscience, Kings College London, London, UK. 4. Imperial Clinical Trials Unit, Imperial College London, London, UK.
Abstract
BACKGROUND AND OBJECTIVE: Prespecification of statistical methods in clinical trial protocols and statistical analysis plans can help to deter bias from p-hacking but is only effective if the prespecified approach is made available. STUDY DESIGN AND SETTING: For 100 randomized trials published in 2018 and indexed in PubMed, we evaluated how often a prespecified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available prespecified analysis, we compared this with the trial publication to identify whether there were unexplained discrepancies. RESULTS: Only 12 of 100 trials (12%) had a publicly available prespecified analysis approach for their primary outcome; this document was dated before recruitment began for only two trials. Of the 12 trials with an available prespecified analysis approach, 11 (92%) had one or more unexplained discrepancies. Only 4 of 100 trials (4%) stated that the statistician was blinded until the SAP was signed off, and only 10 of 100 (10%) stated the statistician was blinded until the database was locked. CONCLUSION: For most published trials, there is insufficient information available to determine whether the results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.
BACKGROUND AND OBJECTIVE: Prespecification of statistical methods in clinical trial protocols and statistical analysis plans can help to deter bias from p-hacking but is only effective if the prespecified approach is made available. STUDY DESIGN AND SETTING: For 100 randomized trials published in 2018 and indexed in PubMed, we evaluated how often a prespecified statistical analysis approach for the trial's primary outcome was publicly available. For each trial with an available prespecified analysis, we compared this with the trial publication to identify whether there were unexplained discrepancies. RESULTS: Only 12 of 100 trials (12%) had a publicly available prespecified analysis approach for their primary outcome; this document was dated before recruitment began for only two trials. Of the 12 trials with an available prespecified analysis approach, 11 (92%) had one or more unexplained discrepancies. Only 4 of 100 trials (4%) stated that the statistician was blinded until the SAP was signed off, and only 10 of 100 (10%) stated the statistician was blinded until the database was locked. CONCLUSION: For most published trials, there is insufficient information available to determine whether the results may be subject to p-hacking. Where information was available, there were often unexplained discrepancies between the prespecified and final analysis methods.