Unsubstantiated conclusions from improper statistical design and analysis of a randomized controlled trial.

Dear Sir,

The proper design and analysis of randomized controlled trials (RCTs) are essential for scientific progress identifying safe and efficacious interventions. We read the RCT recently reported by Jakhotia et al. with interest, which compared the effects of three forms of exercise and a control on health and fitness-related outcomes. Unfortunately, severe problems with the design and analysis of the study make the reported results highly questionable.

Specifically, the allocation of participants is not random. The randomization process generally consists of two steps: (1) Generating an unpredictable random sequence and (2) implementing the sequence in a way that conceals the treatment until subjects have been formally assigned to their groups. The protocol is described as assigning subjects “as they were recruited with n in group 1, n+1 in group 2,… and so on.”[1] It is unclear if they put the blocks of individuals into groups or individuals themselves, but either method is not random if the order of enrollment is exclusively defining treatment allocation. With sequential enrollment, knowledge of the next participant assignment increases the risk of bias to include or exclude a potential participant based on demographic characteristics.[2] By violating both the principles of random allocation, the study reported cannot be properly referred to as an RCT.[2]

There are also concerns with how the statistical analysis was conducted. Table 2 of Jakhotia et al. reports baseline characteristics and tests for differences among the four groups using a repeated measures analysis of variance; this is an inappropriate test because there should only be data from one time point to consider. Furthermore, we attempted to replicate the tests using the provided summary statistics and calculated incompatible results [Table 1]. These errors, including an impossible P > 1 (perhaps a typographical error, as P values are probabilities and must be between 0 and 1), raise further concerns because these calculated baseline imbalances were the basis for the choice of test in later analyses. We found similar errors in Table 3 of Jakhotia et al., where the reported within-group mean difference and standard deviation, 95% confidence interval, test statistic, and P value were incommensurable with one another in several cases.

Table 1

Results from reported baseline “repeated measures” ANOVA and recalculated ANOVA F-tests for an overall difference between circuit, treadmill, Suryanamaskar, and control groups based on Table 2 of the original paper

The objective of this study was to find differences in health-related outcomes between three exercise regimens and a control group, but the core of their conclusions focused on whether pairwise differences within the treatment groups were significantly different from baseline. This approach, where differences in nominal significance within-groups are interpreted as significant differences between the groups, is statistically invalid, producing false positives at a rate up to 87.5% for four groups.[3] That approach should never be used; instead we suggest performing post hoc exploratory pairwise comparisons with a multiple testing correction.

The title of this paper is also misleading; equivalency of treatments can only be properly determined through an equivalency trial,[4] which this study was not. Considering the compromised randomization, the irreproducible baseline statistics, the use of subsequent analytical strategies that depended on the faulty baseline statistics, and the inferences inappropriately based on within-group comparisons, we believe retraction of Jakhotia et al. is consistent with the recommendation of the International Committee of Medical Journal Editors: “Errors serious enough to invalidate a paper's results and conclusions may require retraction”.[5]

Financial support and sponsorship

Supported in part by NIH grants R25HL124208, T32DK062710, R25DK099080, and P30DK056336.

Conflicts of interest

There are no conflicts of interest.