Evaluating the evaluation.

Dear editor

Zhang et al1 sought to determine which adjustment method is the best. That is a laudable objective, but their approach leaves quite a bit to be desired. When we cut to the chase, we find that they pre-supposed that the analysis of covariance (ANCOVA) was ideal, and, presumably, confirmed this empirically by noting that the ANCOVA results were most aligned with the ANCOVA gold standard. This is fairly perplexing logic. Had any of the other methods been chosen instead as the gold standard, then that method would have been found to be the best by virtue of agreeing with its own results. This is hardly a compelling endorsement. Beyond that, even if the authors did use a more reasoned approach, how can one trial be used to validate an analysis?

An analysis is good or bad based on how well its results align with the underlying reality of the situation. In a simulation study, we would know this reality. In actual trials, we do not. There is no gold standard. Moreover, there is only one trial being considered. This is most assuredly not the way to compare analysis techniques. It is worth noting, however, that ANCOVA relies on normality, among other assumptions, for its validity. Since the data are never actually normally distributed, the method is never technically valid.2 This indisputable fact should give us pause before we blindly accept so fanciful a method. There is a valid and exact method that is based on a ranking of the pairs, pre and post, without having to make any assumptions at all.3 Surely, this method, which was developed for categorical data but applies equally well to continuous data, might have been considered as well. Finally, it is stated that “no methods are available for analysis of data that are ‘missing not at random’.” This is patently untrue4 and almost reaches the level of lunacy attained two sentences later when it is stated that because there were not much missing data, the data were assumed to be missing at random. Missing data can never be demonstrated to be missing at random. This is an entirely academic construct with no application in the real world.5

Dear editor

We thank Dr Vance Berger for his interest in and insightful discussion of our paper.1 Essentially, we agree with his sentiments, however we need to clarify the central points of his discussion, namely: 1) the objective of the study; and 2) the handling of missing data in our statistical analyses.

The objective of the study: we designed and conducted the study to assess the sensitivity or robustness2 of the findings from the original trial (MOBILE trial)3 by varying a specific factor in the analysis, in this case the method of analysis. We judged robustness based on the magnitude, direction, and statistical significance of the effect estimate. The stated objectives of the study are clear on this goal. We did not intend to compare the methods on the basis of their statistical properties—something we agree with the author on, that can be done only through simulation. We also discuss this issue in the Discussion section, supplemented with findings from published simulation studies.

The “missing at random (MAR)” assumption in multiple imputation: again we agree with the author that the assumption of MAR cannot be verified. Given this potential limitation, a commonly used approach to handle missing data is to assess the impact of “missingness” on the findings through some sensitivity analyses. The goal is to check whether, under certain assumptions of missingness, the findings would remain robust if the missing data were imputed through some imputation strategy. We did this in our study and we found that the results remained robust irrespective of the method of handling missing data. We also discuss the limitations of the imputation methods used to assess robustness in the Discussion section. We make no claims about the validity of the MAR assumption.

We hope that this response provides some clarity on the objectives and context to our paper.