Missing outcomes in randomized trials: addressing the dilemma.

Although randomized trials have been conducted for several decades now, some aspects of their analysis remain contentious. Two such issues are what to do about trial participants who do not adhere to the protocol (for example, if they do not receive the intended treatment) and how to deal with those for whom outcome assessments are missing (for example, because they are lost to follow-up). Both of these issues are relevant to the adoption of so-called “intention to treat” (ITT) analysis – a topic that, not surprisingly, also causes debate.

ITT analysis is widely recommended as the preferred approach to analyzing the outcomes of randomized trials.1,2 In an ITT analysis, all randomized patients are included in the analysis in their assigned groups regardless of all considerations, including whether they in fact received the designated intervention. ITT analysis should therefore compare outcomes in groups that correspond exactly to the randomization scheme. Any deviation from that principle may introduce bias.

An immediate problem is that some data are missing from almost all randomized trials.3 Clearly, just a few missing outcomes will not be a concern, but one review found that, in about half of randomized controlled trials (RCTs), outcomes are missing for more than 10% of participants.4 A major concern is that being lost to follow-up could be related to a patient’s response to the treatment; indeed, we should assume that this will be so. That concern can be compounded if the reasons for, or frequency of, dropout differs between the treatment groups.

No analysis option is ideal here; there is, in effect, a choice between omitting participants without final outcome data or estimating (imputing) the missing outcome data. What should researchers do? A “complete case” (or “available case”) analysis simply omits those for whom data are incomplete. This commonly used approach loses power, and bias may well be introduced, given that the incompleteness of data will not be random. Further, excluding some patients is not compatible with the ITT principle. Imputation of the missing data allows the analysis to conform to ITT analysis but requires strong assumptions that may be hard to justify.5,6 However, some concerns about “making up the data” are misplaced.7

Methods for the imputation of missing values have been the topic of much methodological and empirical research in recent years. The generally preferred imputation methods are quite complex,5-8 and some simple approaches that have been around a long time are much more popular. One of the simplest and most commonly used of these in the analysis of continuous outcomes is “last observation carried forward” (LOCF) analysis, in which missing final values of the outcome variable are replaced by the last known value before the participant was lost to follow-up. LOCF analysis appeals through its simplicity and ease of application, but there are strong grounds for not using it. Specifically, the method may introduce bias in the results, and this bias can, according to circumstance, be in either direction.9 Also, because in LOCF analysis no allowance is made for the uncertainty of imputation, the resulting confidence intervals are too narrow.10

Even if missing outcomes are random across trial participants, LOCF analysis assumes that the missing final values would be the same as the last recorded values. That assumption is often implausible (even as an average), because dropping out is likely to be associated with response to treatment; obvious examples are failure to respond to treatment and adverse effects. In practice, missing data are very likely to be related to response to treatment and prognosis.

Molnar and colleagues have discussed these issues in the specific context of trials of dementia therapies.9,11 They found that 34 of 57 RCTs used LOCF as the only form of ITT analysis. Not surprisingly, that is a much higher proportion than the 19% observed in a review of trials across various medical specialties published in 4 general medical journals.4

As Molnar and colleagues note,11 their study cannot quantify the magnitude of the effect of the use of LOCF analysis on trial results, but it does highlight the high prevalence of conditions for which this method of analysis could promote bias in favour of more toxic therapies and against less toxic alternatives.

Their study focused on a limited number of trials concerning a single medical condition, but many of the issues it raises apply to all trials. It is known that most trials include some patients whose outcomes are ultimately not known, and for which an ITT analysis is not possible without some type of imputation (although authors commonly mislabel available case analyses as ITT).12,13

Whichever imputation approach is used, it is desirable to report analyses with and without imputation. Also, it may be valuable to explore different imputation approaches. As Molnar and colleagues suggest, “the onus is on the investigators who publish these trials to disprove the possibility that these analyses have introduced bias by performing ITT sensitivity analyses … . This is particularly true for those studies demonstrating higher dropout rates in treatment groups.”11

A further important issue is that this information should be included in the report of a trial. The CONSORT (Consolidated Standards for Reporting of Trials) Statement recommends that authors specify the methods used for all statistical analyses reported.14 The statement’s accompanying explanatory paper included various comments on ITT analysis, but there was no specific mention of imputation.15 Noting that omission, Shapiro wrote: “A variety of options are available to handle missing data from participants who drop out of the trial and it is important for readers to know what strategy the investigators adopted. Without such information, the fact that an intent-to-treat analysis was carried out is only partially informative.”16 That omission will be remedied in the forthcoming 2009 update of the CONSORT Statement.

Transparency of reporting facilitates reliable appraisal of the quality and relevance of health research.17 Successful transparency may be judged by whether others are able to reproduce all the methods used. The International Committee of Medical Journal Editors makes the sensible recommendation to “[d]escribe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to verify the reported results” (http://www.icmje.org/). That is very sound advice for all research articles, not just RCTs; indeed, I see no reason why it should not apply to all study methods.

Molnar and colleagues have gone further than Shapiro by suggesting that the CONSORT group should give guidance on methods for analysing trials when data are missing.11 However, because the purpose of the CONSORT group is to give guidance on the reporting of what was done, and not to advise on what is good or bad methodology, such guidance will not come from that source. But, given that so many others have warned about the dangers of LOCF analysis, it seems clear that its use as the sole form of analysis should be discontinued.8,9,16,18,19

Finally, as Liu and colleagues observed, it is important in the design and conduct of studies to try to prevent losses to follow-up and to try to minimize the bias caused by the inevitable missing data.20 To that end, they provide helpful guidance for both the conduct and reporting of RCTs where missing outcomes are likely:

discuss at the planning stage methods and procedures that maximize the chance of retaining patients (e.g., short course of rescue medicine),

continue to collect data post-withdrawal to preserve the ITT population,

document the reasons for missing data,

anticipate and investigate the types of missing data,

pre-specify primary as well as sensitivity statistical analyses,

fully report the extent and pattern of missing data,

support conclusions based on results from the planned analyses with proper sensitivity assessment.20

Adherence to this advice will lead to better design, analysis, reporting and interpretation of future trials in all medical specialties.