Have last-observation-carried-forward analyses caused us to favour more toxic dementia therapies over less toxic alternatives? A systematic review.

BACKGROUND: Intention-to-treat analysis is used in the analysis of randomized controlled trials to preserve trial power in the presence of missing subject data as well as to control for both known and unknown confounding factors. One form of intention-to-treat analysis is last-observation-carried-forward (LOCF). Concerns exist regarding whether it is appropriate to use LOCF in analyses involving progressive conditions or in situations where missing data are non-random (e.g., subjects drop out because of treatment side effects or differing disease severity).
OBJECTIVE: To examine the use of intention-to-treat imputation of missing data techniques, and specifically LOCF, in randomized controlled trials of the use of cholinesterase inhibitors and memantine to treat Alzheimer's disease, vascular dementia, mixed dementia and mild cognitive impairment.
METHODS: We conducted a systematic electronic search of MEDLINE and the Cochrane Central Register of Controlled Trials from 1984 to 2008 for double-blinded, randomized controlled trials of cholinesterase inhibitors or memantine that examined progressive symptoms in Alzheimer's disease, vascular dementia, mixed dementia and mild cognitive impairment. We collected data on the use of intention-to-treat and non-intention-to-treat analyses and on contraindications to the use of LOCF analysis and we performed quality assessments of included trials.
RESULTS: Of the 57 studies that met the inclusion criteria, 12 did not report intention-to-treat analyses. Of the 34 studies that employed LOCF as the only form of intention-to-treat analysis, 24 reported conditions that could produce biased LOCF analyses favouring the drug under study. The latter finding was more common in cholinesterase inhibitor trials than in memantine studies.
CONCLUSIONS: The published results of some randomized controlled trials of dementia drugs may be inaccurate (i.e., drug effectiveness may be exaggerated) or invalid (i.e., there may be false-positive results) because of bias introduced through the inappropriate use of LOCF analyses. This bias favours cholinesterase inhibitors, potentially preventing funding of and patient access to less toxic treatment options such as memantine. Licensing agencies should consider whether to accept LOCF analyses in research on dementias and other chronic progressive conditions.

It is estimated that 24.3 million people worldwide suffer from dementia and that annual costs for Alzheimer’s disease are as high as $155 billion in the United States (1996 US dollars).1,2 One potential way to decrease the negative impact of dementia on people with this condition, on their families and on societies is to optimize the use of dementia medications,2 with due consideration of both their effectiveness and their toxicity.

The effectiveness of most medications is tested via randomized controlled trials (RCTs). It is inevitable that some participants drop out of such studies before they are completed. Unfortunately, if analyses include only participants who remain in the trial, then study power is lost and erroneous conclusions may be generated. The principle of intention-to-treat (ITT) analysis, in which all patients are included in the analysis according to the group to which they were assigned at randomization, has become the accepted standard for the analysis of RCTs to try to counteract this problem.3 The strength of ITT analysis is that it not only preserves power but also promotes balance between treatment groups for both known and unknown confounders, thereby preserving the benefits of randomization.

Ideally, all possible data are collected on all subjects, including those who drop out of the study; however, this is not always possible. In order for ITT approaches to analyze all patients randomly assigned to a group, several methods to impute missing data have been developed.3-10 Unfortunately, no statistical strategy can deal fully with all the different combinations of reasons for dropping out, dropout rates and different disease courses. At best, these techniques to impute missing data are educated estimates. One commonly employed technique to impute missing data is last-observation-carried-forward (LOCF), also known as end-point analysis.

LOCF substitutes subjects’ missing outcomes with the last measurement taken before they dropped out. It requires that 2 basic assumptions be met: the subjects’ responses would have been constant from the last observed value (i.e., the point at which they dropped out) to the end point of the trial; and, missing values are missing completely at random (i.e., dropout is not related to variables such as drug side effects, group assignment, disease severity or symptoms).5-7

Authors have highlighted 3 factors that cause the second condition to be breached in a manner that introduces bias that will exaggerate the effectiveness of treatments as estimated by LOCF analyses; these include earlier dropouts or greater dropout rates in the treatment group and more rapid disease progression in subjects who drop out of the treatment group.3,4,9-11 These factors result in more subjects who drop out of the treatment group having their decline artificially frozen at an earlier stage of disease, thereby potentially biasing results in favour of the drug under study (i.e., overestimating effectiveness relative to the placebo). By extension, study results may also be biased against the drug under study (i.e., underestimating effectiveness) if there are earlier dropouts or greater dropout rates in the control group or if there are subjects whose disease progresses more rapidly among those who drop out of the control group (Figures 1 and 2)

Figure 1

Differential last observation carried forward (LOCF) bias when there are more or earlier dropouts in the treatment group than the control group.

Figure 2

Differential last observation carried forward (LOCF) bias when there are more or earlier dropouts in the control group than in the treatment group.

Since 1998, researchers have expressed concern that the use of LOCF in dementia drug trials contravenes the assumption of disease stability and the assumption of random missing data and hence risks generating biased results.2,11-21 To better understand the significance of these concerns in dementia research we systematically reviewed the use of ITT and LOCF analyses, contraindications to the use of LOCF analysis, and the use of non-ITT analyses in RCTs of drugs approved for the treatment of Alzheimer’s disease, vascular dementia, mixed dementia and mild cognitive impairment in Canada (i.e., cholinesterase inhibitors such as donepezil, rivastigmine and galantamine, and the N-methyl-D-aspartate (NMDA) receptor antagonist memantine).

Methods

We performed an electronic literature search of MEDLINE and the Cochrane Central Register of Controlled Trials from January 1984 (the year of publication of the McKhann criteria for Alzheimer’s disease22) to February 2008 using the OVID search interface. The search strategy included the following terms: randomized controlled trials, dementia, Alzheimer, vascular dementia, mixed dementia, donepezil, Aricept, rivastigmine, Exelon, galantamine, Reminyl, memantine, Ebixa and cholinesterase inhibitor.

The principal investigator reviewed titles and abstracts to select an overly inclusive list of potential articles to be subjected to a full review of text and reference sections to identify relevant RCTs. The full text of selected RCT reports was then independently reviewed by 2 certified specialists in geriatric medicine with clinical expertise in dementia, formal research methodological training and recognized expertise in the review of dementia drug trials to determine which RCT reports met the inclusion criteria for the systematic review.

Inclusion criteria

We included double-blinded, randomized controlled trials of cholinesterase inhibitors or memantine that examined progressive symptoms (e.g., cognition, function) in Alzheimer’s disease, vascular dementia, mixed dementia or mild cognitive impairment and that employed DSM-IV (Diagnostic and Statistical Manual of Mental Disorders)23 or NINCDS–ADRDA (National Institute of Neurological and Communicative Disorders and Stroke – Alzheimer’s Disease and Related Disorders Association)22 criteria for Alzheimer’s disease or NINDS–AIREN (National Institute of Neurological Disorders and Stroke – Association Internationale pour la Recherche et l’Enseignement en Neurosciences) criteria for vascular dementia. Trials of cholinesterase inhibitors not currently licensed in Canada (tacrine, metrifonate) were not reviewed. The systematic review was restricted to studies with full trial reports published in English-language peer-reviewed journals. The diagnostic criteria for mild cognitive impairment were not specified, as they were in development when the relevant studies were published.

Although the reference sections of open-label studies, reviews, meta-analyses, commentaries, editorials, studies of pooled data from previous studies and tolerability and safety studies were searched for relevant RCTs, the articles themselves were not included in the systematic review. Subgroup analyses and secondary or retrospective analyses were also excluded.

Data collection

Data collected included publication details, investigative site locations, funding, drug comparators, drug doses, diagnostic criteria employed, type(s) of analysis employed, discussion of the limitations of the forms of analysis employed, dropout characteristics (e.g., number, timing, patient characteristics, reasons for dropout), contraindications to the use of LOCF and the results of each study’s primary and secondary outcome measures.

The 2 previously mentioned reviewers independently extracted data from all included studies and then met to review their findings and discuss discrepancies. When consensus could not be achieved, discrepancies were forwarded to a third party for independent review.

Results

Of the 1146 articles identified by the search strategies, 191 papers (including RCT reports, reports of non-randomized trials, commentaries, systematic reviews and meta-analyses) were selected for full text and reference section review. Of these, 57 RCT reports met the eligibility criteria for systematic review (Fig. 3).2,14,20,21,26-79

Figure 3

Selection of studies for review

Reviewer agreement

Although agreement on abstracted items was not formally measured, the methods employed resulted in consensus on almost all abstracted items. After the reasons for different ratings were explained (differences were mostly a result of difficulty finding the relevant data in the studies reviewed), the reviewers agreed on all but 5 final ratings. These were arbitrated by a third party. The kappa score, if it had been measured, would have been unusually high.

Trial characteristics

Details of the 57 included trials are provided in Tables 1 and 2. Forty-five studies enrolled patients with Alzheimer’s disease (21 involved donepezil, 11 rivastigmine [1 of these studies was a donepezil–rivastigmine comparison study], 7 galantamine and 6 memantine), 8 studies enrolled patients with vascular dementia or mixed dementia (3 involved donepezil, 3 galantamine and 2 memantine) and 4 studies enrolled patients with mild cognitive impairment (2 involved donepezil, 1 rivastigmine and 1 galantamine).

Table 1

Characteristics of included studies (n = 57)

Table 2

Types of ITT and non-ITT analyses employed and number of contraindications to LOCF analysis

In 40 trials there was an explicit statement of pharmaceutical industry funding. In 6 trials industry funding was implied (the authors were pharmaceutical industry employees but the source of funding was not explicitly stated). Three studies were funded by industry in partnership with public funders, and 4 studies were entirely publicly funded (Table 1). The source of funding for 4 studies could not be determined. All 57 study reports were rated as demonstrating high-quality methodology with a Jadad–Schultz score greater than or equal to 3 (Table 1).

Reporting of dropouts

Data on dropouts are provided in Tables 3 and 4. Dropouts were described in 94% of cholinesterase inhibitor studies and 100% of memantine trials. Seven of the 49 cholinesterase inhibitor trials (14%) and none of the memantine trials reported data on the timing of dropouts. The reasons for dropout were often difficult to discern, as many were described as adverse events that might have been due to drug side effects but were not reported as such. Cholinesterase inhibitor studies were more likely than memantine studies to demonstrate a higher dropout rate in the treatment group than in the control group (73% of cholinesterase inhibitor studies v. 25% of memantine studies). When cholinesterase inhibitor studies were combined there was a higher dropout rate in the treatment group than in the control group (23.2% in the treatment group v. 16.8% in the control group) (Table 4). When memantine trials were combined the opposite pattern was noted: there were fewer dropouts in the treatment group than in the control group (14.6 % in the treatment group v. 18.5% in the control group) (Table 4). Ten studies (18%) discussed potential bias associated with dropouts.

Table 3

Account of dropouts by drug class

Table 4

Combined data for cholinesterase inhibitor and memantine trials

Types of non-ITT analyses conducted

The most common non-ITT analysis (employed in 35 trials) was observed case analysis (Table 2). Other forms of non-ITT analysis included fully evaluable population analysis (5 RCTs), treatment per protocol analysis (5 RCTs) and completer analysis (3 RCTs) (Table 2).

Types of ITT analyses conducted

Twelve (21%) of the 57 studies did not identify the type of analysis performed (5 studies) or performed only non-ITT analysis (7 studies) (Table 2). Of the 45 studies in which an identifiable form of ITT analysis was performed, 42 (93%) employed LOCF (Table 2). Thirty-four of the trials in which ITT analysis was performed (76%) relied on LOCF as the only form of ITT analysis (Table 2).

Ten of the 57 studies (17.5%) reported employing ITT techniques other than LOCF (Table 2); 6 of 49 cholinesterase inhibitor studies (12%) and 4 of 8 memantine studies (50%) employed ITT techniques other than LOCF. The 6 alternative approaches for ITT imputation of missing data included the following: replacement of missing values with the mean changes observed in the placebo group;42, 58 time-response relationship for change in ADAS-cog/11 (the Alzheimer Disease Assessment Scale – Cognitive Subscale 11-item) score analyzed using generalized linear modelling;50,52 mixed-effects modelling;56,63 mixed-models repeated measures;78,79 replacement of missing data with worst ranks;57 and sensitivity analyses consisting of a number of simulations.68

Of the 42 studies employing LOCF, only 8 reported performing another type of ITT analysis to confirm the results. In 3 of these 8 studies the authors did not comment on the results of the alternative non-LOCF ITT analysis. In 4 of the 5 studies in which the authors commented on the results of the alternative ITT analysis, they did not report the values calculated by this analysis but they did indicate that the direction of the results was unchanged. It is uncertain whether the point estimates of the outcomes were similar when the alternative ITT analyses were performed.

In only 1 study were the point estimates of outcomes measuring drug efficacy generated by LOCF verified with point estimates generated by an alternative form of ITT analysis.42 The values of 3 positive outcomes were verified in this study.

Contraindications to the use of LOCF as the only form of ITT analysis

Of the 34 studies employing LOCF as the only form of ITT analysis, 24 (71%) explicitly demonstrated contraindications (factors that could introduce bias) to its use. It was unclear whether the remaining 10 studies were free of contraindications, because most studies failed to report adequate data regarding the timing of dropouts and the severity of disease of the participants who dropped out. Consequently, Table 2 provides a range of potential contraindications to the use of LOCF for each study (the lower number representing the number of explicitly identified contraindications). Seven of the 57 trials in this review (12%) discussed the limitations of LOCF or non-ITT approaches.

Discussion

Despite previously published cautions that LOCF analysis may introduce bias into dementia research, LOCF remains the most widely employed analytic technique in this research area; its results are rarely verified by other forms of ITT analysis. Further, the majority of the publications reviewed in the present study did not report the results of an ITT analysis, did not verify the results of LOCF with alternative ITT analyses when conditions that could introduce LOCF analytic bias in favour of the study drug existed, or did not comment on the results of alternative ITT analyses that were performed.

These problems were particularly evident in cholinesterase inhibitor trials. In the majority of these trials, either no ITT results were provided or LOCF ITT analysis was performed in the presence of contraindicating factors. For example, a higher dropout rate in the treatment group than in the control group was more common in cholinesterase inhibitor studies than in memantine studies (73% v. 25%), potentially biasing study results in favour of cholinesterase inhibitors and against memantine. Owing to a lack of data on the timing of dropouts and on the severity of disease in study participants who dropped out, our results may in fact underestimate the true prevalence of conditions promoting bias.

The concern that LOCF analysis introduces bias can be explored via ITT sensitivity analyses. If similar outcomes are generated when other forms of ITT analysis are employed, this provides some reassurance (but does not guarantee) that LOCF analytic bias does not alter results. Only 1 study verified the point estimates of efficacy calculated by LOCF analysis with an alternative ITT analysis.42 The 3 positive point estimates verified by alternative ITT analyses in this study are the only ones out of the hundreds of positive outcomes reported for LOCF analyses in dementia trials to have been verified in this way.

Some may erroneously argue that results of previous studies have been adequately confirmed by non-ITT analyses (i.e., techniques that exclude subjects without data from analysis), such as observed case analysis, completer analysis, fully evaluable population analysis or treated-per-protocol analysis. Like LOCF analysis, these non-ITT techniques may be systematically biased in favour of the group with greater, earlier or more severely affected dropouts and, consequently, they are not reliable, valid sensitivity analyses. The biases inherent in these non-ITT techniques have been highlighted by the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use,80 by the European Agency for the Evaluation of Medicinal Products14 and by a number of authors.15,53,67,81

Furthermore, the use of such non-ITT techniques and of LOCF analysis is completely unnecessary: other forms of ITT analysis that do not treat dropouts artificially by freezing values at the point of dropout but rather model for expected natural decline in dropouts could easily be performed either as the primary analysis or as a sensitivity analysis. The available approaches range from techniques that simply apply the rate of decline noted in the control group to all dropouts to more complex modelling procedures that are available in standard statistical programs. More appropriate forms of analysis have been employed in dementia research.42,50,52,56-58,68,78,79 Petersen’s study of mild cognitive impairment68 may serve as a model for future research, as it employs both modelling for dropouts and sensitivity analyses of the effect of various modelling assumptions and approaches.

The present study cannot quantify the magnitude of the effect of the use of LOCF analysis on trial results; it is restricted to highlighting the high prevalence of conditions promoting bias in favour of more toxic therapies and against less toxic alternatives, such as memantine. As verification of results obtained using non-ITT and LOCF analyses requires individual patient data that are not publicly available, the onus is on the investigators who publish these trials to disprove the possibility that these analyses have introduced bias by performing ITT sensitivity analyses as performed in Petersen’s study of mild cognitive impairment.68 This is particularly true for those studies demonstrating higher dropout rates in treatment groups.

These results are meaningful in day-to-day clinical care. Because this bias has likely exaggerated results in favour of more toxic therapies (e.g., cholinesterase inhibitors), this may have created inappropriate barriers to the funding and prescription of less toxic treatment options for dementia (e.g., memantine). Without accurate analyses, physicians cannot optimally counsel patients and families regarding appropriate therapies, and patients and families cannot provide truly informed consent when making treatment decisions. In addition, meta-analyses and pharmacoeconomic studies cannot be performed accurately and we cannot make reliable statements regarding whether trial results truly cross thresholds of clinical significance. These concerns, as well as the fact that LOCF analytic bias may prevent the funding and use of future less toxic treatments, should be of great concern to patient advocacy groups, such as the Alzheimer Society of Canada and the US Alzheimer’s Association.

In summary, it is highly unlikely, given the high prevalence of conditions promoting LOCF analytic bias in this study, that point estimates of some of the hundreds of positive outcomes generated in trials have not been affected in some way. The question is likely not whether bias been introduced, but rather the number of outcomes that have been biased and the degree to which they have been biased. As such, the present results provide empirical support for recent recommendations to researchers, licensing bodies and research guidelines bodies82 regarding their use of LOCF analysis. One of these recommendations is that the CONSORT group (www.consort-statement.org) consider incorporating guidelines regarding appropriate analyses for studies of medications used to treat chronic progressive disorders into the CONSORT statement so that journal editors, funding agencies, ethics review boards and drug formulary committees can request that these recommendations be followed in future studies of dementia and other chronic progressive disorders. In the meantime, researchers should ensure that analyses promoting bias are avoided or scrutinized using alternative ITT sensitivity analyses. Further, licensing agencies (e.g., the US Food and Drug Administration, the European Agency for the Evaluation of Medicinal Products, and Health Canada) should review this situation immediately to determine whether they will continue to accept LOCF analyses in research on dementia and other chronic progressive conditions.