Methodological survey of missing outcome data in an alteplase for ischemic stroke meta-analysis.

OBJECTIVE: Recent national guidelines recommend alteplase treatment for ischemic stroke within 4.5 h of symptom-onset based on meta-analyses of randomized controlled clinical trials (RCT). A detailed description of missing outcome data (MOD) due to participant loss to follow-up has never been published. The objective of this study was to perform a methodlogical survey on missing outcome data in an alteplase for ischemic stroke meta-analysis.
MATERIALS AND METHODS: A methodological survey was performed on a chosen meta-analysis of alteplase for ischemic stroke RCTs that most closely aligns with recent national guideline recommendations. Data were collected to assess the number of participants lost to follow-up; differential lost to follow-up between allocation groups; baseline characteristics of those lost to follow-up; and the imputation methods used by individual trials and the chosen meta-analysis. The number of participants lost to follow-up was compared with the fragility index; and repeated for individually positive RCTs in the meta-analysis.
RESULTS: The methodological survey revealed a substantial degree of missing information regarding MOD in the chosen meta-analysis and in individual RCTs. Single imputation was exclusively used in all RCTs and in the meta-analysis. The number of participants lost to follow-up was greater than the fragility index in the chosen meta-analysis and individually positive component RCTs suggesting that MOD may impact the direction of the reported effect or effect size.
CONCLUSION: This methodological survey of an alteplase for ischemic stroke meta-analysis revealed MOD may be an important source of unrecognized bias. This survey highlights the need for sensitivity analyses using more robust methods of imputation.

INTRODUCTION

Recent national acute ischemic stroke guidelines recommend alteplase treatment within 4.5 h of symptom‐onset based on randomized controlled clinical trials (RCT) and meta‐analyses of RCTs. Randomized controlled clinical trials and meta‐analyses may be prone to biased estimates due to participant lost to follow‐up and missing outcome data (MOD). , , , , , Additionally, imputation methods vary in their robustness in producing unbiased estimates. , , , For example, single imputation methods are widely considered to result in more biased estimates compared with methods such as inverse probability weighting, multiple imputation, and likelihood‐based‐analysis. , , , An imputation study of a sample of systematic reviews found that MOD can have substantive effects on pooled estimates including effect size and effect direction. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) working group recommends evaluating the robustness of meta‐analyses by reporting sensitivity analyses with varying methods of handling MOD.

The Cochrane tool for assessing the risk of bias in RCTs does not recommend a quantitative threshold for which MOD may be considered a high risk of bias. A quantitative threshold may be derived, however, by calculating a fragility index. A fragility index is defined as the minimal number of non‐events changed to events in the group with fewer events resulting in loss of statistical significance. The fragility index may be thought of as a p‐value for a statistical test transformed into an absolute number of events. If the number of participants lost to follow‐up is greater than the fragility index the effect direction could be altered by MOD. Alternatively, if the number of participants lost to follow‐up is less than the fragility index, only the effect size and not effect direction could be altered by MOD. The fragility index has been used in both individual RCTs and meta‐analyses. ,

A methodological survey on the handling of MOD for individual RCTs or meta‐analyses of alteplase treatment for acute ischemic stroke have not been published. The primary purpose of this methodological survey is to describe MOD and imputation methods; and compare the number of participants lost to follow‐up to the fragility index in a meta‐analysis of alteplase treatment for acute ischemic stroke.

METHODS

Meta‐analyses published by the Stroke Thrombolysis Trialists' (STT) Collaborative Group were reviewed. The STT Collaborative Group is a group of 23 authors who pool patient level data from individual RCTs of alteplase for acute ischemic stroke for meta‐analyses. A structured form was developed to standardize data collection. Data abstracted included the name of the individual RCT; the number of participants lost to follow‐up; imputation method used for the primary endpoint; imputation method used for the mortality endpoint; and data needed to calculate the fragility index for the pooled effect of the primary endpoint. Individual RCTs, supplementary publications, and statistical analysis plans were reviewed for completeness.

The number of participants lost to follow‐up was compared with the fragility index of statistically significant results as has been previously described. , This was repeated for individual component RCTs in the chosen meta‐analysis. Statistical analyses were completed using R software, version 4.1.2 (R Project for Statistical Computing). Institutional Review Board approval was waived for this survey based on institutional guidelines.

RESULTS

The STT Collaborative Group have published six meta‐analyses using the same statistical analysis plan. , , , , , Only one of these meta‐analyses restricted inclusion of trial participants that align with dosing and time window treatment recommendations from national guidelines defined as “a 4.5‐hour‐revised US label.” The purpose of this meta‐analysis was to determine alteplase treatment effects based on recommendations made in national stroke guidelines which differ from current United States (US) and European Union (EU) drug labels, respectively.

The meta‐analysis included eight component RCTs: NINDS A (N = 291); NINDS B (N = 333); ECASS II (N = 800); ATLANTIS A (N = 142); ATLANTIS B (N = 613); ECASS III (N = 821); EPITHET (N = 101); and IST‐3 (N = 3035). Multiple subgroup analyses were reported comparing treatment effects for participants that meet current US and EU regulatory labels to participants that meet current national guideline recommendations written by the American Stroke Association/American Heart Association (ASA/AHA) and the European Stroke Organization (ESO), respectively. Subgroups were further stratified by treatment delay, stroke severity, and age. This meta‐analysis and the eight component RCTs were chosen for this methodological survey. , , , , , , , The primary endpoint for the chosen meta‐analysis was a modified Rankin Scale (mRS) score of 0–1. The mRS is a disability scale that ranges from 0 (no symptoms) to 6 (death). In total, 310 participants had MOD imputed within individual trials. The major results are summarized in Table 1.

TABLE 1

Overview of missing outcome data and imputation methods in component randomized clinical trials (RCT) included in the chosen meta‐analysis

RCT	Number of participants allocated to alteplase	Number of participants allocated to placebo	Fragility Index	Lost To follow‐up	Imputation method for disability endpoint a	Imputation method if vital status unknown
NINDS A 26 , 38	144	147	13	12	LOCF or outcome measured after 3 months	N/A b
NINDS B 26 , 38	168	165	5	11	LOCF or outcome measured after 3 months	N/A b
ECASS II 25 , 38	409	391	NS	37	WCI	WCI
ATLANTIS A 23 , 38	71	71	NS	83	LOCF	LOCF
ATLANTIS B 24 , 38	307	306	NS	47	LOCF	LOCF
ECASS III 28	418	403	1	23	WCI	WCI
EPITHET 22	52	49	NS	1	CCA	N/A b
IST‐3 27	1515	1520	NS	96	LOCF	WCI

Abbreviations: CCA, complete case analysis; LOCF, last observation carried forward; N/A, not applicable; NS, non‐significant hypothesis test; WCI, worst case imputation.

Data for a 3‐month modified Rankin Scale score of 0–1 was used for all RCTs except the IST‐3. For the IST‐3 RCT a 6‐month Oxford Handicap Scale score was used.

Full mortality data reported.

Baseline characteristics of participants with missing outcomes

None of the trials provided information on the baseline characteristics of participants lost to follow‐up including what time window they were treated in or if they suffered intracerebral hemorrhage after treatment. , , , , , ,

Differential missing outcome data

Among the eight included RCTs, ECASS III, EPITHET, and IST‐3 provided information on differential lost to follow‐up between allocation groups. In the ECASS III RCT, 10 placebo‐allocated participants and 13 alteplase‐allocated participants were lost to follow‐up. In the EPITHET RCT, 0 placebo‐allocated participants and 1 alteplase‐allocated participants were lost to follow‐up. In the IST‐3 RCT, 54 placebo‐allocated participants and 42 alteplase‐allocated participants were lost to follow‐up. In the other 5 RCTs, there was no information on differential lost to follow‐up between allocation groups.

Imputation methods

A form of single imputation was used for all RCTs except EPITHET which used complete case analysis (CCA) for disability endpoints (Table 1). , , , , , , , Last observation carried forward (LOCF) was used in both NINDS RCTs and both ATLANTIS RCTs. In the NINDS RCTs, LOCF was only used if outcome data after the 3‐month follow‐up period was unavailable. The ECASS‐2 and ECASS‐3 RCTs used worst‐case imputation (WCI) defined as an mRS of 5 (survival with severe disability) in ECASS‐2 and undefined in ECASS‐3. The IST‐3 RCT and the chosen meta‐analysis differentiated imputation whether vital status was known or unknown at 90 days: LOCF was used for participants whose vital status was known to be alive and WCI defined as an mRS of 5 was used if vital status was unknown. , The chosen meta‐analysis statistical analysis plan does not clarify if imputation was used in addition to imputation done by individual RCTs or in lieu of imputation done by individual RCTs. , The number of additional values imputed by meta‐analysis authors, if any, was not present in any of the published meta‐analyses or supplementary publications. , , , , ,

Number of participants lost to follow‐up compared with fragility index

The fragility index was calculated in participants who would have met a 4.5‐h‐revised US label (N = 3326) in the chosen meta‐analysis for an mRS of 0–1. The number of participants lost to follow‐up (310) was greater than the fragility index (58). Because individual RCTs and meta‐analyses do not provide information on the number of participants lost to follow‐up within time windows or treatment labels, additional sensitivity analyses were conducted. In the chosen meta‐analysis, 54% (N = 3326) of the total enrolled participants (N = 6136) met the revised 4.5‐h‐revised US label. Assuming a similar proportion of participants (N = 168) met the revised 4.5‐h‐revised US label were among those lost to follow‐up, the number lost to follow‐up was 2.9 times higher than the fragility index. Similarly, assuming only half this proportion (N = 84) were among those lost to follow‐up, the number of participants lost to follow‐up was 1.4 times the fragility index. The maximum percentage of participants that could have been lost to follow‐up from participants enrolled within the 4.5‐h‐revised US label is 1.7% before a null result could be achieved.

Of the eight individual RCTs included in the meta‐analysis, 2 reported a positive primary efficacy endpoint: NINDS part B and ECASS III. , The primary endpoint in NINDS part B was a composite endpoint of three disability endpoints and the National Institute of Health Stroke Scale (NIHSS) score. To maintain comparability with the meta‐analysis and ECASS III, the fragility index was calculated based on an mRS score of 0–1—one of the three disability endpoints. The fragility index was 5 for the NINDS part B and 1 for the ECASS III. The number of participants lost to follow‐up, 11 and 23, respectively, was greater than the fragility indices in both RCTs.

DISCUSSION

This methodological survey of an alteplase for acute ischemic stroke meta‐analysis revealed a substantial amount of missing information regarding the handling of MOD. The number of participants lost to follow‐up was greater than the fragility index in the chosen meta‐analysis and both component RCTs that were reported positive with respect to their primary endpoint.

Different methods of imputation incur different tradeoffs in terms of estimating the true effect. Complete case analysis and single imputation are considered naive imputation methods. Complete case analysis is an analysis based only on participants with complete outcome data. This may result in loss of statistical power. Single imputation is an analysis based on a fixed imputation rule. , The chosen meta‐analysis of eight component RCTs exclusively employed single imputation methods (Table 1). Single imputation methods do not use information available in observed values and fail to account for uncertainty in imputed values. , , , Imputation of identical values also artificially limits variability reflected by lower p‐values. , , , Single imputation may be especially problematic given the natural history of participants that suffer ischemic stroke. Stroke survivors may suffer from recurrence of their cerebrovascular disease or chest infections that worsen long‐term outcomes regardless of their early outcomes.

The 90‐day mortality endpoint has a high propensity for biased estimates with current single imputation methods. Only both parts of the NINDS rt‐PA Stroke Study and EPITHET report full mortality data. All five other RCTs and the chosen meta‐analysis imputed a value of survival if vital status was unknown. Given the significantly higher 7‐day mortality in participants allocated to alteplase, this is a potentially biased analytical methodology that would mask excess 90‐day mortality if it were present. This is supported by a recent meta‐analysis of participants treated based on neuroimaging guidance, instead of time from last known well, which reported greater loss to follow‐up and higher 90‐day mortality in alteplase allocated participants. This meta‐analysis reported a CCA eliminating the potential bias of imputing survival if vital status is unknown (Figure 1). In the chosen meta‐analysis, excess mortality decreases between Day 7 and Day 90 with alteplase treatment compared with placebo. In the meta‐analysis of alteplase treatment by neuroimaging guidance, excess mortality increases between Day 7 and Day 90 (Figure 1). Although CCA is considered a naive imputation method, it generally over‐estimates benefit and under‐estimates harm. These findings cast doubt on assumptions made that MOD are missing at random (MAR) and is consistent with prior concerns regarding excess mortality with alteplase treatment.

FIGURE 1

Forest plots comparing early and late mortality in the chosen meta‐analysis (top) and a meta‐analysis of treatment based on neuroimaging guidance (bottom). , In the chosen meta‐analysis with imputation of survival, excess mortality decreases between Day 7 and Day 90. In the meta‐analysis of treatment based on neuroimaging guidance using complete case analysis, excess mortality increases between Day 7 and Day 90. Effect sizes are reported as risk differences (RD) and 95% confidence intervals. The vertical dotted line represents the null (RD = 0). *p value <.05

A major difference between the neuroimaging guidance treatment meta‐analysis and the chosen meta‐analysis are de‐facto exclusions of stroke mimics in the former. Given the marked difference in mortality and likelihood that stroke mimics contribute to survival at 90 days, stroke mimics may be an important source of attrition bias in time‐based treatment meta‐analyses. Meta‐analyses of time‐based treatment are constructed such that both full RCTs and subgroups of RCTs based on treatment time windows are pooled. Therefore, covariate balance such as stroke mimic status cannot be assumed between allocation groups due to pooling of subgroups and full randomized samples. Given the large difference in mortality between ischemic strokes and stroke mimics, a small imbalance could have a significant effect on mortality estimates. This artificial inflation of favorable outcomes caused by inclusion of stroke mimics has been previously described in observational studies. ,

Jakobsen and colleagues have popularized a practical guide for handling MOD. Five conditions favor the reporting best–worst and worst–best sensitivity analyses over imputation: (1) missing data is negligible, (2) missing data is substantial, (3) only dependent variable values missing, (4) data is missing completely at random (MCAR), and (5) data is missing not at random (MNAR). If none of those five conditions are met, multiple imputation is the favored imputation method. The primary benefit of multiple imputation is the use of baseline characteristics as predictors in the imputation model. In the case of acute ischemic stroke, baseline NIHSS score and age account for most of the variance in functional outcome making this an appealing method for imputation. A sensitivity analysis using multiple imputation in the chosen meta‐analysis and the meta‐analysis of alteplase treatment based on neuroimaging guidance could overcome analytical concerns about masking excess 90‐day mortality.

For the primary endpoint of the chosen meta‐analysis, the number of participants lost to follow‐up was greater than the fragility index. The number of participants lost to follow‐up was also greater than the fragility index in both component RCTs with positive primary endpoints. , These results suggest that MOD imputation could change the reported direction of the effect emphasizing the importance of MOD reporting and need for further sensitivity analyses.

The results of this analysis are consistent with previously published data. A methodological survey of a random sample of meta‐analyses found that most meta‐analyses do not report sufficient information on MOD or judge the risk of bias associated with MOD. Additionally, only 5% of sampled meta‐analyses provided results of sensitivity analyses to account for MOD.

The major limitation of this survey and risk of bias assessment is the inability to carry out sensitivity analyses with participant‐level data from RCTs and meta‐analyses using multiple different imputation methods as has been previously done. Sensitivity analyses would also be informative to assess the assumption that the data is MAR. For example, post‐randomization characteristics associated with mortality, such as symptomatic intracerebral hemorrhage, could be compared between allocation groups in participants lost to follow‐up. The NINDS rt‐PA Stroke Study data from both parts is publicly available, but imputed outcome values are not differentiated from true values.

Additionally, the reported methodological survey was performed on one pooled effect from a chosen meta‐analysis while a total of six meta‐analyses using patient‐level data with multiple subgroups have been published. Given that all these meta‐analyses are based on the same individual phase 3 RCTs this is unlikely to be a source of bias in the survey. Although national and international guidelines differ slightly in their recommendations for alteplase for acute ischemic stroke, the RCTs or meta‐analyses used to support recommendations do not; and none of the currently published guidelines recommend a higher dose of alteplase used in one RCT excluded from this survey.

To assess the risk of bias related to MOD, the fragility index was used for which there are inherent limitations. This statistical test is only informative regarding the number of events to change a significant result to a non‐significant result; and gives no information regarding the change, if any, in effect size. This test is more useful, however, than general guidance regarding the quantity of acceptable MOD provided in popular risk of bias tools.

In conclusion, imputation of MOD from participants lost to follow‐up may be an important source of biased pooled estimates in meta‐analyses of alteplase for ischemic stroke. Given the worldwide impact of alteplase use for acute ischemic stroke sensitivity analyses using patient‐level data to assess biased estimates should be reported.

CONFLICT OF INTEREST

The authors have no conflict of interest to declare.

PEER REVIEW

The peer review history for this article is available at https://publons.com/publon/10.1111/ane.13656.