Multivariable Prediction Model for Futile Recanalization Therapies in Patients With Acute Ischemic Stroke.

BACKGROUND AND OBJECTIVES: Very poor outcome despite intravenous thrombolysis (IVT) and mechanical thrombectomy (MT) occurs in about 1 of 4 patients with ischemic stroke and is associated with a high logistic and economic burden. We aimed to develop and validate a multivariable prognostic model to identify futile recanalization therapies (FRT) in patients undergoing those therapies.
MATERIALS AND METHODS: Patients from a prospectively collected observational registry of a single academic stroke center treated with MT and/or IVT were included. The dataset was split into a training (N=1808, 80%) and internal validation (N=453, 20%) cohort. We used gradient boosted decision tree machine-learning models after k-NN imputation of 32 variables available at admission to predict FRT defined as modified Rankin-Scale (mRS) 5-6 at 3 months. We report feature importance, ability for discrimination, calibration and decision curve analysis.
RESULTS: 2261 patients with a median (IQR) age 75 years (64-83), 46% female, median NIHSS 9 (4-17), 34% IVT alone, 41% MT alone, 25% bridging were included. Overall 539 (24%) had FRT, more often in MT alone (34%) as compared to IVT alone (11%). Feature importance identified clinical variables (stroke severity, age, active cancer, prestroke disability), laboratory values (glucose, CRP, creatinine), imaging biomarkers (white matter hyperintensities) and onset-to-admission time as the most important predictors. The final model was discriminatory for predicting 3-month FRT (AUC 0.87, 95% CI 0.87-0.88) and had good calibration (Brier 0.12, 0.11-0.12). Overall performance was moderate (F1-score 0.63 ± 0.004) and decision curve analyses suggested higher mean net benefit at lower thresholds of treatment (up to 0.8).
CONCLUSIONS: This FRT prediction model can help inform shared decision making and identify the most relevant features in the emergency setting. While it might be particularly useful in low resource healthcare settings, incorporation of further multifaceted variables is necessary to further increase the predictive performance.

Even with the outstanding efficacy of mechanical thrombectomy (MT) for treatment of large-vessel acute ischemic stroke (AIS), 1 in 5 patients in the randomized controlled trials[1] and 1 in 3 patients in real-world settings[2] have very poor long-term outcome (modified Rankin Scale [mRS] 5–6 at 90 days) despite a technically successful intervention. The term futile intervention was coined to elucidate this issue.[3-5] Similarly, 1 in 5 patients receiving IV thrombolysis (IVT) has very poor long-term outcome despite best available treatment with even higher prevalence rates in elderly patients.[6]

Given the need for informed shared decision making and the high societal and health economics burden of both treatments, there is a need to proof that MT remains (cost-)effective in patients at high risk for futile recanalization therapies (FRTs).[4,7] Although individual patients' preferences may vary, the 5-year quality-adjusted life expectancy in stroke survivors reaching mRS 5 is minimal (0.06).[8-10] Several clinical and imaging biomarkers associated with FRT have been identified.[4,11-18] However, the models developed for MT are insufficient to reliably inform patient and proxies and to guide the decision-making in individual patients.[19-22] Besides the MR PREDICT tool,[23] which can be used for patients undergoing bridging IVT, no model for IVT-only patients not fulfilling endovascular trial criteria has been developed in the context of contemporary MT. A reliable prediction algorithm to identify patients who will have FRT with variables available at baseline is missing. Such an algorithm is needed to realistically inform patient and proxies about the potential risk/benefits of treatment and facilitate patient-oriented informed decision (e.g., withholding maximal therapy if advanced directives state that the patient does not wish to live with severe dependency). In addition, because, in several countries, access to MT and qualified personnel is limited,[24] but with rising strain and workload,[25] reliable FRT prediction could accommodate the rising demand and increasing healthcare costs.

We hypothesized that the combination of several clinical, laboratory, imaging, and workflow variables would enable accurate prediction of FRT after both IVT and MT to better inform shared decision making and possibly even to avoid futile therapies. In addition, we wanted to perform internal validation of the developed model, analyze the clinical utility of the final model, and analyze potential differences in predictors of FRT between patients with and without detectable vessel occlusion.

Methods

This study adheres to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis statement.[26,27]

We included consecutive adult patients from the registry treated with IVT and/or MT between January 2015 and October 2020 and excluded patients with missing outcome at 3 months (12%, n = 322). Patients with missing outcome had a slightly worse prognostic profile with a higher stroke severity, more history of transient ischemic attack, longer onset-to-admission times, and more frequent MT (see eTable 1, links.lww.com/WNL/C129 for full comparison). Patients had to receive IVT alone, MT alone, or a combination (bridging approach). At our center, we have a liberal approach of IVT and/or MT indications performing treatment also in borderline indications such as low stroke severity, distal occlusions, extended time window, and prestroke impairment (see our guidelines for detailed indications/contraindications).[28]

The purpose of the developed model is the prospective prediction of very poor 3-month functional outcome in patients with AIS who are potential candidates for IVT and/or MT. The intended use of the machine learning (ML) algorithm in the acute stroke workflow has been published and is presented in Figure 1.[29] In brief, it is intended to inform shared decision making with the patient and next of kin based on the probability of FRT, after the decision for IVT and/or MT treatment has been made, but before treatment has been actually started.

Figure 1

Intended Use of the Model in a Stroke Workflow

IVT = IV thrombolysis; ML = machine learning; mRS = modified Rankin scale; MT = mechanical thrombectomy. The ML model outputs a probability (risk score) for mRS 5–6 based on variables available ahead of recanalization therapies. This information is provided to the treating physicians after selection of patient for recanalization and could serve as a marker for futile recanalization. Workflow derived from Meier et al. with permission.[29]

Standard Protocol Approvals, Registrations, and Patient Consents

Approval by local ethics committee was granted (Kantonale Ethikkommission, Bern, Switzerland: ID 231/2014 and ID 2020-01696) waiving consent of participants in accordance with Swiss Law.

We randomly split the final data set (n = 2,261) in the training set (80%, n = 1,808) used for development and internal validation (20%, n = 453). The flowchart showing patient inclusion and the study design is shown in eFigure 1 (links.lww.com/WNL/C129).

Clinical, laboratory, imaging, and workflow variables were collected prospectively by dedicated research staff, and missing values completed by using the clinical information system. Initially, 32 baseline admission variables were considered, but we dropped input variables (features) that were not routinely registered when more than 25% of the values were missing. Dropped variables included D-dimer, high-density lipoprotein cholesterol, hemoglobin A1c, triglycerides, troponin, and activated partial thromboplastin time. See eTable 2 (links.lww.com/WNL/C129) for a full list of features considered and their respective definitions. The choice of independent variables was based on existing literature on pathophysiologically plausible associations with FRT that were available and documented in a sufficiently high data quality at our institution.

FRT was defined as mRS 5–6 at 3 months dichotomizing the outcome as a binary target variable and was assessed by certified physicians during routine clinical visits or certified study nurses by semistructured telephone interviews. Death was assessed through linkage with the national mortality registry. Assessors were not blinded but unaware of this project at the time of assessment. Classes of FRT were imbalanced (24% mRS ≥5, 76% mRS <5, ratio 3.2:1). We imputed missing data with k-nearest neighbor (k = 15 neighbors) imputation on normalized data of the training and validation separately. We normalized features to [0,1] interval with min-max normalization.

We implemented Gradient Boosting (XGBoost classifier) by means of Python (3.7.7) scikit learn[30] (0.22.1), XGBoost[31] (1.2.0, for XGBoost classifier) based on a previous project showing that this algorithm had good overall performance and high robustness.[29]

We used a nested, stratified 10-fold cross-validation strategy for model development (XGBoost classifier).[29] In the outer cross-validation loop, the training data set was split into 10 equally sized subsets.[29] Nine of the 10 subsets were used for training and 1 for validation. In the inner loop, hyperparameter optimization was performed based on maximization of F1-score in a 10-fold randomized grid search using the data of the previously formed 9 folds.[29]

For validation, we trained ML algorithms using the setting of the nested cross-validation's inner loop on the complete training data (10-fold randomized grid search) and applied the resulting ML model to separate validation data (n = 453).[29] We repeated this process 20 times using a different random seed for algorithm initialization in each run.

Statistical Analysis

We assessed univariate associations of clinical variables with FRT in the overall cohort as well as training and validation data separately using standard descriptive statistics: χ2 and Fisher exact tests for categorical variables, the Mann-Whitney U test for non-normally continuous or ordinally scaled variables, and the Welsch t test for independent normally distributed data. We used the pmsampsize Stata Package (Stata Statistical Software: Release 16; StataCorp, College Station, TX) to calculate the minimum sample size required using 40 candidate predictors based on an assumed outcome prevalence of 25% and a lower bound for the new model's R2 value of 0.25.[32] This resulted in a final sample size of 1,230 patients.

We report model discrimination, calibration, and clinical utility. For discrimination of the models at the threshold of p = 0.5, we report precision, recall, F1-score, accuracy, balanced accuracy, specificity, and Matthew correlation. Furthermore, we report the area under the curve (AUC) of the receiver operating characteristic (ROC) and the average precision score throughout all possible thresholds. For calibration of the model, we report Brier score and expected calibration error grounded on 10 bins. We computed feature importance using Shapley values for the XGBClassifier. Permutation feature importance was defined as a decrease in F1-score by shuffling the values of a single feature randomly. Decision curve analysis[33] is reported to quantify the clinical utility of the model on validation data. Because the a priori probability threshold will vary according to healthcare resources, we did not define a fix a priori threshold for risk of FRT to assess the net benefit. The results are reported as mean and CI based on 10-fold cross-validation (model development). Performance on validation data was reported as mean ±1 SD over 20 runs with different random seeds for algorithm initialization.

Data Availability

Investigators may request access to anonymized individual patient data including analysis-ready data sets, and data set specifications, after publication. Before using the data, proposals need to be approved by an independent review panel at swissethics.ch/basec, and a signed data sharing agreement will then be approved.

Results

Baseline Characteristics

The final cohort included 2,261 patients: median (interquartile range) age 75 years (64–83 years), 46% female, and median NIH Stroke Scale 9 (4–17). In total, 34% of patients received IVT alone, 41% MT alone, and 25% both acute recanalization treatments (bridging approach). FRT occurred overall in 24% of patients and more often in patients receiving MT alone (34%) as compared with bridging patients (26%) and IVT alone (11%).

Baseline characteristics of patients in training and validation set are presented in Table 1. Taken together, older age, higher stroke severity, active cancer, and a higher cardiovascular risk profile were associated with FRT on univariate analysis.

Table 1

Baseline Characteristics of all Included Patients

Discrimination of ML methods was good. Performance and a complete overview of the results are presented in Table 2. Overall, discrimination for predicting FRT (AUC 0.87 95% CI 0.87–0.88) and calibration (Brier 0.12, 0.11–0.12) was good, and overall performance was moderate (F1-score 0.63 ± 0.004) in the validation data set (see eFigures 2 and 3, links.lww.com/WNL/C129, for ROC curves of the full model in the derivation and validation cohort).

Table 2

Performance of the Model for mRS 5–6 Prediction

Shapley values feature importance revealed that the most important features included clinical variables (higher stroke severity, older age, active cancer, prestroke disability), laboratory values (higher glucose, higher C-reactive protein [CRP], less dyslipidemia), imaging biomarkers (more white matter hyperintensities), and longer onset-to-admission time. The feature importances are shown in Figure 2 (see eFigures 4 and 5, links.lww.com/WNL/C129, for all features in patients with and without detectable vessel occlusion).

Figure 2

Shapley Values Feature Importance

Computed for the XGBoost (mean values of 10 feature permutations and 20 random initializations). (A) All patients, (B) only patients with detectable vessel occlusion, and (C) only patients without detectable vessel occlusion. See the Supplement (links.lww.com/WNL/C129) for full model and features. Frame indicates the 10 most important variables. NIHSS = NIH Stroke Scale.

In patients without detectable vessel occlusion, higher international normalized ratio and atrial fibrillation seemed to be more important, whereas white matter hyperintensity severity and active cancer seemed to be less important.

The decision curve for XGBClassifier is shown in Figure 3. The mean net benefit for p = 0.8 was minimal (0.02 ± 0.01), and relevant net benefit was only present in lower a priori thresholds. The net benefit—in this context—is weighing the profit obtained by classifying an individual with the outcome and the loss caused by falsely classifying an individual without the outcome.[34] No evidence for harm was present throughout all thresholds.

Figure 3

Decision Curve Analysis for XGBClassifier

The blue line indicates the mean over 20 random initializations. The rug plot shows the samples used for computation of the decision curve. No risk denotes the trivial strategy of always predicting mRS 0–4, and All risk denotes the strategy of always predicting mRS 5–6. mRS = modified Rankin scale.

Discussion

The development and validation of a multivariable prediction model for futile thrombolysis and thrombectomy showed the following main findings: (1) FRT overall occurs in 1 in 4 patients and more often in patients receiving MT alone (34%) as compared with IVT alone (11%). (2) The most relevant predictors of FRT included clinical variables (higher stroke severity, older age, active cancer, prestroke disability), laboratory values (higher glucose, higher CRP, higher creatinine), imaging biomarkers (more white matter hyperintensities), and longer onset-to-admission time. (3) The combination of several clinical, laboratory, neuroimaging, and workflow variables at baseline showed good discrimination for prediction of FRT. (4) Our model will help to inform shared decision making, but its usability to withhold treatment is uncertain and depends on healthcare resources. (5) Potential differences for prediction of FRT between patients with and without detectable vessel occlusion were identified.

Despite the success of IVT and MT in improving stroke outcome, rates of FRT remain considerable for both treatments.[26] Given more liberal indications in the real-world as compared with the patients included in the original randomized controlled studies,[35,36] for both treatments, there is a gradual shift from selecting patients to deselecting patients. However, this development is likely to cause an increase in FRT. Although the costs of IVT and EVT are much lower as compared with established treatments, for example, for palliative cancer, the treatments pose enormous logistic, economic, and ethical challenges in acute stroke treatment.

With this analysis, we have developed and internally validated a multivariable prediction algorithm to discriminate between patients who are likely or unlikely to have a very poor clinical outcome despite best available treatment with IVT and/or MT.

The lower rate of FRT in IVT patients might be related to the fact that patients meeting IVT criteria are early presenters, without extensive hemorrhagic changes. Some features included in our model have been partially described before,[14,37-40] but several are novel and their relative importance can now be estimated (Figure 2). This allows clinicians to rapidly clarify the most relevant features in the emergency setting. All variables considered in our model are available on admission or can be obtained within a few minutes until the decision to perform IVT/MT has to be taken.

The intended use of our model is ischemic stroke patients before treatment has been started, but the decision for treatment with IVT and/or MT has been made. The derivation cohort is from a high-volume academic stroke center with low restrictions to perform those treatments. For clear-cut indications for MT, such as patients presenting early with severe stroke, the updated MR CLEAN predict tool[23] might be more useful, but it cannot be applied to borderline indications or to inform decision on IVT-only patients not fulfilling endovascular trial criteria.

Our model has good discrimination and reasonable calibration, both in the derivation as well as the validation cohort. The model output can be used to inform and discuss with patients and/or next of kin the high risk of poor outcome and help to set realistic expectations.[41] However, in high-resource healthcare systems, the algorithm will probably not be used because no patients will be excluded from acute recanalization treatments despite high chances for FRT. In addition, false positives (classifying patients as FRT, despite the fact that they might gain functional independence) should be weighted more than false-negative classifications. In other words, the positive predictive value of any FRT algorithm has to be as great as possible to evade skipping an evidence-based treatment from appropriate patients which may benefit from IVT/MT. Hence, we caution colleagues to apply the appealing AUC reported here and by others[37,42] to individual patients regarding the clinical utility in excluding patients from reperfusion therapies.[29] However, there was no evidence of harm as shown by the decision curve analysis. Randomized controlled trials need to clarify whether the cost-effectiveness of MT is preserved in patients with high risk of poor outcome.

Nevertheless, the situation might be very different in low-income and mid-income countries where access to IVT/MT is limited by shortage of MT devices, IVT medication, staff, or other hurdles. Fittingly, our models exhibited net benefit in the lower probability thresholds. This means that they might be most useful in settings of very limited healthcare resources. Cultural perception and individual preferences might also influence which a priori cutoff will be chosen in an individual scenario.

In addition, this rather simple algorithm provides evidence that prediction of FRT is possible when different information sources (clinical, laboratory, imaging, time metrics) are combined. Further and more sophisticated information sources are promising additions for refinement of this algorithm. Those include ischemic core volume,[43] ischemic core location,[44] penumbra volume, covert brain infarctions, brain atrophy,[45] masseter muscle,[46] and oxygen saturation among others have not been accounted for. Reliable identification of FRT ahead of treatment is of utmost importance,[47] and this model and proposed refinements can serve as a starting point to reach the intended use. However, a recent study found that a large part of the variance in outcome after MT is explained by variables that are only known after the treatment decision has been made challenging the possibility to predict FRT in the emergency setting before knowing the outcome of the intervention—at least for MT patients.[48] Regarding the model performance, previous studies have shown that for most tabulated data sets, the differences in performance between different analytical ML methods and conventional logistic regression are minimal or nonexistent.[21,29,37]

Strengths of this analysis include its large sample size with good quality data of predictors easily obtained in an emergency setting. This study has the limitations of a single center, retrospective registry potentially limiting its generalizability to other settings. Most importantly, no medical comparison group was available. Hence, inference on a potential residual clinical benefit even in a patient with high probability for FRT is not possible. In addition, 12% (322 patients) of the cohort had missing 3-month outcome and those had a slightly worse prognostic profile. Moreover, several advanced imaging parameters such as ischemic core volume, ischemic core location,[44] penumbra volume and mismatch profile,[49] covert brain infarctions, brain atrophy,[45] masseter muscle volume, and oxygen saturation among others have not been accounted for. Another limitation is the choice/availability of independent variables, for example, information on dementia/prestroke cognitive impairment was unavailable. Information on medical history was obtained prospectively in the registry and includes variables obtained during hospital stay, so it is uncertain how many of the predictors incorporated could be obtained within a short time in an emergency setting. Hence, our results need to be replicated by other groups and verified by upcoming randomized controlled trials on this issue in several subgroups with high risk for FRT, such as low ASPECTS (NCT03805308, NCT03811769).

In conclusion, FRT occurs in 1 in 4 patients and more often in patients receiving MT alone (34%) as compared with IVT alone (11%). We identified clinical variables (higher stroke severity, older age, active cancer, prestroke disability), laboratory values (higher glucose, higher CRP, higher creatinine), imaging biomarkers (white matter hyperintensities), and time from onset-to-admission as the most relevant predictors of FRT. The prediction algorithm will help to inform shared decision making and to set realistic expectations. Although the clinical benefit and usability of this algorithm for withholding treatments in settings with high healthcare resources is to be established in future studies, the development of a reliable algorithm for prediction of FRT seems to be within reach and should incorporate more advanced admission imaging features.