A Novel Computerized Clinical Decision Support System for Treating Thrombolysis in Patients with Acute Ischemic Stroke.

BACKGROUND AND
PURPOSE: Thrombolysis is underused in acute ischemic stroke, mainly due to the reluctance of physicians to treat thrombolysis patients. However, a computerized clinical decision support system can help physicians to develop individualized stroke treatments.
METHODS: A consecutive series of 958 patients, hospitalized within 12 hours of ischemic stroke onset from a representative clinical center in Korea, was used to establish a prognostic model. Multivariable logistic regression was used to develop the model for global and safety outcomes. An external validation of developed model was performed using 954 patients data obtained from 5 university hospitals or regional stroke centers.
RESULTS: Final global outcome predictors were age; previous modified Rankin scale score; initial National Institutes of Health Stroke Scale (NIHSS) score; previous stroke; diabetes; prior use of antiplatelet treatment, antihypertensive drugs, and statins; lacunae; thrombolysis; onset to treatment time; and systolic blood pressure. Final safety outcome predictors were age, initial NIHSS score, thrombolysis, onset to treatment time, systolic blood pressure, and glucose level. The discriminative ability of the prognostic model showed a C-statistic of 0.89 and 0.84 for the global and safety outcomes, respectively. Internal and external validation showed similar C-statistic results. After updating the model, calibration slopes were corrected from 0.68 to 1.0 and from 0.96 to 1.0 for the global and safety outcome models, respectively.
CONCLUSIONS: A novel computerized outcome prediction model for thrombolysis after ischemic stroke was developed using large amounts of clinical information. After external validation and updating, the model's performance was deemed clinically satisfactory.

Introduction

The rate of thrombolysis for overall ischemic stroke in the United States and the United Kingdom is less than 5%.1,2 In Korea, this rate was 8.6% among eligible patients within 3 hours of disease onset in 2010.3 One reason for this low rate is physicians' reluctance to treat patients with thrombolysis because weak evidence exists with respect to the risks and benefits of thrombolytic therapy.4 In a survey of emergency physicians, about 40% reported that they were not likely to use thrombolysis in a case of stroke, even in an ideal setting, because of the risk of symptomatic hemorrhagic transformation (sHT).5

Predicting the benefits and risks using prognostic models on an individual patient may improve decision-making in clinical practice.6 Recently, a so-called computerized clinical decision support system (CDSS) constructed using clinical variables and sophisticated models, provided more accurate information and help to physicians than conventional scoring systems.7 Moreover, the Johns Hopkins Venous Thromboembolism Prevention Collaborative showed that a multidisciplinary team approach using a CDSS could improve clinical practice performance.8

However, most outcome prediction models for thrombolysis in acute ischemic stroke have used conventional scoring systems and lack enough prediction power.9,10,11,12,13 Furthermore, as no external validation was performed, the only CDSS-type prediction model, the Stroke Thrombolytic Predictive Instrument,9 had several limitations, including the lack of representativeness because it was developed using clinical trial data and no consideration of a safety outcome such as sHT. In this context, this study aimed to develop a novel CDSS with high predictability, high degree of external validation, and excellent practicality for thrombolysis after ischemic stroke.

Methods

Model development cohort

The prognostic model was developed using a consecutive series of patients with acute ischemic stroke who were admitted to Seoul National University Bundang Hospital between January 1, 2004 and March 31, 2008. Patients hospitalized within 12 hours of stroke onset and showing relevant ischemic lesions on an initial diffusion-weighted magnetic resonance imaging were enrolled using a prospective stroke registry database.14 Of the 960 consecutive patients, 2 were excluded because of inadequate clinical information; therefore, 958 patients were enrolled in the model development cohort. All patients were included for developing the safety prediction model, whereas only 912 patients were included for the global outcome prediction model after excluding 48 patients whose outcome data were unavailable.

Outcome

A modified Rankin scale score (mRS) score of 0-2 (independence in activities of daily living) was used as a global outcome variable. The scores were obtained prospectively 3 months after stroke onset by a telephone interview as part of the quality-of-care monitoring and improvement program for previously hospitalized stroke patients in the participating institutions. Dedicated, trained stroke nurses were responsible for assessing the mRS.

The safety outcome variable was the occurrence of sHT, defined as any neurologic deterioration accompanied by hemorrhagic transformation on brain imaging as well as that considered to be caused by hemorrhagic transformation based on clinical judgment.15 Neurologic deterioration was defined operationally as worsening of ≥2 points of the National Institutes of Health Stroke Scale (NIHSS) score, ≥1 point on the motor items of the NIHSS, ≥1 point on the level of consciousness NIHSS items, or the presence of any new neurologic symptoms or signs that were not thought to be due to nonstroke causes, according to the definition used in prior studies.16,17

Internal validation with bootstrapping

An internal validation of the prognostic model was based on 1,000 bootstrap replicates. Bootstrapping was used to estimate the optimism-corrected model performance estimates.

External validation cohort

After the prognostic model was established, it was validated externally using patient data collected from April 2008 through September 2009 in 5 university hospitals or regional stroke centers participating in the Clinical Research Center for Stroke (CRCS). The CRCS continually collects uniform registry data on all stroke patients hospitalized at participating centers through a web-based database, since March 2007, as a prospective multicenter stroke register.18 From this registry database, 954 patients who were hospitalized within 12 hours of onset and who showed relevant ischemic lesions on an initial diffusion-weighted magnetic resonance imaging (MRI) were identified and included in the external validation of the safety model. For the global outcome model, however, data from only 897 patients were used as an external validation cohort after excluding 57 patients because of missing information. A post-hoc external validation was also performed using 7,448 patients from January 2011 to March 2014 from CRCS database to examine an applicability of the updated model to recent stroke patients.

Standard protocol approvals, registration, and patient consents

The institutional review boards from all participating centers approved the collection of clinical information, without the need for patient's consent, to the registry database whose purpose was for monitoring and improving quality-of-care of stroke patients, based on the anonymization of patient information, minimal risk to participants, and the retrospective nature of the study. Additional approval was obtained to use the registry database and to continue the collection of data through the review of medical records specifically for this study.

Statistical analyses

Comparisons between the development and external validation cohorts were performed using the Wilcoxon rank-sum test and chi-square test for continuous and categorical variables, respectively. The non-parametric Wilcoxon rank-sum test was used in the analysis of initial NIHSS or onset to treatment time as they were non-normally distributed. The predictive value of parameters associated with outcomes in the development cohort was analyzed using a logistic regression model. Two-sided P values of <0.05 were considered the minimum level of statistical significance.

A description of the model development process is as follows.

1) Selection of potential predictors. Predictors needed to be preselected to avoid an increase in type I errors and an overfitting of the prognostic model. In this study, potential predictors were decided using 3 steps: (1) performing a systematic review, (2) checking their availability in the registry database, and (3) having discussions with 6 stroke neurologists who participated in the CRCS to develop a consensus for potential predictors. As a result, 18 and 15 predictors were chosen for the global and safety outcome models, respectively. The 18 predictors for the global outcome model were age, gender, previous mRS, initial NIHSS score, previous stroke history, hypertension, diabetes, hyperlipidemia, atrial fibrillation, previous transient ischemic attack, prior use of antiplatelet drugs, prior use of antihypertensive drugs, prior use of statins, prior use of glucose, initial systolic blood pressure (SBP), thrombolysis, onset to treatment time (i.e., time between onset of symptoms and arrival at the hospital), and lacune. For the safety outcome model, the prior use of an anticoagulant drug variable was included in the model, whereas previous mRS, atrial fibrillation, previous transient ischemic attack, and prior use of antihypertensive drugs were excluded. In this study, lacune in the acute phase after cerebral infarction, i.e., within 48 hours from disease onset, was defined in two ways by stroke physicians, depending on whether the patient had undergone a brain MRI. The first definition included patients who underwent a brain MRI for the diagnosis of penetrating artery infarction of the basal ganglia, corona radiate, thalamus, or pons without a defined cardioembolic source, showing a single lesion with the largest diameter of ≤20 mm in an axial diffusion-weighted image slice. The second definition included patients who did not undergo a brain MRI, but had typical lacunar syndromes and no territorial or embolic infarction in brain CT scans. While intravenous thrombolysis was defined as the intravenous injection of recombinant tissue plasminogen activator, intra-arterial thrombolysis consisted of interventional approaches carried out on the relevant arteries either by applying thrombolytic agents via the intra-arterial route or mechanical thrombectomy using intra-arterial devices, without intravenous thrombolysis. Whereas combined thrombolysis implied initial intravenous thrombolysis followed by subsequent intra-arterial thrombolysis.

2) Evaluation of predictor effects. A difference of -2 log-likelihood (-2 LL) between models with and without the predictors was used to evaluate the effect of potential predictors, taking into account that the predictors with high -2 LL had a greater influence on the outcome than those with a low -2 LL.

3) Prognostic model fitting procedure. The existence of multicollinearity among predictors and assumption of linearity of an event's logit on continuous predictors were checked before developing the model. In the global outcome model, interactions between thrombolysis and other variables were included. However, for the safety outcome model, interactions between thrombolysis and other variables were found to be negligible and hence were not included. In addition, the event per predictor variable was less than 10 for the safety outcome model, and this was another reason to exclude the interactions from the safety model. To develop a final prognostic model for the global outcome, a fast backward elimination method using the Akaike information criterion (AIC) was implemented. For the safety model, however, because of the possibility of decreased predictive power from the so-called estimation bias that occurred because of small event per predictor variable, the Lasso method was used.19,20

4) Internal and external validations. Internal and external validations were performed, and bootstrapping was used for the former.

5) Statistics. Discrimination statistics such as C-statistics (equivalent to the area under the receiver operating characteristic curve) were calculated to indicate how well an entire model was matched with observed values. C-statistics >0.80 were considered acceptable values. Model calibration was assessed by the Hosmer-Lemeshow test and by the plots comparing predicted versus observed probability of outcome. Analyses were performed using R-project 2.11.1 (package "rms" version 4.11).

6) Model update. For the practical application of the model, model updating was necessary to increase its predictive power. This study used a logistic calibration method to update the calibration intercept and slope based on the external validation results.21

Results

Compared with the patients included in the model development cohort, patients in the external validation cohort were older; had hypertension, hyperlipidemia, and atrial fibrillation more frequently; had previously used antiplatelets, had a history of cardioembolic stroke, and had used statins less frequently (Table 1). Subjects in the external validation cohort had lower SBP and received thrombolytic therapy more frequently than those in the development cohort.

Table 1

Baseline characteristics of the patients included in the study

*P<0.05. Median (IQR) for continuous variables and number (percent) for discrete variables; †Onset to treatment time implies time between the onset of symptoms and arrival at the hospital.

NIHSS, NIH stroke scale; DM, diabetes mellitus; TIA, transient ischemic attack; LAA, large artery atherosclerosis; SVO, small vessel occlusion; CE, cardioembolism; SBP, systolic blood pressure; mRS, modified Rankin Scale; HT, hemorrhagic transformation.

Prognostic model for global outcome

To develop the global outcome model, the difference of -2 LL between the model with and without predictors was used to assess the effect of predictors. Among the 18 potential predictors chosen a priori, variables of age, previous mRS, initial NIHSS score, previous stroke, diabetes mellitus, history of statin use, thrombolysis, and lacune influenced a good functional outcome with high predictor effects (P<0.1) (Table 2).

Table 2

Effects of potential predictors on global and safety outcome models

χ2 statistics: difference of -2 log-likelihood between logistic regression models with and without the predictors.

Results of fast backward elimination logistic regression analysis, interaction terms with thrombolysis, and nonlinear terms for SBP and initial NIHSS score are shown in Table 3. In the model, a squared term of the initial NIHSS score, along with its linear term were added, whereas SBP was modeled with a restricted cubic spline function with 4 knots.

Table 3

Independent predictors of the two prognostic models

The intercept is not provided.

See the footnotes of Table 1 for abbreviation.

OTTT indicates onset to treatment time; RCS, restricted cubic spline functions.

Discriminative ability of the developed model turned out to be satisfactory with a C-statistic of 0.89 (95% confidence interval [CI], 0.87-0.91; Figure 1A). The Hosmer-Lemeshow test also showed a high degree of goodness of fit (P=0.52). Internal and external validation results showed high optimism-corrected C-statistics of 0.87 (95% CI, 0.83-0.90) and 0.82 (95% CI, 0.79-0.85; Figure 1A), respectively.

Figure 1

ROC curves of the global and safety outcome models in the development and external validation cohorts.

Safety prognostic model

Compared with the global outcome model, initial NIHSS score, thrombolysis, and lacune variables were found to influence sHT, among the 15 variables selected initially (Table 2). Using the Lasso method, age, initial NIHSS score, thrombolysis, onset to treatment time, SBP, and glucose were selected as predictors for the safety model (Table 3). Lacune was not selected as a variable for the full model because no sHT occurred in patients with lacune. The safety model showed a high C-statistic (0.84; 95% CI, 0.79-0.88; Figure 1B) and a satisfactory goodness of fit using the Hosmer-Lemeshow test (P=0.27).

The external validation C-statistic was still high (0.82; 95% CI, 0.77-0.86; Figure 1B) and a calibration slope was 0.96 (Figure 2B). A good model fitting was also observed after the external validation was performed using the Hosmer-Lemeshow test (P=0.20).

Figure 2

Calibration plots of the global (A) and safety (B) outcome models in the external validation cohort before and after model updating.

Model update

For the global outcome, the optimism-corrected calibration slopes after the internal and external validations were 0.90 and 0.68, respectively (Figure 2A, upper left panel). As a result, a method of updating both the calibration intercept and slope was needed to increase the predictive power of the global outcome model to apply the model to the new population. After the update, the calibration graph showed that a calibration intercept and slope approached 0 and 1, respectively (Figure 2A, upper right panel). The regression coefficients of the updated model are presented in Table 3.

The deviated intercept and slope of the safety model were also recalibrated to enhance the model's performance. This updating process resulted in a calibration graph with a calibration intercept and slope approaching 0 and 1, respectively (Figure 2B, bottom panels). The regression coefficients in the updated safety model are presented in Table 3.

Sensitivity analysis and post-hoc external validation

The thrombolytic modalities were categorized as follows: intravenous alone, intraarterial alone, and combined (intravenous and intraarterial) thrombolysis. As seen in Table 4, the number of patients within each of the thrombolysis modalities was not sufficiently large in our model development dataset. However, in the sensitivity analysis, C-statistics of the global outcome model were calculated for each of these modalities, after excluding patients treated with other modalities, from the external validation cohort, and all were >80% (Figure 3A). For the safety model, C-statistics for each thrombolytic modality were also >80% (Figure 3B). Moreover, the post-hoc external validation of the updated model including the recent patient dataset showed a high C-statistics (0.85; 95% CI, 0.84-0.86), and the calibration slope approached 1 for the global outcome (1.09; Supplementary Figure 1). We further assessed the performance of the updated model with a dataset of 5,757 patients who were admitted within 6 hours of stroke onset by carrying out another post-hoc external validation. The validation results were still satisfactory, showing a C-statistic for this subgroup of 0.84 (95% CI, 0.83-0.85) and a calibration slope of 1.04. This result indicated that, for patients who were admitted after 6 hours onset, the external reliability of the updated model was also maintained.

Table 4

Distribution of the 3-month mRS and sHT according to thrombolytic modalities in the model development dataset

Figure 3

ROC curves of the global (A) and safety (B) outcome models for each thrombolytic modality using the external validation cohort.

Discussion

Using a large amount of clinical information, a novel computerized outcome prediction model was developed to help physicians make decisions on thrombolytic treatment in patients with acute ischemic stroke. This model can be used to predict not only a good functional recovery (mRS 0-2) as a global outcome but also sHT as a major adverse event. To validate the developed model, an external validation was performed using the nationwide multicenter stroke registry database and a model update was conducted. The performance of the model was satisfactory, with C-statistics prediction values >80%. Moreover, to implement this model in real clinical practice, a web-based program has also been developed for use in ubiquitous conditions.

This novel CDSS showed C-statistics of 0.89 and 0.84 for the global and safety outcomes, respectively, which are higher than not only those of previous conventional scoring systems10,11,12,13 but also other additional CDSS for thrombolysis in ischemic stroke cases9 (Table 5). Accuracy is a key feature of any CDSS for practical use because physicians might accept the CDSS results when they believe that the accuracy is higher than that of their own judgment. Selecting predictors and a well-organized model development process are crucial factors needed to ensure the high accuracy of a prognostic model. Therefore, a comprehensive selection of predictors, external validation, and model updating should be considered in the model development process.6

Table 5

C-statistics and predictors in prognostic models of thrombolysis in acute ischemic stroke patients

See the footnote of Table 1 for abbreviations.

CDSS, Computerized Clinical Decision Support System; ND, not done; CI, confidence interval; CT, computerized tomography; MRI, magnetic resonance imaging; DWI, diffusion weighted MRI; ASPECTS, Alberta Stroke Program Early Computed Tomography Score.

There are three points to be noted for selecting potential predictors in this study. Prior use of antiplatelets, antihypertensive drugs, and statins were novel predictors for the global outcome compared with those selected for the development of previous CDSS (the Stroke Thrombolytic Predictive Instrument)9 and scoring systems,10,11 whereas onset to treatment time was a novel predictor selected for development of the safety model compared with those of previous studies.12,13 Second, previous drug use may be associated with functional outcomes in stroke patients,22,23,24 and delayed treatment has also been shown to be a predictor for sHT.25 Last, a rigorous prediction model developmental process was applied in this study, as previously described.26

This novel CDSS model had a few caveats. First, the model did not reach more than 90% accuracy on the C-statistics, which may not be enough to convince physicians to use it. Future considerations of the imaging parameters followed by a focus on specific treatment modalities and an increase in the model's accuracy are therefore necessary. Second, although the model was validated and updated using a nationwide, representative registry database, it was developed from a single-center database, which increased the quality and consecutiveness of data but heavily limited its generalizability. In this context, model validation and updating are recommended before its application in a specific center setting. Third, all types of thrombolytic modalities were combined into the thrombolysis variable, and this may lead to concerns that the prediction power of this model depends on a certain thrombolytic modality. To overcome this issue, a sensitivity analysis performed for each subset of thrombolytic modalities (intravenous only, intraarterial only, or combined approaches) using the external validation dataset, resulted in good prediction power with C-statistics values of >80% for all types of thrombolytic modalities. Finally, considering that physicians usually consider thrombolysis in patients admitted within 6 hours of stroke onset, the usage of our model could be of limited value, as it was developed and externally validated for patients who were admitted within 12 hours of onset. However, our sensitivity analysis performed by post-hoc external validation using data from patients admitted within 6 hours of onset, revealed a reliable performance.

Conclusion

In this study, we developed a novel computerized outcome prediction model for thrombolysis after ischemic stroke. Through an external validation and updating, the model's performance was found to be clinically satisfactory. With the emergence of a large amount of information, including computerized patient data, precision medicine is increasingly being suggested as a solution to various health problems worldwide.27,28 Physicians can get more information from patients faster than before, but tools to interpret such useful information are limited. In the near future, more sophisticated models combined with computerized techniques, such as novel CDSS proposed for thrombolysis in acute ischemic stroke cases, will change daily clinical practice. With continuous monitoring and updating, the proposed model therefore should be feasible and helpful to physicians who make difficult decisions in an emergency setting.