Are you suffering from a large arterial occlusion? Please raise your arm!

Background and purpose: Triage tools to identify candidates for thrombectomy are of utmost importance in acute stroke. No prognostic tool has yet gained any widespread use. We compared the predictive value of various models based on National Institutes of Health Stroke Scale (NIHSS) subitems, ranging from simple to more complex models, for predicting large artery occlusion (LAO) in anterior circulation stroke.
Methods: Patients registered in the SITS international Stroke Register with available NIHSS and radiological arterial occlusion data were analysed. We compared 2042 patients harbouring an LAO with 2881 patients having no/distal occlusions. Using binary logistic regression, we developed models ranging from simple 1 NIHSS-subitem to full NIHSS-subitems models. Sensitivities and specificities of the models for predicting LAO were examined.
Results: The model with highest predictive value included all NIHSS subitems for predicting LAO (area under the curve (AUC) 0.77), yielding a sensitivity and specificity of 69% and 76%, respectively. The second most predictive model (AUC 0.76) included 4-NIHSS-subitems (level of consciousness commands, gaze, facial and arm motor function) yielding a sensitivity and specificity of 67% and 75%, respectively. The simplest model included only deficits in arm motor-function (AUC 0.72) for predicting LAO, yielding a sensitivity and specificity of 67% and 72%, respectively. Conclusions: Although increasingly more complex models yield a higher discriminative performance for predicting LAO, differences between models are not large. Assessing grade of arm dysfunction along with an established stroke-diagnosis model may serve as a surrogate measure of arterial occlusion-status, thereby assisting in triage decisions.

Introduction

Stroke is one of the leading causes of death and disability worldwide, afflicting approximately 15 million people each year, of which 10 million patients either die or suffer from permanent disability.1 For many years, intravenous thrombolysis with recombinant tissue plasminogen activator (r-tPA) alteplase was the only available pharmacological treatment for acute ischaemic stroke, improving long-term functional outcome after stroke.2 Recently, several randomised controlled trials have proven the benefit of endovascular thrombectomy using stent-retrievers in large artery occlusion (LAO) for anterior circulation stroke within 6 hours of symptom onset and in selected cases up to 24 hours after symptom onset.3–5 These results have revolutionised the management of acute ischaemic stroke. However, the beneficial effects of endovascular thrombectomy have been shown to be time-dependent, decreasing with increased time from symptom onset to reperfusion.6 7 The need of prehospital tools rapidly identifying patients likely harbouring an LAO amenable to endovascular treatment have sparked the publication of several scales.8–15 These scales aim at directing patients with a high probability of LAO to comprehensive stroke centres with endovascular capabilities, potentially reducing the time to reperfusion.8–15 None of these scales have so far gained widespread use, and only one of the scales has been validated prospectively in a prehospital setting.13 All scales have incorporated all or subitems of the National Institutes of Health Stroke Scale (NIHSS), as stroke severity has been shown to be an important predictor of LAO.16–18 Using the full NIHSS in the prehospital setting is time-consuming, and the need for simple prediction rules is evident.

The aim of the current study is to compare the performance of various models for predicting LAO based on NIHSS subitems, ranging from simple to more complex models.

Methods and materials

The study was based on patients registered in the Safe Implementation of Treatments in Stroke (SITS)-International Stroke Treatment Register (SITS-ISTR) between 1 December 2012 and 23 October 2015. Only patients with available CT angiography/MR angiography (CTA/MRA) and baseline NIHSS data treated with intravenous alteplase and/or endovascular thrombectomy were considered. The time period was chosen based on the time of implementation in the database of additional variables for registering detailed information on arterial occlusion site using baseline CTA/MRA. Data for the current study were contributed by hospitals which reported specified arterial occlusion sites in at least 20 patients in whom any occlusion was reported as present. This selection of experienced centres was done in order to ensure a high quality registration of arterial occlusion data.

The SITS-ISTR is a multinational, prospective, observational monitoring register documenting data for centres using various treatments in acute ischaemic stroke. The aims of the register, collection of data and structure of the database have been described previously.19 Baseline and demographic characteristics, stroke severity per the NIHSS, arterial occlusion status assessed using CTA/MRA, risk factors, time logistics, medication history, imaging data and 3-month outcome data were registered.

Ethical approval and data monitoring

Ethical approval or patient consent for participation in SITS-ISTR was obtained in countries that required this; other countries approved the register for conduct as an anonymised audit. The SITS-MOST II study was approved by the Stockholm Regional Ethics Committee. The SITS International Coordination Office performed regular online monitoring of the SITS-ISTR data online and checked individual patient data on a regular basis to identify errors or inconsistencies.

Outcome measurements

Arterial occlusion was defined as a complete occlusion or less than 50% filling of the affected vascular territory on CTA/MRA. LAO was defined as occlusion of any segment of the internal carotid artery, carotid terminus, M1 branch of the middle cerebral artery or basilar artery. In cases where more than one arterial segment was reported as being occluded, the most proximal vessel was registered for the analysis. The control population was defined as patients without visible occlusions or with more distal occlusions (M2, M3, distal to M3; A1, A2, distal to A2, P1, P2 and P3).

All assessments of imaging studies were done according to clinical routine at centres participating in the SITS-ISTR, and no central assessment of imaging data was routinely performed.

Statistical analysis

For baseline and clinical data, univariate analysis was performed comparing patients with and without LAO. Proportions were calculated for categorical variables, dividing the number of events with the total number excluding missing/unknown cases. For continuous variables, medians were calculated. Statistically significant differences between proportions were assessed using the χ² method, and the Mann–Whitney U-test was used for medians.

The most highly predictive models, exploratively identified using highest area under the curve (AUC) by receiver operating characteristic (ROC) analysis and varying from single-variable models to more complex models, were developed using binary logistic regression analysis.

We assessed for differences between the models using both comparisons of the area under the ROC curves (likelihood ratio tests and Wald tests) as well as IDI (Integrated discrimination improvement) and NRI (net reclassification improvement) statistics to judge the value of added items. Sensitivities and specificities for predicting LAO were determined at three distinct cut-offs (definition see below). The various models were solely based on NIHSS subitems registered in a continuous fashion separating different degrees of dysfunction.

Using binary logistic regression, we derived the most predictive models for predicting LAO based on subitems of the NIHSS. We investigated 1-item, 2-items, 3-items, 4-items models as well as a model including all NIHSS subitems. For each model, whether a 1-item, 2-items, 3-items, 4-items or full NIHSS subitem model, we investigated which combination of items yielded the highest AUC and subsequently chose these models for further comparison. Within each type of model (1-item, 2-items, 3-items or 4-items model), the AUCs of the different subitem combinations were compared using likelihood ratio tests. In this way, we finally chose one specific item-combination for each 1-item, 2-items, 3-items and 4-items model based on maximising the AUC for LAO prediction. For each model, we assessed the sensitivity and specificity for LAO at three different cut points: at an optimal cut-point maximising the Youden index, at a cut-point maximising the specificity (as close to 80% as possible) and at a cut-point maximising the sensitivity (as close to 80% as possible). In addition to investigating the above-mentioned models, we decided to specifically include a hemiparesis model including the arm and leg NIHSS-subitems (NIHSS item 5 and NIHSS item 6) as well as a model solely based on the total NIHSS score, in contrast to the full-items model which included each NIHSS subitems as a separate predictor. In order to better appreciate the performance of our models, we also performed a comparison with selected pre-existing LAO prediction instruments (PASS, RACE, G-FAST and CPSSS). These scales were computed on our study-base in accordance with the descriptions in the respective original articles.10 12 13 20

Results

Between 1 December 2012 and 23 October 2015, 5112 patients with available CTA/MRA data from large-volume centres were registered in the database. Baseline NIHSS (and complete NIHSS subitem data) was available in 96% of the study group (4923/5112). Of these, 2042 patients (41%) had a LAO, while 2881 patients (59%) had either no occlusion or more distal occlusions. Table 1 summarises the baseline characteristics, comparing patients with LAO with patients without LAO, with complete NIHSS baseline data.

Table 1

Baseline clinical characteristics of patients with and without LAO

Baseline parameter	LAO (n=2042)		Non-LAO (n=2881)		P values
	No/Total	Median or proportion (%)	No./Total	Median or proportion (%)
Age	2035	72	2881	71	0.189
Gender, female	965/2042	47.3%	1261/2881	43.8%	0.015
NIHSS at baseline	2042	16	2881	8	<0.01
Weight, kg	2042	75	2881	78	<0.01
Glucose, mmol/L	1892	6.7	2800	6.6	0.824
Total cholesterol	1259	4.6	2119	4.7	0.165
Systolic BP	1963	150	2848	156	<0.01
Diastolic BP	1963	80	2848	84	<0.01
Prestroke mRS score>2	81/1978	4.1%	135/2800	4.8%	0.234
Hypertension	1345/2033	66.2%	1958/2876	68.1%	0.157
Diabetes	383/2033	18.8%	648/2876	22.5%	<0.01
Hyperlipidaemia	635/2008	31.6%	912/2857	31.9%	0.826
Current smoker	319/1948	16.4%	458/2789	16.4%	0.967
Previous stroke	204/2032	10.0%	396/2870	13.8%	<0.01
Previous TIA	106/2032	5.2%	177/2870	6.2%	0.160
Atrial fibrillation	495/2026	24.4%	486/2867	17.0%	<0.01
Heart failure	176/2021	8.7%	201/2864	7.0%	0.029
Aspirin	566/2026	27.9%	931/2868	32.5%	<0.01
Oral anticoagulants	158/2031	7.8%	105/2871	3.7%	<0.01
Clopidogrel	119/2029	5.9%	165/2870	5.7%	0.864
Oral antidiabetics	218/1999	10.9%	404/2852	14.2%	<0.01
Statins	551/2026	27.2%	803/2871	28.0%	0.551

LAO, any internal carotid artery occlusion and/or M1 occlusion and/or basilar occlusion; non-LAO, M2 M3, ACA, PCA or none.

BP, blood pressure; LAO, large artery occlusion; mRS, modified Rankin Score; NIHSS, National Institutes of Health Stroke Scale.

Patients with and without LAO differed somewhat regarding several baseline variables, however not to any larger extent although being statistically significant, see table 1. The most notable difference between patients with and without LAO was in the total baseline NIHSS score at 16 vs 8, respectively, as well as higher systolic blood pressure in patients without LAO and higher prevalence of atrial fibrillation in patients with LAO.

Models for predicting LAO

The most predictive models with increasing number of included subitems are given below. The sensitivity and specificity of the models at the optimum Youden cut-off, at the maximum specificity cut-off and at the maximum sensitivity cut-off are given in table 2. Figure 1 depicts the AUC for LAO comparing the most predictive 1-item, 4-items and full-items models. The AUCs for the models given below were all statistically significantly different from each other at the 5% level (assessed using likelihood ratio tests and Wald tests). An online supplementary table e-1, displays the proportion of patients being detected using the various models and cut-offs as well as positive and negative predictive values (PPVs and NPVs). Online supplementary table e-2 displays the IDI and NRI statistics, assessing the additional value of adding extra subitems to increasingly complex models.

Table 2

Sensitivity and specificity for predicting LAO for various models at optimum cut-off, high specificity cut-off and high sensitivity cut-off

Model	AUC	P values	Optimum Youden cut-off		High specificity cut-off		High sensitivity cut-off
			Sens	Spec	Sens	Spec	Sens	Spec
NIH item 5	0.72	–	67 (NIHSS 5≥3)	72 (NIHSS 5≥3)	53 (NIHSS 5=4)	81 (NIHSS 5=4)	77 (NIHSS 5≥2)	57 (NIHSS 5≥2)
NIHSS item 5+NIHSS item 2	0.75	<0.05	68	73	64	76	82	53
NIHSS item 5+NIHSS item 2+NIHSS item 4	0.76	<0.05	70	72	61	80	80	59
NIHSS item 5+NIHSS item 2+ NIHSS item 4+NIHSS item 1C	0.76	<0.05	67	75	61	80	79	59
Full-item NIHSS	0.78	<0.05	69	76	62	80	81	59
Total NIHSS score	0.76	<0.05	69	73	60	80	80	57
NIHSS item 5+NIHSS item 6	0.73	<0.05	65	74	53	82	79	52

The p values for the AUCs describe likelihood ratios tests comparing each model with the simple arm paresis model (NIH item 5). For the 1-item model including subitem 5, arm function, we have in italics given the respective subitem cut-off for the three sensitivity/specificity scenarios. NIHSS item 2=gaze; NIHSS item 1C=level of consciousness commands; NIHSS item 4=facial motor function; NIHSS item 5=arm motor function; NIHSS item 6=leg motor function.

AUC, area under the curve; LAO, large artery occlusion; NIHSS, National Institutes of Health Stroke Scale.

Figure 1

AUC for predicting LAO, comparing the most predictive NIHSS 1-item, 4-items and full-items models. The 1-item model included NIHSS subitem 5 (arm function), the 4-items model included subitems 1C, 2, 4 and 5 (level of consciousness commands, gaze, facial and arm motor function. The full-items model included all NIHSS subitems. AUC, area under the curve; LAO, large artery occlusion; NIHSS, National Institutes of Health Stroke Scale.

1-item model

The 1-item model associated with the highest AUC (0.72) including only NIHSS subitem 5, arm motor function. The optimum Youden cut-off (NIHSS item 5≥3) resulted in a sensitivity and specificity of 67% and 72%, respectively. At the high-specificity cut-off (NIHSS item 5=4) a sensitivity and specificity of 53% and 81%, respectively, was seen. The high sensitivity cut-off (NIHSS item 5≥2) resulted in a sensitivity and specificity of 77% and 57%, respectively.

A 1-item model including only NIHSS subitem 6, leg motor function, yielded an AUC of 0.71.

2-items model

The 2-items model associated with the highest AUC (0.75) included NIHSS subitems 5 and 2, arm motor function and gaze, respectively. The optimum Youden cut-off resulted in a sensitivity and specificity of 68% and 73%, respectively, at the high-specificity cut-off a sensitivity and specificity of 64% and 76% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 82% and 53%, respectively.

3-items model

The 3-items model associated with the highest AUC (0.755) included NIHSS subitems 5, 2 and 4, arm motor function, gaze and facial motor function, respectively. At the optimum Youden cut-off a sensitivity and specificity of 70% and 72% was seen, the high-specificity cut-off resulted in a sensitivity and specificity of 61% and 80%, and at the high sensitivity cut-off a sensitivity and specificity of 80% and 59% was seen.

4-items model

The 4-items model associated with the highest AUC (0.76) included NIHSS subitems 5, 2, 4 and 1C, arm motor function, gaze, facial motor function and level of consciousness commands, respectively. The optimum Youden cut-off resulted in a sensitivity and specificity of 67% and 75%, respectively, at the high-specificity cut-off a sensitivity and specificity of 61% and 80% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 79% and 59%.

Full-items model

The full-items model resulted in an AUC of 0.78. The optimum Youden cut-off yielded a sensitivity and specificity of 69% and 76%, at the high-specificity cut-off a sensitivity and specificity of 62% and 80% was seen, and the high sensitivity cut-off resulted in a sensitivity and specificity of 81% and 59%.

Hemiparesis model

The hemiparesis model, including arm and leg function (NIHSS subitems 5 and 6), was associated with a AUC of 0.73. At the optimum Youden cut-off a sensitivity and specificity of 65% and 74% was seen, the high-specificity cut-off resulted in a sensitivity and specificity of 53% and 82%, and at the high sensitivity cut-off a sensitivity and specificity of 79% and 52% was seen.

Total NIHSS score model

The model including the total NIHSS score as a predictive model was associated with a AUC of 0.76. The optimum Youden cut-off (NIHSS ≥13) score resulted in a sensitivity and specificity of 69% and 73%, at the high-specificity cut-off (NIHSS ≥14) a sensitivity and specificity of 60% and 80% was seen, and the high sensitivity cut-off (NIHSS ≥9) yielded a sensitivity and specificity of 80% and 57%.

Sensitivity analysis including M2/A1

When including M2, A1 and P1 occlusions in the definition of LAO, AUC of the 1-item model including arm paresis was 0.71, the AUC of the 4-items model including NIHSS subitems 5, 2, 4 and 1C was 0.76 and the AUC of the full-items NIHSS 0.77.

Sensitivity analysis excluding basilar artery occlusions

When excluding basilar artery occlusions from the definition of LAO, AUC of the 1-item model including arm paresis was 0.72, the AUC of the 4-items model including NIHSS subitems 5, 2, 4 and 1C was 0.76 and the AUC of the full-items NIHSS 0.77.

Pre-existing LAO models

The results of the comparison of the arm paresis model with the four selected pre-existing LAO prediction scales can be seen in online supplementary table e-3.

Discussion

In the era of endovascular thrombectomy in acute ischaemic stroke, reducing time to reperfusion in patients harbouring LAO is of the utmost importance.7 Although baseline stroke severity measured by the NIHSS score predicts the presence of arterial occlusions,8 16–18 21 22 the full NIHSS is time-consuming and probably too complicated in the prehospital setting. Several attempts at developing simplified scores for use in the prehospital setting have been done; however, none of these has to our knowledge gained any widespread use.8–10 12–15

In this study, we show that although increasing numbers of included NIHSS subitems in LAO prediction models leads to better prediction, the differences do not seem to be clinically relevant. A simple predictive model only including the arm motor function item of the NIHSS shows both sensitivities, specificities, PPVs and NPVs (67%, 72%, 63% and 75%, respectively) quite close to the values for a model including the full range of NIHSS subitems (69%, 76%, 67% and 77%, respectively). To our knowledge, no study has investigated such a simple model previously.

Different settings and situations may need different trade-offs between sensitivity and specificity. In a recent article from the SITS collaborators, two different clinical scenarios were described.23 In the first scenario, a patient assessed by paramedics at an early time-point or with a short transfer-time to a comprehensive stroke centre with endovascular capabilities was considered. In such a patient, a predictive model with a high sensitivity would be desirable. Using an unadjusted cut-off NIH item 5 (arm function)≥2 gave a sensitivity, specificity, PPV and NPV of 77%, 57%, 57% and 78%, quite similar to the adjusted full-items NIH model, which gave a sensitivity, specificity, PPV and NPV of 81%, 59%, 58% and 81%, respectively. This corresponds to the high-sensitivity cut-off of the total NIHSS score ≥9. In the second scenario, a patient evaluated by paramedics close to the end of the IVT time-window or with a long transfer-time to a comprehensive stroke centre was considered. A high specificity for LAO would be desirable in this situation, in order to reduce the number of futile transfers. Using a cut-off NIH item 5 (arm function)=4 gave a specificity, sensitivity, PPV and NPV of 81%, 53%, 66% and 71%, not very far from the adjusted full-items NIH model which gave a specificity, sensitivity, PPV and NPV of 80%, 62%, 69% and 75%. This corresponds to the high-specificity cut-off of the total NIHSS score ≥14.

In summary, although the more complex models perform statistically significantly better than our simple arm dysfunction model (see table 2 and online supplementary table e-2), differences are not large and the simple model performs reasonably well in both clinical scenarios. We performed a comparison of the simple arm paresis model with four pre-existing LAO prediction instruments (CPSSS, PASS, G-FAST and RACE scores), see online supplementary table e-3. The AUCs of three of the pre-existing models differed significantly from the arm-paresis model, CPSSS and RACE performing better and PASS worse. G-FAST did not differ significantly. However, the AUCs of all five models were similar, suggesting a similar predictive performance.

Prediction of LAO without vessel imaging will always lead to either missing patients harbouring a treatable arterial occlusion or unnecessary transfer of patients without treatable occlusions to comprehensive stroke centres. However, simple prediction rules may improve patient selection and potentially lead to improved onset-to-reperfusion times. Using the high-specificity cut-off (NIH item 5=4) as a triage tool to endovascular centres would identify 33% of our study population as possible LAO candidates (see online supplementary table e-1), the PPV amounting to 66%, ie, nearly 7/10 successful transfers. Using the high-sensitivity cut-off (NIH item 5≥2) as a triage tool would identify roughly 60% of our study population (see online supplementary table e-1), the PPV amounting to 57%, ie, nearly 6/10 successful transfers (see online supplementary table e-1). The choice of cut-off depends on the clinical situation and local circumstances as outlined above. We suggest that a thorough screening for severity of arm motor function can be used as an LAO screening step after confirmation of likely stroke diagnosis has been performed using a good stroke recognition score at the primary screening step like the FAST score.24 Previous studies have shown an excellent inter-rater reliability of the motor subitems of the NIHSS score as well as the possibility of its use by non-physicians undergoing a brief training period.25 26 We therefore believe that our simple arm paresis model would suit the prehospital setting.

This study bears with it certain limitations inherent to the observational design. Our definition of LAO did not include M2 occlusions which may be amenable to treatment; however, a sensitivity analysis including M2 occlusions in the LAO definition showed very similar results to those presented in the main results. In addition, a subanalysis excluding basilar occlusions from the definition of LAO also presented very similar results. Although we extend our results to a prehospital setting, assessments of the NIHSS registered in the database were performed in a hospital-setting. Also, our study is based on patients treated with intravenous-tPA and/or endovascular thrombectomy and may not be valid for unselected patients with a prehospital suspicion of stroke. However, previous studies on unselected ischaemic stroke cohorts have shown LAO proportions similar to our cohort.27 Our selection of patients with documented ischaemic stroke is a potential source of bias, as both haemorrhagic stroke and stroke mimics will be a part of the assessed population in the prehospital phase. Despite these limitations, we believe that our findings are of interest.

Conclusion

Although increasingly more complex clinical models yield a higher discriminative performance for predicting LAO, the difference compared with extremely simple models is not very large. Balancing predictive performance against ease of use and practicality, simply assessing the grade of arm dysfunction may serve as a surrogate measure of LAO status, and in conjunction with a validated stroke recognition instrument, may assist in decision making in guiding high-risk patients to comprehensive stroke centres with endovascular treatment capabilities. These findings need to be confirmed in a future prospective study on prehospital triage.