Validation of risk scoring models for predicting stroke-associated pneumonia in patients with ischaemic stroke.

OBJECTIVE: Various risk scoring models have been developed to predict stroke-associated pneumonia (SAP). We aim to determine whether these risk models could effectively predict SAP in Chinese patients with ischaemic stroke (IS).
METHODS: Consecutive patients with IS in West China hospital between January 2011 and September 2013 were included to assess the predictive performance of risk scoring models, including Chumbler's score, A2DS2 and AISAPS. The area under the receiver operating characteristic curve (AUROC) was used to evaluate the performance of each risk model in predicting pneumonia.
RESULTS: A total of 1569 consecutive patients with IS within 30 days of onset in West China hospital were included. The incidence of pneumonia is 15.3%. The AUROC of Chumbler's score, A2DS2 and AISAPS was 0.659, 0.728 and 0.758, respectively, and AISAPS had the highest AUROC.
CONCLUSIONS: A2DS2 and AISAPS had acceptable discriminatory abilities to predict SAP in Chinese patients with IS within 30 days of onset.

Introduction

Pneumonia is a common complication in patients with stroke.1 It occurs in about 9% of patients with stroke, and 28% of patients with stroke in the intensive care unit.2 A variety of factors, such as older age, dysphagia and higher stroke severity score, which are assessed by the National Institutes of Health Stroke Scale (NIHSS), are associated with stroke-associated pneumonia (SAP).3 SAP potentially increases the risk of a poor and fatal outcome,4 and increases the economic burden on patients with stroke.5 Although a preventive antibiotic cannot improve the prognosis of patients with stroke, it is proved to be effective to decrease the incidence of SAP.6 Oral care is also proved to be effective to reduce the incidence of SAP.3 Recognising patients with stroke at high risk of pneumonia and providing preventive intervention can be helpful to reduce the incidence of SAP.

Several risk scoring models were reported to improve the recognition of the high risk of pneumonia in patients with stroke, including Kwon's score,7 the PANTHER-IS score,8 Chumbler's score,9 A2DS210 and AISAPS.11 There are many advantages to the wide use of these risk models, such as assisting clinicians to identify patients with stroke with high risk of pneumonia and facilitating the selection of patients in clinic trials. However, these models have not been applied extensively to patients in different cohorts from various races and areas. They need to be externally validated before they can be used in clinical practice.

We conducted this study to determine whether these risk models could effectively predict SAP during hospitalisation in Chinese patients with IS. Since the risk scores of ischaemic stroke-associated pneumonia (like A2DS2 and AISAPS) are different from that of haemorrhagic stroke (like intracerebral haemorrhage-associated pneumonia score (ICH-APSs)),12 we only focused on patients with ischaemic stroke to control potential confounders.

Methods

Study population

We identified all consecutive patients with IS in West China hospital between January 2011 and September 2013. Inclusion criteria: aged≥18; patients with IS who were admitted to hospital within 30 days of onset; ischaemic stroke was diagnosed based on the WHO criteria; IS was confirmed by brain CT or MRI. Exclusion criteria: patients with transient ischaemic attack (TIA); pneumonia happened before admission. Severity of stroke was assessed by NIHSS.13 Consciousness was evaluated by Glasgow Coma Scale.14 The subtype of IS was classified based on the Oxfordshire Community Stroke Project criteria: partial anterior circulation infarct, total anterior circulation infarct, lacunar infarction and posterior circulation infarct.15 Dysphagia was evaluated by a neurological clinician with the water drinking test. Prestroke dependence was defined as modified Rankin Scale ≥3.

Outcome measures

The primary outcome was the presence of SAP in hospitalised patients with IS. SAP was diagnosed by the treating physician according to the modified criteria of US Centers for Disease Control and Prevention criteria for hospital-acquired pneumonia. Pneumonia was diagnosed when at least one of the first and one of the latter criteria were fulfilled: (A) abnormal respiratory examination, pulmonary infiltrates in chest X-rays; (B) productive cough with purulent sputum, microbiological cultures from the lower respiratory tract or blood cultures, leucocytosis and elevation of C reactive protein.16

Risk prediction scores of SAP

We indentified five risk scoring models of SAP in total. Kwon's score was derived from 286 patients with stroke in a single hospital, as was the PANTHER-IS score (derived from 223 patients with IS). Chumbler et al developed their risk model from 925 patients in multicentre hospitals. A2DS2 was derived from 15 335 patients in multicentre communities and hospitals, as was AISAPS (derived from 8820). We excluded Kwon's score because it involved the variable (mechanical ventilation) which we thought would be a medical intervention rather than the original character of the patient. We excluded the PANTHER-IS score because it focused on patients with IS in the middle cerebral artery territory in the neurological intensive care unit, which would be not suitable to common patients with stroke.

Statistical analysis

Given the non-normal distribution of data, median values with 25th and 75th centiles were calculated for continuous variables and comparisons between groups were performed using the Nonparametric test. Categorical variables were described as counts and percentages, while comparisons between two groups were made with the χ2 test.

The model fit was measured with the Hosmer-Lemeshow goodness-of-fit statistic. The discriminatory ability of each risk model was evaluated using the area under the receiver operating characteristic curve (c statistic). The c statistic for each pair of scores was compared with the DeLong method.

We considered a p value <0.05 (2-tailed) to be statistically significant. All analyses were performed using MedCalc.

Results

Between January 2011 and September 2013, 1814 patients with IS were consecutively referred to the hospital. Of them, 140 patients were with TIA, 2 patients younger than 18 years, 104 patients with stroke onset more than 1 month or with pneumonia prior to admission. A total of 1569 patients were finally included in the present study. The median age was 65 years (IQR, 25–75), and 60% were male. Among all participants, 240 (15.3%) patients acquired pneumonia during hospitalisation (table 1).

Table 1

Baseline characteristics of patients with and without stroke-associated pneumonia

	Pneumonia group	Non-pneumonia group	p Value
Age, median (IQR), year	70.0 (61.0, 78.0)	64.0 (53.0, 73.0)	<0.001*
Sex			0.205
Male, n, %	134 (55.8)	800 (60.2)
Female, n, %	106 (44.2)	529 (39.8)
Hypertension, n, %	141 (58.8)	764 (57.5)	0.715
Diabetes mellitus, n, %	71 (29.6)	371 (27.9)	0.597
Hyperlipaemia, n, %	16 (38.1)	142 (10.7)	0.057
History of stroke or TIA, n, %	42 (17.5)	169 (12.7)	0.046*
Valvular heart disease, n, %	2 (0.8)	7 (0.5)	0.563
Coronary heart disease, n,%	24 (10.0)	58 (4.4)	<0.001*
Atrial fibrillation, n, %	72 (30.0)	109 (8.2)	<0.001*
Congestive heart failure, n, %	3 (1.3)	1 (<0.1)	<0.001*
COPD, n, %	15 (6.3)	16 (1.2)	<0.001*
History of pneumonia, n,%	17 (7.1)	50 (3.8)	0.019*
Current smoking, n, %	33 (13.8)	148 (11.1)	0.243
Excess alcohol consumption, n, %	25 (10.4)	156 (11.7)	0.555
Found down at onset, n, %	42 (17.5)	80 (6.0)	<0.001*
Dyaphasia, n, %	130 (9.8)	272 (20.5)	<0.001*
NIHSS, median (IQR) score	9.5 (5, 14.0)	3.0 (1.0, 7.0)	<0.001*
GCS, median (IQR) score	14.0 (11.0, 15.0)	15.0 (14.0,15.0)	<0.001*
OCSP			<0.001*
Lacunar infarction, n, %	7 (2.9)	66 (4.9)
Total anterior circulation infarct, n, %	39 (16.3)	61 (4.6)
Partial anterior circulation infarct, n, %	165 (68.8)	939 (70.7)
Posterior circulation infarct, n, %	29 (12.1)	263 (19.8)
Glu, median (IQR) mmol/L	6.73 (5.20, 8.60)	5.64 (4.87, 7.13)	<0.001*
WCC, median (IQR) 1012/L	8.34 (6.55, 10.71)	6.69 (5.44, 8.22)	<0.001*

*p<0.05.

COPD, chronic obstructive pulmonary disease; GCS, Glasgow Coma Scale; NIHSS, National Institutes of Health Stroke Scale; OCSP, Oxfordshire Community Stroke Project; TIA, transient ischaemic attack; WCC, white cell count.

Table 2 demonstrated the items of each risk scoring model included in the study. The abilities of these three risk models of predicting pneumonia are summarised in table 3. The discriminatory abilities of all models were described as the area under the receiver operating characteristic curve (AUROC), which is shown in figure 1. The AUROC of all models was 0.659 (p <0.001, Chumbler's score), 0.728 (p <0.001, A2DS2) and 0.758 (p<0.001, AISAPS), respectively. The differences in AUROC (ΔAUROC) between any two risk models were significant (p<0.05). The calibrations of all models assessed by the Hosmer-Lemeshow test were not significant (p>0.05). Although the incidence of SAP was underpredicted by all of these risk models, the predicted incidence of AISAPS was closer to the observed incidence (table 4).

Table 2

The content of each risk model

Number	Chumbler's score (5 items)	A2DS2 (5 items)	AISASP (11 items)
1	age	age	age
2	dysphagia	dysphagia	dysphagia
3	NIHSS	NIHSS	NIHSS
4	found down at onset	atrial fibrillation	atrial fibrillation
5	history of pneumonia	male	GCS
6	–	–	congestive heart failure
7	–	–	COPD
8	–	–	current smoker
9	–	–	prestroke dependence
10	–	–	OCSP subtype
11	–	–	admission glucose

COPD, chronic obstructive pulmonary disease; GCS, Glasgow Coma Scale; OCSP, Oxfordshire Community Stroke Project; NIHSS, National Institutes of Health Stroke Scale.

Table 3

Comparison of various risk models for stroke-associated pneumonia

Risk models	OR	p Value	Hosmer-Lemeshow test	AUROC	p Value
Chumbler's score	1.8138	<0.0001	p=0.6243	0.659	<0.001
A2DS2	1.4248	<0.0001	p=0.0953	0.728	<0.001
AISAPS	1.2365	<0.0001	p=0.8927	0.758	<0.001

AUROC,area under the receiver operating characteristic curve.

Figure 1

The AUROC of each risk model. (A). The AUROC of Chumbler's score; (B). The AUROC of A2DS2; (C). The AUROC of AISAPS; (D). The comparison of AUROC of three risk models. AUROC, area under the receiver operating characteristic curve.

Table 4

Observed and predicted rate of stroke-associated pneumonia

	Incidence of SAP (%)
Observed	15.3
Predicted by
Chumbler's score	10.7
A2DS2	10.8
AISAPS	11.7

SAP,stroke-associated pneumonia.

The optimal cut-off value of risk models for SAP is displayed in table 5. The optimal cut-off value was 1 for Chumbler's score, whose sensitivity was 71.49%, specificity was 52.93%, positive predictive value (PPV) was 21.66% and negative predictive value (NPV) was 91.07%. The optimal cut-off value was 3 for A2DS2, whose sensitivity was 69.83%, specificity was 65.56%, PPV was 26.96% and NPV was 92.27%. The optimal cut-off value was 6 for AISAPS, whose sensitivity was 76.45%, specificity was 62.56%, PPV was 27.19% and NPV was 93.59%.

Table 5

Optimal cut-off value of risk models for stroke-associated pneumonia

	Y-index	Cut-off point	Sensitivity (%)	Specificity (%)	PPV	NPV (%)
Chumbler's score	0.2442	1	71.49	52.93	21.66	91.07
A2DS2	0.3540	3	69.83	65.56	26.96	92.27
AISAPS	0.3900	6	76.45	62.56	27.19	93.59

NPV, negative predictive value; PPV, positive predictive value.

Discussion

We found that the Chumbler score, A2DS2 and AISAPS models could reasonably predict pneumonia after IS in the Chinese population. Among them, the A2DS2 and AISAPS models (whose AUROC was higher than 0.7) had acceptable discriminatory abilities. The discriminatory ability of every risk model was lower in this external validation cohort than in their derivation cohort, which was consistent with the AISAPS study. The AISAPS study developed the AISAPS model to predict pneumonia after IS, as well as compared it with other risk models. They found Chumbler's score, A2DS2 and AISAPS model had acceptable discriminatory abilities in derivation and validation cohort, except Kwon's score only had acceptable discriminatory ability in derivation cohort. Chumbler's score had acceptable discriminatory ability in the validation cohort, which differed from this study. In addition, the AISAPS study did not report the difference in other risk models except AISAPS.

We found that AISAPS had the highest discriminatory ability, while Chumbler's score had the lowest discriminatory ability, which is consistent with the AISAPS study. The AISAPS model may be the most effective one for Chinese patients with IS. Further study is needed to explore the predictive performance of the AISAPS model in other races and regions.

We also analysed the independent risk factors of SAP in our cohort to investigate the reasons why these risks models had different predictive performances in this study. We found that age (OR=1.035, 95% CI 1.021 to 1.049), atrial fibrillation (OR=2.733, 95% CI 1.837 to 4.067), chronic obstructive pulmonary disease history (OR=5.006, 95% CI 2.143 to 11.693), NIHSS score (OR=1.124, 95% CI 1.092 to 1.157), dysphagia (OR=2.908, 95% CI 1.092 to 1.157) and the count of first time white cell count during hospitalisation (OR=1.169, 95% CI 1.116 to 1.224) were independent risk factors for SAP. The independent risk factors in our cohort had the biggest overlap with the items of AISASP, and smallest overlap with Chumbler's score. This may explain the differences of predictive performances between these risk models.

In diverse populations, the predictive performances of risk models could be different. So they need to be externally validated before they are used in clinic. As recommended in the guideline about clinical decision rules, the rules at level 1 must be prospectively validated in different populations and have one impact analysis demonstrating change in the clinician's behaviour with beneficial consequences, so that rules at level 1 can be used in a wide variety of settings.17

There are some limitations in this study. First, our study was a retrospective study with potential confounding factors. Second, we only included patients in a single hospital, which could cause selected bias.

For now, whether the clinical application of these risk models can assist in decreasing the incidence of SAP is still uncertain. Prospective studies with a large sample containing different races and a regional population are needed.