Development and internal validation of a prediction rule for post-stroke infection and post-stroke pneumonia in acute stroke patients.

INTRODUCTION: Patients with acute stroke are at high risk for infection. These infections are associated with unfavourable outcome after stroke. A prediction rule can identify the patients at the highest risk for strategies to prevent infection. We aim to develop a prediction rule for post-stroke pneumonia and other infections in patients with acute stroke. PATIENTS AND METHODS: We used data from the Preventive Antibiotics in Stroke Study, a multicentre randomised trial comparing preventive ceftriaxone vs. standard stroke care in patients with acute stroke. Possible predictors for post-stroke pneumonia or infection were selected from the literature. Backward elimination logistic regression analysis was used to construct prediction rules for pneumonia or infection. Internal validation was performed and a risk chart was constructed. We adjusted for preventive antibiotic use.
RESULTS: Pneumonia was diagnosed in 159 of the 2538 included patients, and infection in 348. Pneumonia was predicted by higher age, male sex, pre-stroke disability, medical history of chronic obstructive pulmonary disease, more severe stroke, dysphagia and intracerebral haemorrhage (rather than ischaemic stroke). Infections were predicted by higher age, male sex, history of diabetes, chronic obstructive pulmonary disease, more severe stroke, dysphagia, use of bladder catheter, preventive antibiotic use and intracerebral haemorrhage. With the prediction rule developed, risks for pneumonia ranged from 0.4% to 56.2% and from 1.8% to 88.0% for infection. Discrimination of the score was good (C-statistic, 0.84; 95% CI: 0.81-0.87 and 0.82; 95% CI: 0.79-0.84 for pneumonia and infection).
CONCLUSIONS: The Preventive Antibiotics in Stroke Study pneumonia and infection rule identify patients at the highest risk for post-stroke pneumonia or infection and may be used for future studies and novel therapies, after confirmation in an external population.

Introduction

Post-stroke infections occur in 30% of the patients with acute stroke and have a strong relation with unfavourable outcome.[1-3] In two recent large randomised trials, preventive antibiotic therapy did not improve functional outcome in relatively unselected patients with acute stroke.[4,5] New approaches are needed to prevent infection and thereby improve outcome in acute stroke patients. One option would be to prevent infection only in those with high risk of developing these infections.

Risk scores aiming to predict pneumonia in patients with acute stroke have previously been developed. These were mostly derived from retrospective cohort studies or large stroke registries and often were not specifically designed to predict post-stroke infections in general.[6-11] In contrast to many models aimed at predicting post-stroke pneumonia, only one study in 568 stroke patients aimed at predicting post-stroke infection in general. The model developed in this study was neither internally nor externally validated. Predictors for pneumonia from these studies were higher age, sex, more severe stroke, history of congestive heart failure, history of pneumonia or chronic obstructive pulmonary disease (COPD), current smoking, alcoholism, prestroke dependence, reduced level of consciousness, intracerebral haemorrhage (rather than ischaemic stroke), higher systolic arterial blood pressure, higher blood glucose, higher white blood cell (WBC) count and infratentorial location, intraventricular extension and volume of the haematoma. For any infection, higher age, diabetes and more severe stroke were predictors.

Aims

The current study aimed to construct a prediction rule both for pneumonia and for any post-stroke infection in patients with acute ischaemic stroke or intracerebral haemorrhage, with data from the ‘Preventive Antibiotics in Stroke Study’ (PASS).

Methods

Patients and definitions of outcome variables

Data were used from PASS, a multicentre, prospective, randomised, open-label, clinical trial with blinded endpoint assessment (PROBE), including 2550 patients in the acute phase (<24 h) of ischaemic stroke or intracerebral haemorrhage. Patients with signs of an infection at study inclusion were excluded. In this trial, preventive treatment with intravenous ceftriaxone during four days in addition to standard stroke care was compared with standard stroke care alone. Ethics approval was obtained from the local institutional review board and all patients or their legal representatives provided written informed consent. Baseline characteristics and outcome parameters were prospectively collected. Dysphagia was assessed by performing a swallowing test according to national guidelines.[12] We assessed for each patient whether the swallowing test was performed, the result of the test or the reason why it was not performed (good recovery, reduced consciousness or other reasons). A patient was considered dysphagic in case of an abnormal swallowing function on the test. A patient was considered not dysphagic in case of a normal swallowing function on the test, or when the test was not performed because of good recovery. When the test was not performed due to lowered consciousness or other reasons, dysphagic status was considered unavailable.

Definition of post-stroke infections

In the PASS, post-stroke infections were categorised as diagnosed by the clinician, and as judged by an independent adjudication committee blinded for treatment allocation with modified Centres for Disease Control and Prevention criteria, as described in the study protocol.[5,13] In the current analyses, the occurrence of pneumonia or infection as assessed by the clinician during admission was used.

Selection of candidate predictors

Previously described risk factors for post-stroke pneumonia and infection were identified by literature search (search strategy see Supplemental Appendix). To be considered as candidate predictors, risk factors had to be frequently described in literature and had to be collected in the PASS dataset. The maximum number of candidate predictors should range between the numbers of pneumonia or infection divided by 10–15 according to the events-per-variable rule.[14] For example, if the maximum number of candidate predictors was 10, we chose the 10 most frequently reported risk factors from literature. Each factor had to be known at admission, since the aim of the prediction rule is to assess infection risk early in clinical course. Dysphagia assessment was performed at admission by a trained nurse before any intake by a water swallowing test according to the Dutch Stroke Guideline.[12] Since pneumonia is a subcategory of infection, predictors of pneumonia were also used for prediction of infection. We also considered treatment allocation as a candidate predictor in both prediction rules.

Statistical analysis

All statistical analyses were performed in R version 3.2.4.[15] We developed two prediction rules; one for post-stroke pneumonia and one for post-stroke infections.

For each variable, the proportion of missingness was assessed. Missing data were imputed with single imputation with 20 iterations using baseline and outcome variables and the Mice Package.[16] For continuous variables, linearity was assessed by plotting the data and examining the risk for each decile. Logistic regression analyses were performed to study the association between candidate predictors as defined beforehand (literature search) and the outcomes. Predictors for pneumonia or infection were selected with backward selection, using the Akaike Information Criterion as stopping rule.[14] Next, the performance of both prediction rules was assessed by investigating the explained variance, calibration curve and Hosmer–Lemeshow test, and discrimination by c-statistic. Both prediction rules were internally validated to prevent overfitting. Regression coefficients were corrected for optimism by a shrinkage factor obtained by bootstrapping (250 samples) including the full backward model. When treatment allocation was selected as independent predictor, we adjusted the intercept and presented the predicted risks based on the standard care subset of the data. We constructed prediction rules based on the regression coefficients of the two models. For the performance of this study, we adhered to the Tripod statement and completed the checklist (Supplemental Appendix).[17]

Results

Patients

Between July 2010 and March 2014, 2550 patients were enrolled. Twelve patients withdrew consent immediately after randomisation, leaving 2538 patients available for the analysis; 57% of patients were male, the median NIHSS was 5 (IQR 3–9) and 26% of patients were dysphagic (for all baseline characteristics see Table 1). During admission, pneumonia was diagnosed in 159 and any infection in 348 patients (159 patients were diagnosed with pneumonia, 173 patients with urinary tract infections, 50 with other infections; see main article PASS for specification of other infections[5]); 306 patients were diagnosed with a single infection, and 48 with more than one infection: 29 patients had two infections, 11 patients had 3 infections and 2 patients had 4 infections as diagnosed by the treating physician.

Table 1.

Baseline characteristics preventive antibiotics in stroke study.

	Ceftriaxone group (n = 1268)	Control group (n = 1270)
Age, years	73 (63–81)	74 (63–81)
Male sex	719 (57%)	725 (57%)
History
Atrial fibrillation/flutter	184/1265 (15%)	207/1269 (16%)
Stroke	406/1266 (32%)	421/1270 (33%)
Hypercholesterolaemia	332/1260 (26%)	333/1258 (27%)
Hypertension	694/1266 (55%)	706/1268 (56%)
Myocardial infarction	172/1266 (14%)	159/1270 (13%)
Cardiac valve diseasea	95/1265 (8%)	78/1270 (6%)
Peripheral vascular disease	91/1262 (7%)	99/1267 (8%)
Obstructive pulmonary disease	115/1267 (9%)	93/1266 (7%)
Diabetes mellitus	251/1268 (20%)	251/1270 (20%)
Alcoholism	58/1268 (5%)	64/1270 (5%)
Malignancy	112/1268 (9%)	122/1270 (10%)
Immunocompromisedb	53/1268 (4%)	31/1270 (2%)
Current smoker	319/1253 (26%)	301/1256 (24%)
Medication at baseline
Anticoagulants	142/1267 (11%)	141/1270 (11%)
Antiplatelet therapy	514/1267 (41%)	504/1269 (40%)
Statins	473/1266 (37%)	476/1270 (38%)
Angiotensin-converting enzyme inhibitors	347/1264 (28%)	297/1268 (23%)
β blockers	428/1266 (34%)	457/1267 (36%)
Proton pump inhibitors	327/1265 (26%)	328/1266 (26%)
Pre-stroke Modified Rankin Scale scorec	0 (0–1)	0 (0–1)
National Institutes of Health Stroke Scale scored	5 (3–9)	5 (3–9)
Dysphagia	307/1178 (26%)	316/1193 (27%)
Acute stroke treatment
Intravenous thrombolysis	437/1268 (35%)	399/1270 (31%)
Coagulant therapy	26/143 (18%)	19/125 (15%)
Discharge diagnosis
Ischaemic stroke	1058 (83%)	1067 (84%)
Transient ischaemic attack	44 (4%)	49 (4%)
Intracerebral haemorrhage	143 (11%)	126 (10%)
Other	23 (2%)	28 (2%)

Data are median (IQR) or n/N (%).

aCardiac valve disease was defined as cardiac valve insufficiency, stenosis, or replacement. bImmunocompromised was defined as changed immune status, diabetes mellitus, alcoholism, malignancy or immunosuppressive medication.

cScores on the modified Rankin Scale range from 0 to 6, with 6 indicating death; modified Rankin Scale scores before onset of stroke symptoms were assessed in 2538 patients (1268 in the ceftriaxone group and 1270 in the control group).

dScores on the National Institutes of Health Stroke Scale range from 0 to 30, with 30 indicating highest degree of stroke severity; these scores were assessed in 2538 patients (1268 in the ceftriaxone group and 1270 in the standard treatment group).

Candidate predictors

The search strategy resulted in 2706 articles. After screening titles, abstracts and full-text, 46 articles were used for the extraction of risk factors for pneumonia and infection (see Supplemental Appendix). For pneumonia, the most consistently reported risk factors (>13 studies) were higher age, male sex, more severe stroke and dysphagia. Other frequently reported risk factors were decreased level of consciousness at admission, pre-stroke dependence, medical history of diabetes, COPD, atrial fibrillation and intracerebral haemorrhage. For infection, the most consistently reported risk factors were higher age, male sex, more severe stroke and a reduced consciousness on admission. Other relatively frequently reported risk factors were pre-stroke dependence, medical history of diabetes, previous stroke, atrial fibrillation, malignancy, immune suppression (HIV, immunosuppressive medication, splenectomy) and urinary catheterisation. The aforementioned risk factors were used as candidate predictors. Age and stroke severity were the only continuous variables and had a linear association with pneumonia and infection. The pre-stroke Modified Rankin Scale Score (mRS) and GCS were a priori categorised in three groups (mRS: 0, 1–2, 3–5; GCS <13, 13–14 and 15).

Handling of missing data

There were no missing data for the candidate predictors age, sex, intracerebral haemorrhage, pre-stroke dependence, stroke severity and immune suppression. For dysphagia, data were missing in 176 (6.9%) patients, and in 199 patients (7.8%), the verbal score of the GCS was missing (mostly because these patients were aphasic). For all other candidate predictors, data were missing in less than 0.3%.

Prediction rule

Post-stroke pneumonia was predicted by higher age, male sex, pre-stroke disability, medical history of COPD, more severe stroke, dysphagia and intracerebral haemorrhage. Post-stroke infections were predicted by higher age, male sex, medical history of diabetes, COPD, more severe stroke, dysphagia, use of bladder catheter, preventive antibiotic use and intracerebral haemorrhage. Because of selection of preventive antibiotic use as predictor for infection, the linear predictor for infection was adjusted to the placebo population. The linear predictors for pneumonia and infection are shown in Table 4 of supplementary appendix, together with the performance measures. After internal validation by bootstrapping, the linear predictor for pneumonia discriminated well between patients with and those without pneumonia (C-statistic 0.83 95%CI 0.80–0.86). The study was well calibrated (Hosmer–Lemeshow test p = 0.94). The linear predictor for infection discriminated well between patients with and those without infection (c-statistic 0.82 95% CI 0.79–0.84) and had a good calibration (Hosmer–Lemeshow test p = 0.44). The calibration plots for pneumonia and infection are shown in Figure 1.

Figure 1.

Calibration plot for pneumonia and infection. Legend ideal: line drawn for the situation in which predicted probabilities perfectly match the observed probabilities. Nonparametric: line displaying observed probabilities. Grouped observations: observed probabilities for 10 groups of equal size.

Derivation of a prediction rule and chart

We derived the PASS pneumonia rule and the PASS infection rule, 22 and 24 points scores respectively, which were both subdivided in four categories (Figure 2). Predicted risk for the lowest and highest score on the prediction rule ranged from 0.4 to 56.2% for pneumonia and 1.8% to 88.0% for infection. In Figure 2, the observed risks in the PASS population are shown for each point on the PASS pneumonia and infection rule. Discrimination of the derived scores remained good (C-statistic 0.84 95%CI 0.81–0.87 and 0.82 95% CI 0.79–0.84 for pneumonia and infection, respectively).

Figure 2.

Score on PASS prediction rule and predicted risks for pneumonia and infection. PASS: preventive antibiotics in stroke study.

Discussion

Post-stroke infections were predicted by higher age, male sex, history of diabetes, history of COPD, more severe stroke, dysphagia, use of a bladder catheter and intracerebral haemorrhage. Post-stroke pneumonia was predicted by higher age, male sex, pre-stroke disability, history of COPD, more severe stroke, dysphagia and intracerebral haemorrhage.

Our study is the first to develop and internally validate a prediction score for post-stroke infection. Post-stroke pneumonia is the most studied infection because of the strongest association with mortality and unfavourable functional outcome. However, other infections such as urinary tract infection, phlebitis, gastro-intestinal infections, sepsis and infections without determined focus can complicate and influence the clinical course after stroke as well. Infections have been shown to complicate the clinical course in 30%, while pneumonia occurs in 10% of stroke patients.[3] Because infections are also associated with mortality and unfavourable functional outcome, it seems necessary to aim therapies not only at prevention of pneumonia, but at preventing all infections.[2,3] In PASS, preventive antibiotic therapy did not have an effect on functional outcome in the overall population, but did reduce infection rate. Whether preventive antibiotic therapy in a high-risk subgroup does improve functional outcome remains to be investigated. The PASS prediction rule has the potential, after external validation, to identify the patients at high risk of infections and could be an important first step for selection of patients for future trials or intensive monitoring for the development of an infection during admission.

Previous models have been developed to predict pneumonia after stroke (see Table 2 for overview of all models). Four scores were also externally validated: the A2DS2 and AIS-APS scores for ischaemic stroke, the ICH-APS for intracerebral haemorrhage and the ISAN-score for both ischaemic stroke and intracerebral haemorrhage.[7-9,11] These four scores were developed from and mostly validated in large stroke registries. The other available prediction models for prediction of pneumonia after stroke were mostly based on a smaller number of patients and often too many predictors were included according to the events- per-variable rule, which can lead to overfitting and poorer performance of the model.[6,10,14,18] The present study adds to previous models because it was constructed from data from a large prospective randomised trial in which infection and pneumonia were predefined outcome parameters and might therefore have been more rigorously assessed. Also, patients with infection at admission were excluded, which is difficult to control in stroke registries. After external validation, the current study might therefore be better applicable when selection of patients in a randomised trial is considered. Finally, this study includes patients with either ischaemic stroke or intracerebral haemorrhage. The A2DS2 score was not developed in patients with intracerebral haemorrhage, the ISAN score was shown to perform less well in patients with intracerebral haemorrhage and other scores were made for either ischaemic or intracerebral haemorrhage.[19] If our results are confirmed in an independent population, the PASS pneumonia rule and infection rule could be applicable to patients with either ischaemic stroke or intracerebral haemorrhage (Table 3).

Table 2.

Models to predict pneumonia or infection in acute ischaemic stroke or intracerebral haemorrhage patients.

Author, year, name	Stroke type	Study design	No. of patients in derivation group	Predictors	C-statistic	Validation
Pneumonia
Kwon et al., 2006[10]The pneumonia score	I&H	Cohort study	286	Age, sex NIHSS, dysphagia, mechanical ventilation	NR	None
Chumbler et al., 2010[6]	I&H	Retrospective cohort	926	Age, stroke severity, dysphagia, history of pneumonia, patient being ‘found down’ at symptom onset	0.78 (D); 0.76 (V)	Internal
Hoffmann et al., 2012[7]A2DS2-score	I	Registry	15,335	Age, sex, stroke severity, dysphagia, atrial fibrillation	0.84 (D); 0.84 (V)	External
Ji et al., 2013[9]AIS-APS	I	Registry	8820	Age, history of atrial fibrillation, congestive heart failure, COPD, current smoking, prestroke dependence, dysphagia, NIHSS, GCS, stroke subtype, blood glucose	0.79 (D); 0.79 (V)	External
Ji et al., 2014[8]ICH-APS	H	Registry	2998	Age, NIHSS, prestroke dependence, GCS, dysphagia, current smoking, alcoholism, COPD, infratentorial location of ICH, intraventricular extension, hematoma volume.	0.75 (D); 0.76 (V)	Internal
Harms et al., 2013[18]Pantheris-score	I on ICU	RCT	114	age, GCS, systolic arterial blood pressure, WBC count	0.85 (D); 0.88 (int. V)	Internal
Smith et al., 2015[11]ISAN-score	I&H	Registry	11,551	Age, sex, NIHSS, prestroke independence	0.79 (D); 0.78 (V)	External
Kumar et al, 201720ACDD4-score	I&H	Retrospective cohort	1644	Age, congestive heart failure, dysarthria, dysphagia	0.82 (D); 0.81 (V)	Internal
Westendorp et al., 2017PASS score	I&H	RCT	2538	Age, sex, pre-stroke disability, medical history of COPD, stroke severity, dysphagia, intracerebral haemorrhage	0.82 (IV)	Internal
Infection
Friedant et al., 201521	I	Retrospective cohort	568	Age, diabetes, stroke severity	NR	None
Westendorp et al., 2017 PASS score	I&H	RCT	2538	Age, male sex, diabetes, medical history of COPD, stroke severity, dysphagia, bladder catheter, intracerebral haemorrhage	0.84 (IV)	Internal

I: ischaemic; H : haemorrhagic; WBC : white blood cell; COPD : chronic obstructive pulmonary disease; NIHSS: National Institutes of Health Stroke Scale score; GCS : Glasgow coma scale; NR: not reported; D : derivation; V : validation; IV : internal validation.

Table 3.

Prediction chart for PASS pneumonia rule and PASS infection rule.

PASS pneumonia rule			PASS infection rule
Characteristic		Points	Characteristic		Points
Age			Age
- 51–60		1	- 51–60		1
- 61–70		2	- 61–70		2
- 71–80		3	- 71–80		3
- >80		4	- >80		4
Male sex		2	Male sex		−1
History of			History of
- COPD		2	- COPD		2
			- diabetes		1
Pre-stroke disability
- mRS 1–2		2
- mRS > 2		1
Intracerebral haemorrhage		1	Intracerebral haemorrhage		1
Stroke severity (NIHSS)			Stroke severity (NIHSS)
- 6–10		2	- 6–10		2
- 11–20		4	- 11–20		4
- 21–30		6	- 21–30		6
			Use of bladder catheter		6
Dysphagia		5	Dysphagia		4
Score	Risk category	Pneumonia % (n/N)	Score	Risk category	Infection % (n/N)
0–5	Low	1 (7/1072)	0–5	Low	4 (47/1315)
6–10	Moderate	4 (34/855)	6–10	Moderate	11 (72/633)
11–15	High	17 (88/507)	11–16	High	31 (116/379)
16–22	Very high	29 (30/104)	17–24	Very high	54 (113/211)

PASS: preventive antibiotics in stroke study; COPD: chronic obstructive pulmonary disease; NIHSS: National Institutes of Health Stroke Scale score.

This study has limitations. First, the PASS dataset did not contain all characteristics reported in previous studies as possible risk factors for infection or pneumonia. For example, history of dementia, ‘being found down at symptom onset’, location of stroke, ‘tongue pressure movements’ or laboratory markers such as monocytic HLA-DR, IL-1 or IL-6 were not prospectively collected in the PASS. Also, CT- or MRI-characteristics of an infarct or bleeding were not used for the development of this model. Since the strongest predictors for infection – as described in literature – were included, it is unlikely that this strongly affected the results and that these factors would otherwise have been included in the model. Also, patients included in PASS had a relatively mild stroke severity that decreased infection and pneumonia rates. External validation should be performed to assess the performance of the model in stroke patients with higher stroke severity and infection rates. Next, the predictive value of the use of a bladder catheter could be overestimated, since it cannot be excluded that asymptomatic bacteriuria was interpreted as bladder infection in patients with a bladder catheter. Also, in standard stroke practice in the Netherlands, the majority of patients will receive a bladder catheter within the first 24 h when needed, but we did not collect the exact date and time of placement of the catheter. Finally, intracerebral haemorrhage was selected as a predictor for both infections and pneumonia, but this might be explained by increased length of stay. Patients with intracerebral haemorrhage had a longer length of stay than patients with ischaemic stroke, and we used the length of admission as exposure time for infections and pneumonia.

In conclusion, we present internally validated risk scores for post-stroke infection and pneumonia. After external validation, these scores can be used for selection of patients at high risk of infection and pneumonia after stroke.

Click here for additional data file.

Supplemental material, Supplementary appendix for Development and internal validation of a prediction rule for post-stroke infection and post-stroke pneumonia in acute stroke patients by Willeke F Westendorp, Jan-Dirk Vermeij, Nina A Hilkens, Matthijs C Brouwer, Ale Algra, H. Bart van der Worp, Diederik WJ Dippel, Diederik van de Beek and Pual J Nederkoorn; on behalf of the PASS-investigators in European Stroke Journal