Multinational development and validation of an early prediction model for delirium in ICU patients.

RATIONALE: Delirium incidence in intensive care unit (ICU) patients is high and associated with poor outcome. Identification of high-risk patients may facilitate its prevention.
PURPOSE: To develop and validate a model based on data available at ICU admission to predict delirium development during a patient's complete ICU stay and to determine the predictive value of this model in relation to the time of delirium development.
METHODS: Prospective cohort study in 13 ICUs from seven countries. Multiple logistic regression analysis was used to develop the early prediction (E-PRE-DELIRIC) model on data of the first two-thirds and validated on data of the last one-third of the patients from every participating ICU.
RESULTS: In total, 2914 patients were included. Delirium incidence was 23.6%. The E-PRE-DELIRIC model consists of nine predictors assessed at ICU admission: age, history of cognitive impairment, history of alcohol abuse, blood urea nitrogen, admission category, urgent admission, mean arterial blood pressure, use of corticosteroids, and respiratory failure. The area under the receiver operating characteristic curve (AUROC) was 0.76 [95% confidence interval (CI) 0.73-0.77] in the development dataset and 0.75 (95% CI 0.71-0.79) in the validation dataset. The model was well calibrated. AUROC increased from 0.70 (95% CI 0.67-0.74), for delirium that developed <2 days, to 0.81 (95% CI 0.78-0.84), for delirium that developed >6 days.
CONCLUSION: Patients' delirium risk for the complete ICU length of stay can be predicted at admission using the E-PRE-DELIRIC model, allowing early preventive interventions aimed to reduce incidence and severity of ICU delirium.

Introduction

Delirium is an acute organic brain dysfunction characterised by disturbances of attention and cognition with a fluctuating course, as a direct consequence of an underlying medical condition [1]. Delirium occurs frequently in intensive care unit (ICU) patients [2, 3], and is associated with poor outcome [3-6].

Systematic delirium assessment in ICU patients is important to deliver adequate patient care by allowing clinicians to detect and treat delirium at an early stage [7-9]. Several promising interventions to prevent delirium are available that target cognitive impairment, sleep deprivation, immobility, and visual and hearing impairment [10-15]. Although one study found no beneficial effects of haloperidol as prophylaxis and treatment for ICU delirium [16, 17], other studies have shown that prophylactic haloperidol may exert beneficial effects in critically ill patients [18, 19].

As part of a process to improve delirium care and to further facilitate prevention of ICU delirium, identification of high-risk patients is paramount [20], since these are the most fragile patients who require maximum preventive efforts [21]. Also, when preventive measures can be restricted to high-risk patients, both the number of patients who will be unnecessarily exposed to potential harmful side effects [22] and costs are likely less. Moreover, the effectiveness of prevention seems more pronounced in patients with a higher risk for delirium [18]. Finally, prediction of the ICU delirium risk provides relevant information for family members and caregivers and will facilitate stratification of patients in future delirium prevention studies. This stratification will positively influence the efficient use of research resources.

Currently, an ICU delirium prediction model is available [23, 24]. However, the usefulness of this model is limited by the fact that it requires predictors obtained during the first 24 h of ICU admission. A relevant number of ICU patients develop delirium within 24 h following admission [25, 26], and preventive measures should ideally be applied as early as possible. This calls for a prediction model that can reliably predict delirium at the time of ICU admission. Therefore, the objective of this multinational study was to develop and validate a model based on data available at ICU admission to predict the development of delirium during a patient’s complete ICU stay, and to determine the predictive value of this model in relation to the time of development of delirium.

Some results of this study have been previously presented at the ISICEM congress in Brussels, March 2014, the ESICM congress in Barcelona, October 2014 [27], and the EDA congress in Cremona, November 2014 [28].

Methods

Design and study population

A multinational prospective cohort study was carried out in 13 ICUs from seven countries (Australia, Belgium, England, Germany, Spain, Sweden, and the Netherlands) (Table E1 online data supplement). Each participating ICU included all consecutive patients aged ≥18 years (surgical, medical, neurology/neurosurgical, or trauma patients) during a continuous period of approximately three months between October 2011 and June 2012. Patients were excluded if: they were delirious at ICU admission; had an ICU stay shorter than one day; were unable to reliably assess for delirium (e.g., coma during entire ICU stay; unable to understand the language spoken; severely mentally disabled; serious receptive aphasia; serious auditory or visual disorders); or if the compliance rate of the CAM-ICU for delirium screening was <80 % during an individual patient’s ICU stay (Fig. 1).

Fig. 1

Study flow chart

This study was approved by the medical ethical research committee Arnhem-Nijmegen region, The Netherlands (CMO Region Arnhem-Nijmegen, no. 2010/365), and conducted in accordance with the applicable rules concerning the review of research ethics committees and informed consent. All participating centres obtained ethics approval from their committees to conduct the study.

Data collection

Data of 18 candidate delirium predictors were collected at ICU admission in an electronic clinical report form using a secured website. Candidate predictors were selected based on the PRE-DELIRIC model [23, 24], a systematic review about risk factors for delirium [29] and on expert opinion [gathered during the annual meeting of the working group Postoperative Delirium and Cognitive Dysfunction (PoDeCoD) of the European Society of Intensive Care Medicine (ESICM)]. A proposal with candidate predictors, based on the PRE-DELIRIC model and the systematic review, was presented and discussed. Candidate predictors were removed from/added to the final list based on consensus of the members of this working group.

Candidate predictors available at time of ICU admission were: age, gender, history of cognitive impairment (history of dementia, mild cognitive impairment or delirium), history of alcohol, nicotine and drugs abuse, history of vascular disease, Glasgow Coma Scale (GCS) score, diabetes, blood urea nitrogen (BUN), use of opiates in the 24 h before ICU admission, use of anti-psychotics before ICU admission, admission category (surgery, medical, trauma, neurology/neurosurgery), urgent admission, mean arterial blood pressure (MAP), (strong suspicion of) infection, use of corticosteroids, or respiratory failure (see Table E2 online data supplement for exact definitions). Data on sedation and illness severity (APACHE) were not collected, as these are not available at ICU admission.

Delirium assessment

All consecutive patients were screened for delirium by trained ICU nurses at least once every 8- or 12-h shift using the CAM-ICU. In accordance with previous prediction studies [23, 24], delirium was diagnosed when the CAM-ICU was positive [7, 30], and/or when a patient was treated with haloperidol or other anti-psychotics because of delirium (see Table E2). Patients were screened using the CAM-ICU as soon as delirium assessment was possible after awakening from sedation. To exclude a potential source of bias, assessors of the CAM-ICU were not made aware of the fact that they were collecting data for this study [31].

In order to check the quality of the delirium assessments, compliance with the CAM-ICU was per participating centre calculated monthly as the percentage of assessments performed per day in relation to the total number of assessments that should have been performed. Also, inter-rater reliability (IRR) measurements of the CAM-ICU were performed by duplicate measurements of the CAM-ICU for all admitted ICU patients, on one given day per month per centre, by comparing CAM-ICU scores assessed by the ICU nurse with the scores assessed by a CAM-ICU expert. CAM-ICU experts and ICU nurses were blinded to each others delirium assessments. If per centre, the CAM-ICU screening compliance was above 80 % and the Cohen’s kappa was above 0.80, their data were considered reliable. Centres that did not meet these criteria were per centre excluded from the analysis, after which re-analysis was performed. If exclusion of a centre did not significantly affect the performance of the E-PRE-DELIRIC model, the centre was included in the final analysis, as the data were considered to be of sufficient quality [23, 24].

Sensitivity and specificity of the delirium assessments were calculated using the CAM-ICU outcomes obtained by CAM-ICU experts compared with those of the ICU nurses.

Outcome definition

The primary outcome of this study was the development of ICU delirium, defined as a positive assessment for delirium during a patient’s complete ICU stay.

Statistical analysis

Statistical analyses, including information regarding missing values, are described in Supplemental file 1 online data supplement. The model was developed using multiple logistic regression analysis on data of the first two-thirds of the patients of all participating hospitals. Selection of the final predictors of the E-PRE-DELIRIC model took place through manual backward selection of the candidate predictors using a P value of >0.10 for exclusion. The model was validated on data of the last one-third of the patients. Discriminative power was assessed using the area under the receiver operating characteristic curve (AUROC) [31]. Calibration was assessed graphically by plotting the observed outcome frequencies against the mean predicted outcome probabilities or risks, within subgroups of patients that were ranked by increasing estimated probability [32]. To evaluate the predictive value of the E-PRE-DELIRIC model related to the moment at which delirium first occurred, the database was divided into four groups based on the quartiles of the time to development of delirium at: days 0–1; day 2; days 3–6; >day 6.

Data were analyzed using IBM® SPSS® Statistics 20.01 and R statistics R 3.0.1 [33]. In this study, we used the Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD), a strict methodology for data reporting, to check our adherence to suggested optimal levels of transparency and completeness of reporting of a prediction model [34].

Sample size calculation

The E-PRE-DELIRIC model was compiled from 18 candidate predictors. For each candidate predictor, at least ten patients with delirium were needed for the validation and calibration of the model [35]. With an anticipated delirium incidence of 15–30 %, an expected attrition of 25 %, we aimed to enrol at least (18 × 10/0.15)/0.75 = 1600 patients.

Results

In total, 5352 patients were screened, of whom 2438 patients did not fulfil the inclusion criteria (Fig. 1). The cohort consisted of 2914 patients of whom 1962 patients were included in the development dataset. The ICU delirium incidence was 24.5 %. The remaining 952 patients were included in the validation dataset with an ICU delirium incidence of 21.8 %. The demographic characteristics of both groups were comparable (Table 1).

Table 1

Patient characteristics

Variable	Development dataset (n = 1962)	Validation dataset (n = 952)	Whole study population (n = 2914)
Age in years, mean (Q1–Q3, min/max)	61.7 (53–74, 18/95)	60.6 (51–73, 18/94)	61.3 (52–73, 18/95)
Male, n (%)	1166 (59.4)	550 (57.8)	1716 (58.9)
Admission category, n (%)
Surgery	1019 (51.9)	476 (50.0)	1495 (51.3)
Medical	683 (34.8)	338 (35.5)	1021 (35.0)
Trauma	90 (4.6)	44 (4.6)	134 (4.6)
Neurology/neurosurgery	170 (8.7)	94 (9.9)	264 (9.1)
Urgent admission, n (%)	1163 (59.3)	570 (59.9)	1733 (59.5)
LOS-ICU in days, median (Q1–Q3, min/max)	2.0 (1–6, 1/133)	2.0 (1–5, 1/125)	2.0 (1–5, 1/133)
Delirium, n (%)	481 (24.5)	208 (21.8)	689 (23.6)

CAM-ICU compliance and inter-rater reliability

As exclusion of the centres with a CAM-ICU screening compliance <80 % and/or a Cohen’s kappa <0.80 did not significantly affect the performance of the E-PRE-DELIRIC model, all centres were subsequently included in the final analysis. The overall CAM-ICU compliance was 83 % (Q1–Q3: 78–93). In total, 648 inter-rater CAM-ICU measurements were performed. The mean IRR measurements were 0.83 [95 % confidence interval (CI) 0.78–0.88] Cohen’s kappa (Table E3 online data supplement). The CAM-ICU had a sensitivity of 0.82 (95 % CI 0.76–0.87) and a specificity of 0.98 (95 % CI 0.96–0.99).

Discrimination and calibration of the E-PRE-DELIRIC model

The developed E-PRE-DELIRIC model consists of nine predictors: age, history of cognitive impairment, history of alcohol abuse, BUN at time of ICU admission, admission category, urgent admission, MAP at the time of ICU admission, use of corticosteroids, and respiratory failure (Table 2). The discriminative power (AUROC) was 0.76 (95 % CI 0.73–0.78) in the development dataset. This database was divided in groups based on the quartiles of the predicted probabilities for delirium development: very low (0–10 %), low (10–20 %), moderate (20–35 %), and high risk for delirium (>35 %). For each of these groups, the sensitivity, specificity and likelihood ratios were calculated (Fig. 2). At a cut-off of 24.5 %, which was the ICU delirium incidence in the development dataset, the sensitivity and specificity were 71 and 69 %, respectively. The AUROC was 0.75 (95 % CI 0.71–0.79) in the validation dataset (see Figure E1 online data supplement for the calibration plot of the validation dataset). Fitting the model Log(Odds) = b0 + b1 × LP on the observed data showed that recalibration was not necessary. The slope of the model was 0.961 with an intercept of −0.094.

Table 2

Variables of the E-PRE-DELIRIC model and regression coefficients and formula for the E-PRE-DELIRIC model

Variable	Regression coefficient	Odds ratio (95 % CI)
Age	0.025	1.025 (1.017–1.033)
History of cognitive impairment	0.878	2.406 (1.700–3.404)
History of alcohol abuse	0.505	1.657 (1.170–2.347)
Admission category
Surgery	RC	RC
Medical	0.370	1.448 (1.058–1.982)
Trauma	1.219	3.384 (1.997–5.735)
Neurology/neurosurgery	0.504	1.655 (1.064–2.575)
Urgent admission	0.612	1.843 (1.326–2.563)
MAP at the time of ICU admission	−0.006	0.994 (0.988–1.001)
Use of corticosteroids	0.283	1.327 (0.996–1.768)
Respiratory failure	0.982	2.670 (2.099–3.396)
BUN at time of ICU admission	0.018	1.018 (1.006–1.031)

Data development set. Formula for the E-PRE-DELIRIC model: Risk of delirium = 1/[1 + exp − (−3.907 + 0.025 × age + 0.878 for history of cognitive impairment + 0.505 for history of alcohol abuse + 0 for surgery or 0.370 for medical or 1.219 for trauma or 0.504 for neurology/neurosurgery + 0.612 for urgent admission − 0.006 × MAP at the time of ICU admission + 0.283 for use of corticosteroids + 0.982 for respiratory failure + 0.018 x BUN in mmol/l at time of ICU admission)]

RC reverence category

Fig. 2

Area under the receiver operating characteristic curve (AUROC) of development dataset (AUROC = 0.76)

Prediction of early and late development of delirium

In patients who developed delirium at days 0–1 the AUROC was 0.70 (95 % CI 0.67–0.74) with a sensitivity and specificity of 62 and 67 %, respectively. This increased to 0.76 (95 % CI 0.72–0.80) (development of delirium at day 2), 68 and 67 %; 0.77 (95 % CI 0.74–0.80) (development of delirium at days 3–6), 72 and 68 %; and 0.81 (95 % CI 0.78–0.84) (development of delirium after day 6), 78 and 68 %, respectively. The predictive value of the model improved significantly over time (Fig. 3).

Fig. 3

Area under the receiver operating characteristic curve (AUROC) during ICU stay. Error bars 95 % CIs

Discussion

In this prospective, international multicentre study, we developed and validated a model allowing early prediction of patients likely to develop delirium during ICU admission. Using nine predictors that are readily available at ICU admission, it is now possible to stratify ICU patients for delirium risk for the complete ICU length of stay with a high discriminative power. In contrast to the earlier prediction model [23, 24], which reliably predicts delirium using data available only after 24 h in the ICU, our E-PRE-DELIRIC model has a moderate-to-good performance using data available immediately at ICU admission. Of interest, the predictive accuracy of the model is higher for delirium that develops later compared to delirium that develops early following ICU admission.

In view of the high prevalence of delirium in ICU patients [2, 3] and its serious consequences [3-6], prediction of ICU delirium is clearly of clinical relevance as early risk identification may serve different purposes. Both caregivers and patients families can be informed early about the risk of developing delirium in the ICU, and this may, for example, assist them in taking decisions related to the use of preventive measures. In addition, the model enables stratification for the risk to develop delirium, which may be helpful in future trials that aim to investigate preventive interventions. Our group previously reported the effects of prevention with low-dose haloperidol in critically ill patients using a prediction model and demonstrated that patients with the highest risk appear to benefit most from prevention [18]. With the E-PRE-DELIRIC model, stratification is possible immediately after ICU admission. Various other preventive measures that target modifiable delirium risk factors have been found beneficial in non-critically ill patients [8, 10, 11] and need confirmation in ICU patients. In these confirmation trials, stratification of the a priori risk to develop delirium should be determined to demonstrate that treatment groups are adequately randomized. As a consequence, possible interventions that may be expensive and/or labour intensive can then be provided only to those patients with highest risk. In this way, improved quality and efficiency of care may be achieved in a cost-effective manner [36].

The predictors that build the E-PRE-DELIRIC-model are largely consistent with previously reported risk factors for delirium, including age, pre-existing dementia, history of alcoholism, and a high severity of illness at admission [8, 37, 38]. Other factors that confirm previously reported predictors include admission category, urgent admission, BUN [23, 24], and respiratory failure [37-39]. While use of corticosteroids was not reported as a relevant factor in another study [2], in our study the use of corticosteroids was a significant predictor for delirium. We only collected data about corticosteroids use at ICU admission, in contrast, the authors of [2] collected daily data about corticosteroids use during the whole study period. Of importance, as relevant differences between countries may exist, the current study was a multinational study using data of ICUs from different countries for the development of a delirium prediction model, which increases its generalisability. Interestingly, while the delirium incidence remains stable over time in the four groups in this study, the predictive value of the model increases over time. However, examining differences between early and late delirium did not fall within the scope of this study, and therefore should be studied in future research.

Several limitations of this study need to be addressed. First, despite the fact that availability of candidate predictors at ICU admission and the effort it would take to collect the data in daily clinical practice were leading in determining candidate predictors, three of the predictors from the E-PRE-DELIRIC model (BUN, history of alcohol abuse, and cognitive impairment) had a fairly high rate of missing values. Probably due to unawareness of the importance of these predictors for delirium prediction, currently these predictors are not collected in a standardised manner in daily practice. When health professionals need the information about these predictors to be able to use the E-PRE-DELIRIC model, they will (most often) be able to collect the information as they are readily available. We used imputation techniques to handle missing data. Multiple imputation helps to reduce bias and increases precision by including variation of data [31]. Second, the performance of the CAM-ICU in daily practice is not optimal [40-42]. However, in contrast to the evaluation study of Van Eijk et. al [40], in the current study ICU patients were screened for delirium at least two times daily and during the whole ICU length of stay. This increases the chance of correct delirium scores, because of the fluctuating course of delirium. The scores were determined in the same way in all participating ICUs, positively influencing possible measurement bias. In addition, data on the use of haloperidol and other anti-psychotics for delirium were used [23]. This usage was consistent among the different centres. Therefore, we may assume that sensitivity of the bedside nurse to recognize delirium anywhere during ICU admission is high. The quality of data of the CAM-ICU screening was further improved by determination of compliance and inter-rater reliability in each ICU. Third, although the full model approach to include all 18 candidate predictors in the predictive model is likely to avoid overfitting and selection data [43], we chose to perform manual backward selection of predictors to improve the clinical feasibility. Fourth, based on the predetermined exclusion criteria, a large number of patients had to be excluded. One of these exclusion criteria was an ICU stay shorter than one day. Since the E-PRE-DELIRIC model will be used prospectively following ICU admission, one may not be able to anticipate a patient’s ICU length of stay (LOS). Therefore, for each patient admitted to the ICU, the E-PRE-DELIRIC score will be calculated. Predictions for patients with an ICU LOS shorter than one day might be different than those for patients with an ICU LOS longer than one day. Therefore, in future research, it is recommended to include these patients to determine the influence on the performance of the E-PRE-DELIRIC model. Finally, it should be taken into account that selection of candidate predictors in the E-PRE-DELIRIC model might be biased. However, the candidate predictors were based on multiple scientific sources, including the previously published PRE-DELIRIC model [23, 24], a systematic review about delirium risk factors [29], and expert opinion. Although external validation of the E-PRE-DELIRIC model in new centres was not performed, this prediction model was internationally developed and prospectively validated in 13 ICUs from seven different countries. The E-PRE-DELIRIC model is a result of a relevant sample from these ICUs, and therefore widely applicable and probably generalisable to other countries. ICUs should consider, however, that the prediction of delirium derived from the E-PRE-DELIRIC model is an approximation; there can always be some over- or underestimation in the predicted delirium risk. Therefore, for future research, it is recommended to test the external validity of the E-PRE-DELIRIC model in other centres to further improve its generalisability.

In conclusion, in this multinational study, we developed and validated an early ICU delirium prediction model that revealed sufficient validity. It enables the clinician to identify those patients likely to develop delirium following ICU admission using only nine predictors. The model allows early delirium preventive interventions in ICU patients with a high risk of delirium.

An automatic version of the E-PRE-DELIRIC model (Excel and web-based) will soon be available at https://www.radboudumc.nl/Research/Organisationofresearch/Departments/intensive%20care/Pages/vandenBoogaard.aspx.

(In multiple languages available). Furthermore the E-PRE-DELIRIC model will soon be integrated in the DeliriumICU app which makes it possible to use it bedside (http://itunes.apple.com/us/app/deliriumicu/id511306390 and https://play.google.com/store/apps/details?id=dotsdigits.deliriumicu).

Supplementary material 1 (DOCX 38 kb)