On the utility of diagnostic instruments for pediatric delirium in critical illness: an evaluation of the Pediatric Anesthesia Emergence Delirium Scale, the Delirium Rating Scale 88, and the Delirium Rating Scale-Revised R-98.

PURPOSE: Delirium is a poor-prognosis neuropsychiatric disorder. Pediatric delirium (PD) remains understudied, particularly at pediatric intensive care units (PICU). Although the Pediatric Anesthesia Emergence Delirium (PAED) scale, the Delirium Rating Scale (DRS-88), and the Delirium Rating Scale-Revised (DRS-R-98) are available, none have been validated for use in PICU settings. The aim of the present study was to investigate the use of the DRS/PAED instruments as diagnostic tools for PD in the PICU.
METHODS: A prospective panel study was conducted, under circumstances of routine clinical care, investigating the diagnostic properties of the PAED, DRS-88, and DRS-R-98 in PICU patients at a tertiary university medical center. A total of 182 non-electively admitted, critically ill pediatric patients, aged 1-17 years, were included between November 2006 and February 2010. Sensitivity, specificity, and receiver operating characteristic (ROC) curves were calculated. Three psychometric properties were analyzed: (1) internal consistency (2) proportion of items not rateable, and (3) discriminative ability.
RESULTS: The PAED could be completed in 144 (93.5%) patients, much more frequently than either the DRS-88 (66.9%) or the DRS-R-98 (46.8%). Compared with the clinical gold standard diagnosis of delirium, the PAED had a sensitivity of 91% and a specificity of 98% (AUC 0.99). The optimal PAED cutoff score as a screening instrument in this PICU setting was 8. Cronbach's alpha was 0.89; discriminative ability was high.
CONCLUSIONS: The PAED is a valid instrument for PD in critically ill children, given its reliance on routinely rateable observational signs and symptoms.

Introduction

Delirium is a neuropsychiatric disorder secondary to a general medical condition and represents a serious complication of the underlying disease or its treatment [1]. Three subtypes of delirium can be distinguished: (1) hypoactive delirium, which is characterized by apathy, decreased responsiveness, and withdrawal; (2) hyperactive delirium, which is characterized by restlessness, emotional lability, and agitation; (3) mixed delirium, which is a combination of the two other subtypes [2]. The condition is frequently seen in critically ill adult and geriatric patients and is associated with poor prognosis, reflected by worse functional and cognitive outcome, prolonged hospital stay, and a higher mortality rate after discharge from hospital [1, 3–5]. Although the clinical picture of delirium in adults is well known, comparatively scarce attention has been paid to its counterpart in children—pediatric delirium (PD)—even though data suggest that PD is a similarly serious condition [2, 6–8].

It is important to correctly diagnose delirium in children, in order to make early intervention possible. Adult delirium is often missed by clinicians, because it commonly manifests as the hypoactive subtype [9-11]; anecdotal evidence suggests that hypoactive forms, and misclassification thereof, may be even more prevalent in the pediatric population. As a consequence, there is an urgent need for a valid instrument which can be used by staff at pediatric intensive care units (PICU) [2, 12] to assess PD. The Pediatric Confusion Assessment Method for the Intensive Care Unit (pCAM-ICU), the Pediatric Anesthesia Emergence Delirium (PAED) scale, the Delirium Rating Scale (DRS-88), and the Delirium Rating Scale-Revised (DRS-R-98) are instruments that have been developed to diagnose delirium in clinical practice. The pCAM-ICU has recently been validated as a highly reliable instrument in pediatric critically ill children (8–15 years) [13].

The PAED, which only includes items covering behavioral symptoms, has been validated in children in the age range of 18 months–6 years in a post-anesthesia setting [14]. However, in that study, the diagnosis of delirium was only made if the anesthetist decided that treatment of the condition was necessary. Therefore, delirium could have been underdiagnosed. The DRS-88 and the DRS-R-98 focus more on cognitive functioning [15] and have only been validated in adults, and not in ICU settings [12]. Although Turkel and colleagues [16] reported that the DRS-88 could be applied to diagnose delirium in children (6 months–19 years), their study was carried out in retrospect and no control group was available. Thus, to our knowledge, only the pCAM-ICU has been tested in PICU settings in children, whereas the DRS/PAED instruments have not been tested in a PICU setting.

Therefore, the main objective of the present prospective panel study was to investigate, under circumstances of routine clinical care, the use of the DRS/PAED instruments as diagnostic tools for PD in a PICU setting, taking into account both item and sum score levels of the instruments. It was hypothesized that (1) the PAED would be more suitable than the DRS in this setting, as it contains only behavioral items, which are easier to assess in young critically ill children and, therefore, (2) a significant negative impact of younger age was expected with regard to the proportion of rateable items for both versions of the DRS (as it relies on cognitive items that cannot be assessed in the youngest or most ill children), but not for the PAED.

Method

Setting and patients

Data pertained to a cohort of critically ill (defined as a threatening failure of the brain, heart, or lungs) pediatric patients, admitted from November 2006 until February 2010. The setting was a tertiary eight-bed PICU at Maastricht University Medical Centre (MUMC+) in the Netherlands. All non-electively admitted patients to the PICU between the ages of 1 and 17 years, ventilated and non-ventilated, were included. The sampling frame also included patients who had been admitted to the PICU after an elective surgical procedure and who were still at the PICU after 48 h. Patients were excluded if they were (1) non-Dutch speakers and (2) not in need of intensive care. As the information necessary to rate the DRS/PAED instruments is collected as part of routine clinical care at the PICU, under Dutch law the study did not fall under the remit of the medical ethics committee, provided data anonymity is preserved at all times.

Measurement procedures

The Pediatric Anesthesia Emergence Delirium (PAED) scale

The PAED was developed by Sikich and Lerman [14] in order to trace emergence delirium (ED). ED is defined as a disorder occurring during awakening from anesthesia in the immediate postoperative period. It is characterized by confusion, hallucinations, delusions and can be accompanied by moaning, restlessness, and agitation. The scale consists of five items which are rated on a 5-point scale ranging from ‘extremely’ to ‘not at all.’ A score of 10 or higher indicates diagnosis of delirium [14]. For the purpose of the present analyses, a maximum of two missing items (2/5 = 40%) was allowed a priori; a weighted sum score was computed in the case of one or two missing items.

The Delirium Rating Scale

The DRS-88 was developed by Trzepacz and colleagues [17]. It consists of ten items encompassing a range of areas which can be affected by delirium such as hallucinations, psychomotor behavior, cognitive status, sleep-wake cycle disturbance, and lability of mood. A score of 10 or higher indicates a diagnosis of delirium [18, 19]. For the purpose of the present analyses, a maximum of four missing items (4/10 = 40%) was allowed a priori; a weighted sum score was computed in the case of four or fewer missing items.

The Delirium Rating Scale-Revised (DRS-R-98)

The DRS-R-98 is a revised form of the original DRS and consists of 16 items, of which the last three are optional and scored only when there is suspicion of delirium (score of 15.25 or higher on the first 13 items); the three optional items serve to differentiate between delirium and other disorders. For the purpose of the present analyses, a maximum of five missing items was allowed a priori if only the first 13 items were scored (5/13 = 39%) and a weighted sum score was computed in the case of five or fewer missing items. If all 16 items were scored, a maximum of 6 missing items was allowed (6/16 = 38%) and a weighted sum score was computed in cases of 6 or fewer missing items [15].

Clinical/expert judgment of the gold standard

The PICU pediatric neuropsychiatrist (JS) routinely examined all children using the criteria of the revised fourth edition of the DSM-IV [1]. He reviewed and discussed all available information and opinions (from parents, nurses, and the medical charts) with the other members of the PICU team before making a final diagnosis [7, 12]. He also made further categorizations into the hyperactive, hypoactive, and mixed states of PD.

Procedure

Patients at our PICU are routinely assessed for the presence of delirium under circumstances of quality improvement in clinical care since 2005, using a schedule of twice-daily examinations. The PAED, DRS-88, and DRS-R-98 are completed during the first assessment, which is carried out by the intern over the course of their attachment to the unit, having received on-site clinical training by the pediatric neuropsychiatrist of the unit. The second assessment was carried out—within 3 h—by the pediatric neuropsychiatrist, who was unaware of the screening results. PAED, DRS-88, and DRS-R-98 were completed once a day. Once PD was diagnosed, appropriate management was initiated in collaboration with the PICU team.

Statistical analysis

Data were analyzed using STATA, version 11.0 [20]. For each patient, a single assessment day during hospitalization was selected. For patients diagnosed with PD, PAED, DRS-88, and DRS-R-98 assessments on the first day of the PD gold standard diagnosis were selected; for patients without PD, PAED, DRS-88, and DRS-R-98 assessments were randomly selected in such a way that groups were frequency-matched on number of admission days until assessment. Exclusion criteria for analysis were unconsciousness or deep sedation according to the clinical judgment of the responsible intensivist treating the patient.

A patient was regarded non-rateable on a scale if more than the maximum allowed number of items was missing, precluding diagnosis. Non-rateability of items was examined as a function of age using one-way analysis of variance (ANOVA, one-tailed).

Three psychometric properties of the PAED, DRS-88, and DRS-R-98 (with exclusion of the three diagnostic items of the last scale, because scoring of these items is optional) were evaluated: (1) internal consistency, expressed as Cronbach’s alpha; (2) number and proportion of items not rateable; (3) discriminative ability, expressed as the area under the receiver operating characteristic (ROC) curve, was calculated per instrument item and for instrument sum scores, with 95% confidence intervals (95% CI). ROC curves allow for exploration of the relationship between the sensitivity and specificity of a continuous predictor (in this case the scores for each item on a scale) and a dichotomous outcome (in this case PD) [21]. Instruments with the largest area under the curve (AUC) are most accurate in distinguishing between patients with and without delirium.

Given the fact that PD is a serious condition requiring immediate intervention, sensitivity was given more weight than specificity in determining the optimal cutoff scores for PD. Cutoff scores for optimal sensitivity can be inferred from the curve. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for the diagnosis of delirium were calculated, as well as the likelihood ratio for a positive test result (the ratio between true and false positive rate).

Results

Population characteristics

From November 2006 to February 2010, 182 acute, non-electively admitted patients were included. Twenty-eight patients without a diagnosis of delirium were excluded because of deep sedation or coma on the assessment day selected for the analysis. Of the remaining 154 patients, 26 (16.9%) were diagnosed with delirium. Mean age in PD patients was 7.2 years [SD (standard deviation) = 5.4] compared to 6.7 years (SD = 5.0) in non-PD patients. Two routinely used illness severity indicators are the pediatric index of mortality (PIM) and the pediatric risk of mortality (PRISM II). Mean PIM score in the current sample was 8.8 (SD = 16.7) in patients with and 3.6 (SD = 5.6) in patients without delirium. Mean PRISM II score was 8.8 (SD = 19.2) in patients with and 3.6 (SD = 5.8) in patients without delirium. Mean length of stay in the PD group was 7.4 days (SD = 4.9) compared to 3.9 days (SD = 3.3) in the non-PD group. In both groups, the most common primary PICU indication was a respiratory disorder, followed by neurological disorders (Table 1).

Table 1

Population characteristics (N = 154)

Characteristics	Patients with delirium (N = 26)		Patients without delirium (N = 128)
	N	%	N	%
Male	14	53.8	81	63.3
Female	12	46.2	47	36.7
Mechanical ventilation	14	53.8	15	11.7
Sedation	15	57.7	13	10.2
Primary PICU indication
Respiratory disorder	8	30.8	42	32.8
Neurological disorder	7	26.9	29	22.7
Circulatory disorder	6	23.1	20	15.6
Metabolic disorder	0	0	7	5.5
Surgicala	3	11.5	10	7.8
Multipleb	1	3.8	11	8.6
Others	1	3.8	9	7.0
Retrospective data	3	11.5	31	24.2

PICU pediatric intensive care unit

aPost-surgical interventions

bMultiple medical indications for PICU admission

Sum score analysis

Of 154 patients, the PAED could be rated, applying the a priori criteria for rateability specified above, in 144 (93.5%), the DRS-88 in 103 (66.9%), and the DRS-R-98 in 73 (46.8%). For the DRS-88, the mean age of rateable (8.0 years) and non-rateable patients (4.3 years) differed significantly from each other (F (1,152) = 20.39; SD = 5.05; p < 0.001). This also held for the DRS-R-98 (rateable patients 9.6 years, non-rateable patients 4.3 years, F (1,152) = 57.68; SD = 5.05; p < 0.001). For the PAED, the difference between rateable and non-rateable groups was in the same direction albeit smaller and not significant (6.9 and 4.9 years respectively; F (1,152) = 1.45; SD = 5.05; p > 0.05). Owing to non-rateability, the PAED missed 3, the DRS-88 12, and the DRS-R-98 22 of the 26 diagnoses of delirium made by the gold standard.

Of the 144 cases in which the PAED could be assessed, 23 were diagnosed with delirium by the gold standard, and 21 of these were correctly identified by the PAED (Table 2). The PPV and NPV were 91.3 and 98.3%, respectively. The LR for a positive diagnosis was 55.2 and the AUC was 0.99 (Fig. 1). The optimum cutoff score was 8 (sensitivity = 100%; specificity = 92.6%). Secondly, out of 103 cases in which the DRS-88 could be assessed, 12 were diagnosed with delirium by the gold standard. Eleven of these patients were correctly identified by the DRS-88. The PPV and NPV were 98.9 and 100%, respectively (Table 2). Finally, out of 73 cases in which the DRS-R-98 could be assessed, 4 were diagnosed with delirium by the gold standard. The DRS-R-98 identified 3. The PPV and NPV were 100 and 98.6%, respectively (Table 2).

Table 2

Patients diagnosed with delirium

	Patients (N)	FN (N)	Sensitivity (%)	FP (N)	Specificity (%)
Pediatric neuropsychiatrist diagnosis	26	–	100	–	100
Hyperactive delirium	18
Hypoactive delirium	4
Mixed delirium	4
PAED outcome, delirium	21	2	91.3	2	98.3
Hyperactive delirium	15	1		1
Hypoactive delirium	3	1
Mixed delirium	3			1
DRS-88 outcome, delirium	11	1	91.7	0	100
Hyperactive delirium	5	1
Hypoactive delirium	4
Mixed delirium	2
DRS-R-98 outcome, delirium	3	1	75.0	0	100
Hyperactive delirium	3	1
Hypoactive delirium	0
Mixed delirium	0

FN false negatives, FP false positives

Fig. 1

ROC curve for PAED. The 15th dot from the right is at the PAED value of 8 and higher

Per-item analysis

Cronbach’s alpha was 0.89 for the PAED, 0.69 for the DRS-88, and 0.57 for the DRS-R-98. All items of the PAED had a low proportion of missing values and high discriminative ability, as evidenced by a high AUC (Table 3). The cognitive items of both the DRS-88 and DRS-R-98 had the highest proportion of missing values. Additional data regarding the item characteristics of DRS-88 and DRS-R-98 are shown in Tables 4 and 5. However, in cases in which these items could be rated, they demonstrated high discriminative ability, with the exception of two items: delusions (both scales) and long-term memory (DRS-R-98).

Table 3

Item characteristics for PAED (N = 154)

PAED	Missing values		AUC
	N	%		95% CI
Eye contact	15	9.7	0.93	0.89–0.98
Goal directedness of movements	25	16.2	0.93	0.86–1.00
Awareness of surroundings	19	12.3	0.90	0.83–0.98
Restlessness	1	0.6	0.97	0.95–0.99
Inconsolability	7	4.5	0.90	0.84–0.96

Table 4

Item characteristics for DRS-88 (N = 154)

DRS-88	Missing		AUC
	N	%		95% CI
Cognition
Perceptual disturbances	72	46.8	0.85	0.67–1.00
Hallucination type	70	45.5	0.86	0.68–1.00
Delusions	80	51.9	0.63	0.38–0.87
Cognitive status during formal testing	72	46.8	0.93	0.80–1.00
Affect
Lability of mood	54	35.1	0.76	0.58–0.94
DSM-IV criteria
Temporal onset of symptoms	12	8.4	0.93	0.87–0.99
Variability of symptoms	18	11.7	0.93	0.86–1.00
Physical disorder	0	0.0	–a	–a
Sleep-wake cycle
Sleep-wake cycle disturbance	29	18.8	0.83	0.71–0.96
Behavior
Psychomotor behavior	9	5.8	0.94	0.90–0.97

aPhysical disorder was always scored as ‘severe’ because all patients at the PICU are critically ill. Therefore, the AUC and CI of this item could not be estimated

Table 5

Item characteristics for DRS-R-98 (N = 154)

DRS-R-98	Missing		AUC
	N	%		95% CI
Cognition
Perceptual disturbances and hallucinations	70	45.5	0.82	0.62–1.00
Delusions	80	51.9	0.63	0.38–0.87
Orientation	68	44.2	0.98	0.97–1.00
Attention	56	36.4	0.93	0.81–1.00
Visuospatial ability	112	72.7	–a
Language	76	49.4	0.81	0.60–1.00
Thought process abnormalities	83	53.9	1.0	1.00–1.00
Short-term memory	88	57.1	1.0	–b
Long-term memory	100	64.9	0.49	–b
Affect
Lability of affect	54	35.1	0.76	0.58–0.95
Sleep-wake cycle
Sleep-wake cycle disturbance	29	18.8	0.83	0.71–0.96
Behavior
Motor agitation	5	3.2	0.93	0.88–0.98
Motor retardation	19	12.3	0.49	0.42–0.57

aVisuospatial ability was either scored as being normal or unrateable (due to sedation, severity of illness, and young age). Therefore, the AUC and CI of this item could not be estimated

bShort-term and long-term memory were scored as being normal or unrateable (due to sedation, severity of illness, and young age) and rarely as affected. Therefore, their standard error (SE) and thus their CI could not be estimated

Discussion

To our knowledge, this is the first prospective study evaluating the utility of the DRS and the PAED as assessment tools for PD in a PICU setting. The PAED was rateable most often (93.5%) and proved to be excellent in discriminating between patients with and without PD (AUC = 0.99). This is in line with the findings of two recent studies which both found an AUC of 0.98 for the PAED in detecting ED in a post-anesthesia setting [22, 23]. It seems, therefore, that the PAED can be used in both settings. Both versions of the DRS could not be rated as frequently as the PAED, as these scales contain numerous items that are difficult to assess in young children, especially regarding cognition. As the largest part of the PICU population consists of children under the age of 3, this is a severe drawback [7]. In addition, these tools require substantially more time to score. Taken together, in this study, the DRS scales were found to be far less suitable for routine PD assessment at a PICU. However, the PAED is not flawless: there were two false negatives (1 hypoactive and 1 hyperactive) that were probably due to the fluctuating nature and/or the subtle presentation of delirium and two false positives (1 hyperactive and 1 mixed). In these cases, it subsequently appeared that the presentations (emotional and/or behavioral disturbances) were better explained by other somatic complications.

Taken together, the strengths of this study are (1) it was executed during daily regular clinical care, and (2) it also included the (near) majority of the PICU children, namely those in the age range greater than 1 and less than 5 years. This broad inclusion could explain the incidence of PD (16.9%) in our study compared to the 13% found by Smith and colleagues [13] who only included children above the age of 5 years. However, because the pCAM-ICU is relatively new we did not have sufficient data points to evaluate its use.

Some limitations are also apparent. Firstly, data for 40 of 182 patients (22%) were assessed in retrospect, because patients were not observed during weekends or holidays. In these cases, we had to rely on collateral information from parents, nurses, intensivists, and child neurologists to reconstruct ratings. However, there were no significant differences between the results with or without retrospective data (data not shown). Second, given the fluctuating nature of delirium, it could be that some cases were missed. Thirdly, only one observer rated the PAED each day. Because the PAED contains an element of subjectivity, depending on what a given observer defines as ‘a little bit’ or ‘extreme’ in rating a given symptom, inflated random error may have been the result. More objective anchors should be developed for the PAED to overcome this problem. Unfortunately, reliability analysis was not possible given that the PICU staff over the study period was not yet participating as rater of the different instruments—this is the topic of a current study. Fourth, this is one of the first psychometric studies conducted in a PICU setting [24] where a higher proportion of missing data can be expected because assessments are complicated by factors such as severity of critical illness, young age, mechanical ventilation, sedation, and limited amount of time available for assessments. Because listwise deletion would lead to too much data loss we used a conservative strategy and deleted subjects for which less than 60% of the items were scored for a specific scale. Missing data were corrected using a weighted sum procedure. Fifth, the findings suggest that the incidence of delirium is much lower than in adults, especially with respect to the hypoactive subtype. The difficulties with the assessment of delirium in young and/or non-verbal patients could be an explanation for the relatively low rate of PD; in addition, the exclusive reliance of the PAED on subtle behavioral alterations may explain the relatively low rate of hypoactive delirium. Finally, replications of the current findings are necessary as it remains to be seen whether results can be generalized to other PICU settings. A sufficient level of involvement of child psychiatrists and their trainees is therefore recommended.

Conclusion

Given (1) the potentially dangerous nature of delirium, (2) the high workload at the PICU, (3) a shortage of pediatric neuropsychiatrists, and (4) the fact that 75% of the children at the PICU are below the age of 3 years, an instrument is needed that is easy to use at the bedside, not time-consuming, and suitable for young children as well. The PAED seems to fulfill these requirements. It is therefore concluded that the PAED should be used for PD screening at the PICU, preferably multiple times per day. Whenever a score of 8 or higher (or other scores, depending on the patient mix of the PICU in question) is obtained for a patient, a pediatric neuropsychiatrist should be consulted to assess whether the child is delirious or not. In this way, the likelihood that PD will be missed may be reduced. Future studies could be directed towards the development of an improved screening instrument for PD, based on the PAED.