Psychometric Properties of Nursing Delirium Screening Scale in Patients Admitted to Intensive Care Units.

BACKGROUND: Nursing Delirium Screening Scale (Nu-DESC) is a new instrument for determining delirium by nurses. The study aimed to investigate the psychometric properties of Nu-DESC and determined the sensitivity and specificity of it.
METHODS: Two evaluators assessed delirium by Nu-DESC in nonintubated patients admitted to intensive care unit (ICU) wards of Ardabil educational and medical centers. For determining psychometric properties of the instrument, the methods of determining content validity, structural validity, criterion validity (the DSM-5 criteria was used as a standard tool), internal consistency, and inter-rater reliability were used.
RESULTS: Ninety-six participants were assessed two times using the Nu-DESC. The mean age of the participants was 58.84, and 51 (53.1%) of them were male. Due to the high correlation of the Nu-DESC with the study criterion (DSM-5), the criterion validity of the instrument is confirmed. By using DSM-5 instrument, the cutoff score of 2 shows the best sensitivity and specificity. The kappa and alpha coefficients were obtained as r = 0.96 and α = 0.86, which indicate a good agreement between the evaluators and acceptable consistency.
CONCLUSION: Nu-DESC can be used as an efficient and reliable instrument by nurses in the ICU. It was also found that taking medical history can help nurses to better interpret the Nu-DESC score at diagnosing delirium. HOW TO CITE THIS ARTICLE: Amirajam Z, Noran EA, Molaei B, Adiban V, Heidarzadeh M, Darghah MH. Psychometric Properties of Nursing Delirium Screening Scale in Patients Admitted to Intensive Care Units. Indian J Crit Care Med 2021;25(6):680-684.

Introduction

Delirium is a fluctuating state of confusion and misperception characterized by impaired consciousness, attention, cognition, memory, perception, and speech.[1] The prevalence of delirium in patients admitted to general wards is 14 to 24%, in the elderly undergoing surgery is 15 to 35%, and in patients admitted to intensive care units (ICUs) is 70 to 87%.[2] Delirium is associated with several complications, including an increase in mortality at 6 months, an increase in hospital stay, an increase in the incidence of cognitive disorders, and a decrease in life expectancy after discharge.[3]

Due to the nature of delirium and its many complications, the diagnosis of delirium is very important, especially in ICUs; however, its diagnosis is difficult and time-consuming. On the contrary, because this disease is associated with many complaints and symptoms of psychiatric disorders (dementia, psychosis, and depression), they may be easily confused.[4] The diagnosis of delirium with DSM-5 is usually made using clinical interviews and examinations, but this method is relatively time-consuming and requires specialized specialists who cannot be accessed at any time.[5] To address this, several tools have been developed as a symptom checklist for delirium diagnosis, including the Nurse Delirium Screening Scale (Nu-DESC), developed in 2006 by Gaudreau et al. Nu-DESC is an observational instrument for delirium that examines five items: Disorientation, inappropriate behaviors, inappropriate communication, illusion or hallucinations, and psychomotor retardation. Each item has a score between 0 and 2 (total score range: 0–10). One of the features of Nu-DESC scale is its easy and fast use for nurses in diagnosing delirium, which distinguishes it from other instruments.[6] In the United States, Hargrave et al. psychometrically evaluated the Nu-DESC and showed that Nu-DESC is a proprietary instrument for the diagnosis of delirium.[7] Other studies have confirmed the psychometric properties of the Nu-DESC instrument. However, the important point in these studies was the disagreement on the cutoff point of this tool, so that the cutoff point in these studies ranged from 1 to 3.[6,8,9] Since Nu-DESC is a new tool and its psychometric properties have not been approved in many societies, including Iranian society, the present study aimed to investigate the psychometric properties of Nu-DESC and determined the sensitivity and specificity of this tool.

Materials and Methods

The present study is a methodological study that investigates the psychometric properties of Nu-DESC in nonintubated patients admitted to ICU wards of Ardabil educational and medical centers. Inclusion criteria included hospitalization in the ICU, having at least 18 years of age, no intubation of the patient, no speech disorder, and consent to participate in the study. If there was a history of cognitive disorders (such as dementia and Alzheimer's) and psychosis, they would be excluded from the study based on their history and medical records. There were a total of 63 ICU beds in three educational and medical centers affiliated to Ardabil University of Medical Sciences, which were sampled from February to July 2018. A total of 112 patients were eligible; of which 11 patients were excluded due to dissatisfaction with the study, two due to deterioration during the study, and three due to a history of psychosis and dementia; Finally, the data obtained from 96 eligible patients were reviewed. The sample size calculator (PASS) showed that the minimum of 18 delirious patients from a total of 90 patients (72 nondelirious patients) gives a power of 0.8 and Type I error of 0.05.

Instrument

In this study, in addition to questions related to demographic characteristics, Nu-DESC was also completed by the evaluators according to the status of the participants. The DSM-5 criteria were also used as a standard tool. DSM-5 is a classification of mental disorders that facilitates the diagnosis of mental illness by providing accurate and relevant diagnostic criteria.[2]

Procedure

After obtaining permission from the Nu-DESC tool designer,[6] the translation to Persian and back translation of the scale was done. To translate the Nu-DESC, the steps introduced by Wild et al. were used,[10] so that Nu-DESC was translated into Persian by two fluent English speakers. Then, by comparing the two translations, the ambiguities were revised and corrected, and the original Persian version was prepared. Then, the translation of the tool into English was done by a third person familiar with English, and the translation was sent to the tool designers and was approved by them. Then, in order to determine the psychometric properties of the instrument, the methods of determining content validity, structural validity, criterion validity, internal consistency, and inter-rater reliability were used. Content validity index (CVI) was used to perform content validity; for this purpose, 10 experts, including four intensive care nurses, two nursing faculty members, two anesthesiologists, and two clinical psychologists, were asked to indicate the relevance, clarity, and simplicity of each of the Nu-DESC items and mark them with a score of 1 to 4, so that higher scores indicate more relevance, clarity, and simplicity. To obtain the CVI, the percentage of those who gave a score of 3 or 4 for each of the options of relevance, clarity, and simplicity was calculated.[11] Considering the CVI of more than 93.3% in all five items, the content validity of the Nu-DESC was confirmed.

To determine the construct validity of the instrument, since the instrument has five items in the form of one dimension, confirmatory factor analysis was used to confirm or reject the single-factor Nu-DESC. To determine the criterion validity, DSM-5 was used as a criterion. It was assumed that if the Nu-DESC instrument had a correlation of more than 0.7 with the standard instrument, the validity of the criterion would be confirmed.

After selecting an eligible patient, at this stage, delirium evaluators use the DSM-5 criteria (including two anesthesiologists and a clinical psychologist in the three ICU wards understudy, who are typically responsible for delirium testing in those wards) to check for the presence or absence of delirium. Then, without knowing the results of previous evaluations, the first evaluator (main evaluator) examined delirium using Nu-DESC. At this stage, to check the reliability between the evaluators, the second evaluator simultaneously but separately (without knowing how the main evaluator scored) performed delirium analysis using Nu-DESC. Delirium analysis was repeated by the mentioned evaluators for the second time at an interval of 8 to 12 hours. These two evaluators were trained by the psychologist to use the Nu-DESC in a 2-hour session, in which delirium was explained and discussed.

DSM-5 instrument was used as gold standards; and sensitivity, specificity, positive predictive value, and negative predictive value were calculated. Due to the fact that the participants in this study, based on the test criterion, included in two groups with delirium disorder and without delirium disorder, the scores obtained from Nu-DESC in the participants were analyzed using the receiver operating characteristic curve to determine sensitivity and specificity. Descriptive tests (mean, median, frequency, percentage, and standard deviation) and inferential tests (correlation tests and structural equation modeling) were performed using SPSS software version 22 and LISREL version 8.8.

Ethical Considerations

After obtaining permission from the ethics committee of Ardabil University of Medical Sciences (ethics code: IR.ARUMS.REC.1397.171), people who met the inclusion criteria were selected. Demographic characteristics of patients were completed according to the form. The objectives of the study, the procedure, and the optionality of participating in the study were explained to patients or their companions, and then informed consent was obtained from them.

Results

According to the inclusion and exclusion criteria, a total of 96 patients were evaluated with Nu-DESC and DSM-5. The mean age of the participants was 58.84, and their age range was between 18 and 87 years. Also, 51 were male (53.1%) and 45 were female (46.9%). The reasons for the hospitalization of patients in ICUs were Cardiovascular problems (40.6%), pulmonary problems (21.9%), gastrointestinal problems (13.5%), drug poisoning (7.3%), and other diseases (16.7%).

Determining the Criterion Validity

By setting the DSM-5 instrument as the gold standard for detecting the presence or absence of delirium (zero means no delirium and one means delirium), the correlation of Nu-DESC scores (assessing by the main evaluator) using the Eta test was investigated. Due to the high correlation of the Nu-DESC with the study criterion (correlation coefficient = 0.79), the criterion validity of the instrument is confirmed.

Sensitivity, specificity, positive predictive value, and negative predictive value tests were performed in accordance with the DSM-5 criterion. The results of the study showed that the cutoff score of 2 shows the best sensitivity and specificity (Table 1).

Table 1

Sensitivity, specificity, positive predictive value, and negative predictive value of Nu-DESC* considering DSM-5 as instrument criterion

Gold Standard	Cutoff point	True negative	False positive	False negative	True positive	Sensitivity	Specificity	Positive predictive value	Negative predictive value
DSM-5	Score 1	62	13	0	21	100	82.7	0.62	0.100
	Score 2	69	6	1	20	92	95.2	0.77	0.985

The table above shows that despite the suitability of cutoff points 1 and 2, cutoff point 2 for Nu-DESC tools has better sensitivity, specificity, and positive predictive value than score 1

To determine the construct validity, the 5-item Nu-DESC single-factor structure was examined using confirmatory factor analysis. For this purpose, the data of the main evaluator (n = 96) were analyzed. The results showed that the ratio of chi-square-to-the-degree of freedom in the one-factor model is 5.24, which indicates the rejection of the one-factor model. On the contrary, the high root-mean-square error (RMSEA = 0.21) is further evidence of model rejection. “A model fits well when the RMSEA is close to 0.06 or lower,” Brown states.[12] Also, other fit indices, including comparative fit index (0.93), normed fit index (0.92), non-normed fit index (0.87), relative fit index (0.85), goodness-of-fit index (0.90), and standardized root-mean-square residual (0.075), showed that the 5-item single-factor model of the Nu-DESC tool has a poor fit.

To evaluate the reliability of the evaluators, the agreement coefficient (kappa coefficient) between the two evaluators was calculated. For this purpose, the agreement coefficient of the two evaluators was examined in both the first and second evaluations, and the kappa coefficients were obtained as r = 0.96 and r = 0.92, respectively, which indicate a good agreement between the evaluators.

To calculate Cronbach's alpha (internal consistency) and item analysis, participants were divided into three groups: Participants with delirium diagnosis based on DSM-5 (n = 21), participants with no delirium diagnosis based on DSM-5 (n = 75), and total participants (n = 96). Given that Nu-DESC is considered as a single-factor model, Cronbach's alpha coefficient for all five items was calculated in the form of a factor and in accordance with the above classification. Internal consistency results were obtained for participants with delirium diagnosis (α = 0.65), participants with no delirium diagnosis (α = 0.78), and total participants (α = 0.86). Then, for further item analysis, the mean and standard deviation of the total score and the scores of the items of the Nu-DESC were performed based on the classification of the participants. The results of this section showed that items 4 and 5 in patients with delirium do not increase in proportion to other items (Table 2).

Table 2

Nu-DESC item scores based on participant classification

Items of Nu-DESC***	Total participants (n = 96)				Participants with delirium diagnosis based on DSM-5 (n = 21)				Participants with no delirium diagnosis based on DSM-5 (n = 75)
	Frequency (%) of every score				Frequency (%) of every score				Frequency (%) of every score
	0	1	2	M (SD)**	0	1	2	M (SD)	0	1	2	M (SD)
N1*	67 (69.8)	18 (18.8)	11 (11.5)	0.42 (0.69)	1 (4.8)	11 (52.4)	9 (42.9)	1.38 (0.59)	66 (88)	7 (9.3)	2 (2.7)	0.15 (0.42)
N2*	78 (81.3)	10 (10.4)	8 (8.3)	0.27 (0.61)	6 (28.6)	7 (33.3)	8 (38.1)	1.10 (0.83)	72 (96)	3 (4)	0 (0)	0.04 (0.20)
N3*	73 (76)	14 (14.6)	9 (9.4)	0.33 (0.64)	1 (4.8)	13 (61.9)	7 (33.3)	1.29 (0.56)	72 (96)	1 (1.3)	2 (2.7)	0.07 (0.34)
N4*	87 (90.6)	7 (7.3)	2 (2.1)	0.11 (0.38)	13 (61.9)	6 (28.6)	2 (9.5)	0.48 (0.68)	74 (98.7)	1 (1.3)	0 (0)	0.01 (0.11)
N5*	81 (84.4)	13 (13.5)	2 (2.1)	0.18 (0.44)	12 (57.1)	9 (42.9)	0 (0)	0.43 (0.51)	69 (92)	4 (5.3)	2 (2.7)	0.11 (0.39)
Total score Nu-DESC	M (SD) 1.31 (2.36)				M (SD) 4.67 (2.08)				M (SD) 0.37 (1.15)

N1: Disorientation, N2: Inappropriate behavior, N3: Inappropriate communication, N4: Illusions or Hallucination, N5: Psychomotor retardation

M (SD), mean (standard deviation)

Nurse Delirium Screening Scale

The table above shows items 4 and 5 in patients with delirium do not increase in proportion to other items

Discussion

The present study used content validity, criterion validity, construct validity, inter-rater reliability, internal consistency, and item analysis to investigate the psychometric properties of Nu-DESC in Iranian participants. In this study, DSM-5 instrument as the most important criterion for detecting delirium has been the criterion of the present study. The results showed that there is a high correlation between the scores obtained from the Nu-DESC instrument and its criterion (DSM-5), and this confirms the criterion validity of the Nu-DESC. There are few studies that have examined the psychometric properties of Nu-DESC, and few studies have confirmed the Nu-DESC criterion validity. For example, Abelha et al. in their study in Portugal considered the Intensive Care Delirium Screening Checklist (ICDSC) as the gold standard and showed that there is a high correlation between Nu-DESC and ICDSC.[13] Also, in the study of Hargrave et al., it was found that there is a high correlation between Nu-DESC and DSM-5 (as the gold standard).[7]

The results of the study showed that score 1 and score 2 are suitable cutoff points for Nu-DESC. However, by using DSM-5, the best cutoff point for Nu-DESC is score 2, at which point in overall the sensitivity, specificity, and positive and negative predictive values are higher than score 1. Studies to determine the cutoff point in different societies have obtained different numbers. For example, in the study of Abelha et al. in Portuguese society, the cutoff point 2 with 100% sensitivity and 86% specificity, 57% positive predictive value, and 100% negative predictive value was the best.[13] However, in the study of Hargrave et al., cutoff point 1 had the best sensitivity and specificity.[7] Moreover, in a study conducted by Spedale et al. in Italy, which considered the confusion assessment method instrument as a test criterion, the cutoff point 3 showed the best sensitivity and specificity.[9] Confirmation of score 1 as the Nu-DESC cutoff point in the present study and the study of Hargrave et al.[7] confirms some points. This indicates that for a person admitted to the ICU, even if only one of the 5 Nu-DESC items is positive, the probability of delirium is raised to a significant percentage. This can have positive or negative points for the tool: First, it can help nurses diagnose delirium faster in ICU patients, because nurses, as soon as they see one of the positive (even mild) items in Nu-DESC, can strongly consider the presence of delirium in the patient and plan to treat it or prevent its symptoms from worsening in patients. Another issue is that any of the symptoms listed on the Nu-DESC can have several causes other than delirium, including the impaired level of consciousness in patients admitted to ICUs for reasons other than delirium, such as encephalopathy or encephalitis, traumatic or ischemic brain injuries, thyroid diseases, etc.[14] This means that although nurses can quickly detect delirium at the first sign of the symptoms, they cannot differentiate between delirium and other causes of these symptoms; and nurses using the Nu-DESC need to be able to cover this weakness by taking an accurate and continuous history of the patients.

To determine the construct validity of the instrument, the single-factor structure of Nu-DESC was examined using confirmatory factor analysis. The results showed that due to the weakness of the goodness-of-fit indicator, the one-dimensional model is rejected. This means that the five items of the tool are not approved in the form of one factor, and the items do not have the appropriate compatibility with each other to fit in one factor. To further investigate this issue, the internal consistency of five items was calculated by examining Cronbach's alpha coefficient in three modes (total participants, participants with delirium, and participants without delirium). Cronbach's alpha coefficient in participants with delirium was lower than normal (α = 0.65). Although a part of the reason for this is related to the smaller sample size in participants with delirium (n = 21), it seems that one of the main reasons is related to the heterogeneity of items, which causes that the five items do not have a good correlation. For example, the analysis of items in this field shows that the mean score of items 4 and 5 (with a score range of 0.43–0.48) is not equal to items 1, 2, and 3 (with a score range of 1.3–1.4). In addition, these items do not show any increase (equivalent to zero) in 60% of patients with delirium. Overall, the results of this part of the study showed that the items of Nu-DESC do not have a good correlation with each other and cannot be put together in one dimension. There were no similar studies to examine the construct validity of the Nu-DESC using the structural equation modeling method.

To determine the inter-rater reliability in Nu-DESC, the correlation between the evaluators was calculated and the desired results were obtained. Other studies in this field have mainly confirmed the reliability between evaluators.[8,9,13] For example, Leung in his study in China, showed that the reliability coefficient between the two evaluators in the evaluation of the elderly admitted to ICUs is good (r = 0.94).[8] Spedale in Italy obtained reliability between evaluators in Nu-DESC as r = 0.87.[9] According to the results of the present study on the reliability of evaluators, it can be said that nurses’ perception of the symptoms of delirium in the Nu-DESC is the same and there is no different interpretation.

Limitations

In this study, nonintubated patients admitted to ICUs were studied and patients with cognitive disorders (such as dementia and Alzheimer's) and psychosis were excluded from the study. Therefore, the results cannot be generalized to this population, and consequently more studies are needed to evaluate pain in these patients.

Conclusion

Based on the results of the present study, Nu-DESC has a high correlation, sensitivity, and specificity for the diagnosis of delirium by nurses and can be used as an efficient and reliable tool by nurses in the ICU. It was also found that Nu-DESC at cutoff point 2 could distinguish delirium cases from healthy individuals with high positive and negative predictive values. The important point in this study is to use the Nu-DESC tool to diagnose delirium to obtain an accurate history of the patient's underlying disease status to help interpret the results of the Nu-DESC. It is also suggested to examine the validity of Nu-DESC in patients with underlying cognitive disorders, such as dementia, Alzheimer's, etc.

Acknowledgment

This study was supported and approved by the Ethics Committee of Ardabil University of Medical Sciences (ethics code: IR.ARUMS.REC.1397.171). The authors would like to thank all the patients, their families, the personnel of ICUs, and the Vice Presidency of Research Ardabil University of Medical Sciences, and all who helped us in doing this study.

Orcid

Zohreh Amirajam https://orcid.org/0000-0002-4930-1301

Elham A Noran https://orcid.org/0000-0002-3270-4593

Behnam Molaei https://orcid.org/0000-0002-9859-6521

Vahid Adiban https://orcid.org/0000-0002-9531-9369

Mehdi Heidarzadeh https://orcid.org/0000-0003-4302-8882

Mohammad H Darghah https://orcid.org/0000-0002-2145-9027