Validity and Reliability of the Memorial Delirium Assessment Scale-Thai Version (MDAS-T) for Assessment of Delirium in Palliative Care Patients.

BACKGROUND: Delirium, a neuropsychiatric syndrome that occurs throughout medical illness trajectories, is frequently misdiagnosed. The Memorial Delirium Assessment Scale (MDAS) is a commonly used tool in palliative care (PC) settings. Our objective was to establish and validate the Memorial Delirium Assessment Scale-Thai version (MDAS-T) in PC patients.
MATERIALS AND METHODS: The MDAS was translated into Thai. Content validity, inter-rater reliability, and internal consistency were explored. The construct validity of the MDAS-T was analyzed using exploratory factor analysis. Instrument testing of the MDAS-T, the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU-T), and the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition as the gold standard was performed. The receiver operating characteristic (ROC) curve was used to determine the optimal cutoff score. The duration of each assessment was recorded.
RESULTS: The study enrolled 194 patients. The content validity index was 0.97. The intraclass correlation coefficient and Cronbach's α coefficient were 0.98 and 0.96, respectively. A principal component analysis indicated a homogeneous, one-factor structure. The area under the ROC curve was 0.96 (95% confidence interval [CI], 0.93-0.99). The best combination of sensitivity and specificity (95% CI) of the MDAS-T were 0.92 (0.85-0.96) and 0.90 (0.82-0.94), respectively, with a cutoff score of 9, whereas the CAM-ICU-T yielded 0.58 (0.48-0.67) and 0.98 (0.93-0.99), respectively. The median MDAS-T assessment time was 5 minutes.
CONCLUSION: This study established and validated the MDAS-T as a good and feasible tool for delirium screening and severity rating in PC settings. IMPLICATIONS FOR PRACTICE: Delirium is prevalent in palliative care (PC) settings and causes distress to patients and families, thereby making delirium screening necessary. This study found that the MDAS-T is a highly objective and feasible test for delirium screening and severity monitoring in PC settings and can greatly improve the quality of care for this population.

Introduction

Delirium is a common neuropsychiatric syndrome 1, 2 affecting patients’ behavior 3, 4 that can occur throughout medical illness trajectories 2. It is characterized by fluctuating and altered levels of awareness, memory, psychomotor activity, and sleep‐wake cycle disturbances 5. Delirium is prevalent, especially in palliative care (PC) and intensive care unit (ICU) settings, varying from 13%–88% and 7%–50%, respectively 6, 7, 8. It is associated with increased mortality rates, shorter survival times, and increased symptom distress scores 9, 10. Moreover, the majority of the patients can recall when they were delirious and report high levels of distress 3, 4. Therefore, early detection of delirium is crucial. Unfortunately, it is frequently overlooked and misdiagnosed 11, 12. This may be due to a lack of knowledge and awareness of delirium 13, 14.

Although the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM‐5) is the gold standard for the diagnosis of delirium 15, it cannot be easily used by nonpsychiatrist personnel 16. Also, delirium symptoms fluctuate continually, and severity monitoring is essential in palliative care settings. Therefore, having an instrument that has high objectivity, can be easily assessed, and can monitor delirium severity is a necessity 17. The Memorial Delirium Assessment Scale (MDAS) is a commonly used tool in PC settings, and it has been well validated 18. To our knowledge, however, it has not been translated into Thai, and no psychometric properties have been reported. The Confusion Assessment Method for the Intensive Care Unit (CAM‐ICU) is another widely used diagnostic tool with good sensitivity and specificity in ICU settings 19, 20; still, it has not been tested on palliative care patients.

Our primary objective was to establish and validate the Memorial Delirium Assessment Scale‐Thai version (MDAS‐T). Our secondary objective was to compare the psychometric properties of the MDAS‐T, the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM‐ICU‐T), and DSM‐5 as the gold standard.

Materials and Methods

The protocol was approved by institution review board of the Faculty of Medicine Siriraj Hospital, Mahidol University. The participants were newly consulted palliative care patients, either outpatient or inpatient, 18 years of age or older, and with a diagnosis of advanced cancer or major organ failure. Excluded were patients with a score of 7 or more for any single item in the Edmonton Symptom Assessment System (ESAS). Patients with a diagnosis of dementia, mental retardation, coma, or communication difficulty caused by an endotracheal intubation or a tracheostomy that had been confirmed by a psychiatrist were also excluded. The patients or their proxy gave signed informed consent.

The MDAS was translated into Thai by two researchers (a clinical psychologist and a palliative care physician). A backward translation was subsequently made by another researcher, a geriatrician, and that version was then compared with the original MDAS by the two researchers. The MDAS‐T was next tested for content validity by three experts (a psychiatrist, an anesthesiologist, and another palliative care physician). Following that, the first two researchers assessed the inter‐rater reliability using the first 30 participants.

We enrolled 194 participants for instrument testing. A clinical psychologist first assessed the patients using the MDAS‐T, which was followed by a research assistant administering the CAM‐ICU‐T. Finally, a psychiatrist used the DSM‐5 to undertake a delirium diagnosis of all 194 patients. All assessors were blinded to each other. They assessed each patient individually, and nearly immediately after each other. An independent research assistant subsequently collected all of the data. The time taken for each assessment was recorded.

Instruments

The MDAS is a 10‐item, delirium assessment tool. It uses a 4‐point rating scale (0–3) that is based on the current interactions with a patient as well as the interactions over the preceding several hours. It is also able to prorate scores in the event that some of the 10 MDAS items cannot be administered. It has good sensitivity and specificity, and it can be administered in approximately 10 minutes 18, 21.

The CAM‐ICU is a well‐validated diagnostic tool for delirium in the ICU. It has become widely used because it uses more objective criteria than the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; has high sensitivity and specificity 20; can be easily used by nonpsychiatrists; and saves much time 22. It has been translated into a Thai version, which is now in common use in ICUs in Thailand 19.

Statistical Analysis

We calculated the required sample size for internal consistency and sensitivity of the MDAS‐T before the study's inception. For internal consistency, the desired sample size was determined using the formula of Bonett 23. A sample of 51 participants each responding to 10‐item questionnaire would achieve 90% power to detect the difference between Cronbach's α under the null hypothesis of 0.80 and Cronbach's α under the alternative hypothesis of 0.90, based on Breitbart et al. 18, by using a two‐sided F‐test with a significance level of 0.05. The sample size required for the sensitivity estimate was 194 participants, given a sensitivity based on Breitbart et al. 18 of 65%, with ±0.095 error, 95% confidence level, and a delirium prevalence of patients in Siriraj Palliative Care Center of 50%. Because there is no definite rule for sample sizes in latent profile analyses, Wurpts and Geiser 24 recommended that a sample size well into the hundreds be used.

Data were recorded and analyzed using PASW Statistics for Windows, version 18.0 (SPSS Inc., Chicago, IL). Descriptive statistics were used to describe the demographic and clinical characteristics. The quantitative data have been described as mean and SD or median and interquartile range, as appropriate. Qualitative data were expressed as number and percentage.

The content validity of the MDAS‐T was examined using the content validity index. The intraclass correlation coefficient was used to assess the inter‐rater reliability of the two raters, and the internal consistency of the entire questionnaire was tested via Cronbach's α coefficient. Cronbach's α was also examined when individual items were deleted.

An exploratory factor analysis (EFA) was performed to evaluate the construct validity of the MDAS‐T. The suitability of the data for carrying out such an analysis was tested using Bartlett's test of sphericity, based on a criterion of p < .05. The Kaiser‐Meyer‐Olkin statistic was used to assess the sample adequacy, based on a criterion of ≥0.6. Principal component analysis was used to assess the main factors. The criterion of an eigenvalue >1 was used to determine the number of factors derived from the factor analysis.

A receiver operating characteristic curve analysis of the MDAS‐T score was performed to identify the most appropriate cutoff value for the judgment of delirium, using DSM‐5 as the gold standard. The sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratio, and accuracy were then calculated.

For the subgroup analysis of patients with an MDAS‐T score ≥9, a Gaussian finite mixture model by an expectation‐maximization algorithm called a latent profile analysis was used to classify the patients into groups, based on the MDAS‐T scores and using the R package mclust 25. Four indices were used to select the correct number of latent classes: log‐likelihood, Bayesian information criterion (BIC), integrated complete‐data likelihood, and bootstrap likelihood ratio test. Lower BIC and integrated complete‐data likelihood values coupled with higher log‐likelihood values indicate a better model fit. The bootstrap likelihood ratio test was used to compare the fit of the estimated model, where a significant change in −2 log‐likelihood indicated that the model with the larger number of classes provided a better fit to the data.

Results

We consecutively recruited 308 newly consulted patients between July 2017 and April 2018. In total, 42 patients were excluded from the study. Fifteen patients were excluded because of having a persistent symptom score of 7 or more in the Edmonton Symptom Assessment System. Of those, nine had pain, one had dyspnea, and five had depression. Twenty‐seven more patients were excluded: 15 patients had a diagnosis of coma, 8 had dementia, and 4 had communication difficulty due to an endotracheal intubation or a tracheostomy. The patients or caregivers refused in 57 cases, and we were not able to obtain the consent of 15 patients because of their altered state of consciousness and as their caregivers were unavailable at the time. The demographic data of the 194 patients who were finally enrolled in the study are detailed in Table 1.

Table 1

Patient characteristics

Characteristics	n = 194
Sex, male, n (%)	100 (51.5)
Age, mean ± SD, yr	63.9 ± 13.3
Palliative performance scale, median (IQR)	30 (20–40)
Level of education, n (%)
Primary school and below	87 (44.9)
High school	60 (30.9)
Bachelor's degree and above	47 (24.2)
Cancer diagnosis, n (%)	191 (98.5)
Cancer type, n (%)
Locally advanced	60 (31.4)
Metastatic	120 (62.9)
Recurrent	10 (5.2)
Relapsed	1 (0.5)
Delirium diagnosis, n (%)	99 (51.0)
Delirium type, n (%)
Hyperactive	8 (8.1)
Hypoactive	38 (38.4)
Mixed	53 (53.5)

Abbreviation: IQR, interquartile range.

The mean ± SD of the MDAS‐T total score was 11.98 ± 9.42. The calculated scale content validity index for the MDAS‐T was 0.97, whereas the item content validity index was 0.67 to 1. The inter‐rater reliability of the two‐raters of the MDAS‐T total score showed an intraclass correlation coefficient of 0.98 (95% confidence interval [CI], 0.96–0.99). As to the reliability analysis, Cronbach's α coefficient of the 10‐item MDAS‐T was 0.96. The corrected item‐total correlation and Cronbach's α coefficient if an item was deleted are detailed in Table 2.

Table 2

Reliability of Memorial Delirium Assessment Scale‐Thai version

MDAS‐T item	Mean ± SD	Corrected item‐total correlation	Cronbach's α if item deleted	Weighted κ
1	0.87 ± 1.12	0.91	0.95	0.98
2	1.45 ± 1.25	0.80	0.95	0.91
3	1.93 ± 1.09	0.69	0.96	0.78
4	1.77 ± 1.00	0.76	0.95	0.84
5	1.10 ± 1.20	0.89	0.95	0.91
6	0.88 ± 1.19	0.87	0.95	1.00
7	0.82 ± 1.15	0.80	0.95	0.74
8	0.54 ± 1.07	0.73	0.95	0.88
9	1.15 ± 1.11	0.88	0.95	0.91
10	1.47 ± 0.95	0.73	0.95	0.86

Abbreviation: MDAS‐T, Memorial Delirium Assessment Scale‐Thai version.

The construct validity was analyzed using an EFA. The suitability of data for the factor analysis was assessed. Several criteria for the factorability assessment were used. First, the correlation matrix revealed the presence of coefficients ranging from 0.44 to 0.89, which suggests appropriate factorability. In addition, the Kaiser‐Meyer‐Olkin statistical measure of sampling adequacy was 0.93, which exceeded the recommended value of 0.6. Moreover, Bartlett's test of sphericity achieved statistical significance (χ2 [45] = 1957.2; p < .001), which supported the factorability of the correlation matrix. Furthermore, all of the diagonals of the anti‐image correlation matrix exceeded 0.9, which supported the inclusion of each item in the factor analysis. Finally, the communalities were all above 0.6, as presented in Table 3; this confirmed that each item shared some common variances with other items. Based on the foregoing, the EFA was conducted on all 10 items of the MDAS‐T. The EFA of the MDAS‐T indicated a homogeneous, one‐factor structure with all items loading on one factor with an eigenvalue of 7.16; the presence of the one factor structure explains 81% of the variance.

Table 3

Factor loading and communality for the Memorial Delirium Assessment Scale‐Thai version items

Item	Scale item	Factor I	Communality
1	Reduced level of consciousness	0.93	0.86
2	Disorientation	0.84	0.70
3	Short‐term memory	0.74	0.55
4	Impaired digit span	0.80	0.65
5	Reduced ability to maintain and shift attention	0.92	0.84
6	Disorganized thinking	0.90	0.81
7	Perceptual disturbance	0.85	0.71
8	Delusion	0.78	0.61
9	Decreased or increased psychomotor activity	0.91	0.82
10	Sleep‐wake cycle disturbance	0.78	0.60

The area under the receiver operating characteristic curve of the MDAS‐T total score was 0.96 (95% CI, 0.93–0.99; p < .001; Fig. 1). The sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratio, and accuracy for each cutoff point of the MDAS‐T and CAM‐ICU‐T are listed in Table 4.

Figure 1

Receiver operating characteristic (ROC) curve and the area under the curve (AUC) for MDAS‐T, compared with delirium diagnosis by Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition diagnostic criteria.

Abbreviation: CI, confidence interval.

Table 4

Sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratio, and accuracy of the MDAS‐T and CAM‐ICU‐T (n = 194)

MDAS‐T cutoff score	Sensitivity (95% CI)	Specificity (95% CI)	Predictive value		Likelihood ratio		Accuracy (95% CI)
			Positive	Negative	Positive	Negative
7	0.98 (0.93–0.99)	0.76 (0.67–0.84)	0.81	0.97	4.23	0.03	0.88 (0.82–0.92)
8	0.96 (0.91–0.98)	0.84 (0.76–0.90)	0.86	0.95	6.08	0.05	0.90 (0.85–0.94)
9	0.92 (0.85–0.96)	0.90 (0.82–0.94)	0.90	0.91	8.73	0.09	0.91 (0.86–0.94)
10	0.90 (0.82–0.94)	0.91 (0.83–0.95)	0.90	0.89	9.49	0.11	0.90 (0.85–0.94)
11	0.87 (0.79–0.93)	0.92 (0.84–0.96)	0.91	0.87	10.32	0.14	0.89 (0.84–0.93)
CAM‐ICU‐T	0.58 (0.48–0.67)	0.98 (0.93–0.99)	0.96	0.68	27.35	0.43	0.78 (0.71–0.83)

The best cutoff score (9) appears in bold.

Abbreviations: CAM‐ICU‐T, Thai version of Confusion Assessment Method for the Intensive Care Unit; CI, confidence interval; MDAS‐T, Memorial Delirium Assessment Scale‐Thai version.

Table 5 summarizes the subgroup analysis of those patients who had an MDAS‐T score of ≥9 in the latent profile analysis, which classified the MDAS‐T scores into two to five classes. Based on the aforementioned criteria used to select the appropriate number of latent classes and clinical practice, the four‐class solution with the lowest BIC and ILC was chosen as the optimal solution as it yielded classifications that were clearly distinct and interpretable and had adequate class sizes. The four classes were labeled as “mild delirium” (an MDAS‐T score of 9–13), “moderate delirium” (14–18), “severe delirium” (19–25), and “profound delirium” (26–30), which represented 26 (25.7%), 29 (28.7%), 19 (18.8%), and 27 (26.8%) patients with delirium, respectively. The accuracy of the CAM‐ICU‐T compared with the DSM‐5 was 42.6% (95% CI, 28.8–64.5), 44.8% (95% CI, 28.4–62.5), 78.9% (95% CI, 56.7–91.5), and 92.6% (95% CI, 76.6–97.9) for the mild, moderate, severe, and profound delirium classes, respectively.

Table 5

Fit indices for correct number of latent classes in latent profile analysis

Number of latent classes	Log‐likelihood	BIC	ICL	BLRT	p value
2	−297.94	−618.96	−620.37	38.20	.001
3	−293.28	−623.47	−646.39	15.83	.001
4	−303.50	−643.93	−654.41	21.76	.001
5	−297.91	−641.98	−651.94	11.18	.006

p value associated with the BLRT.

The number of classes that yielded the best BIC and ICL value (4) appears in bold.

Abbreviations: BIC, Bayesian Information Criterion; BLRT, bootstrap likelihood ratio test; ICL, integrated complete‐data likelihood.

The medians (interquartile range) of the assessment times for the MDAS‐T, DSM‐5, and CAM‐ICU‐T were 5 (4–6), 6 (5–10), and 3 (2–5) minutes, respectively.

Discussion

Our study showed that it is valid and feasible to use the MDAS‐T for delirium screening in PC settings. It was found to have excellent internal consistency, with the Cronbach's α coefficient of 0.96 consistent with previous studies of versions that had been translated and validated in other languages 26, 27. As to the item analysis, all 10 MDAS‐T items had a corrected item‐total correlation above 0.7, which confirmed that each of the 10 items was correlated with the overall scale. If deleted, no item had a Cronbach's α greater than 0.96, which indicates that no item disproportionately affected the overall reliability.

The corrected item‐total correlation was high, with the third MDAS item yielding the lowest value. This item asks patients to repeat and recall 3 words. We found from the subgroup analysis that in the case of the mild delirium subgroup, the third MDAS item was scored the most frequently, followed by the fourth item, which tests attention. This may indicate that cognitive disturbance is the earliest symptom of delirium. Therefore, a screening tool which includes thorough cognitive function testing is necessary for the early detection of delirium.

An analysis of the construct validity using the EFA found that the MDAS‐T has a homogeneous, one‐factor structure. This might be because our research population mainly consisted of mixed type delirium, and thus mostly all items of the MDAS‐T were scored, resulting in a high correlation of communality. Therefore, we could not extract the MDAS‐T items into three factors (hypoactive, hyperactive, and mixed delirium) using the factor analysis, which had been suggested by Breitbart et al. 18. In contrast, Lawlor et al., Grassi et al., and Shyamsundar et al. consistently reported that they found multiple factors in their studies 28, 29, 30. Further research should be done to confirm the number of factors by using confirmatory factor analysis.

We found the highest accuracy of 0.91 with a cutoff score of 9. A previous study by Breitbart et al. suggested a cutoff score of 13, whereas Lawlor et al. recommended 7. These discrepancies might be due to the study populations. Breitbart et al. investigated the MDAS in patients who had already been treated for delirium, so they should have recovered to some degree from the delirium 18. Lawlor et al. recruited patients using a cognitive screening tool, which may have resulted in the study population having a higher prevalence of delirium than our study 28. These may affect the pretest probability and ultimately result in a different cutoff score. However, we believe that cutoff score of 9 is suitable for the usual PC encounter because we included all newly consulted PC patients without any intervention.

Most of the previous CAM‐ICU studies reported a high sensitivity 31. In contrast, we found that the CAM‐ICU‐T had a quite low sensitivity with our population. Our subgroup analysis showed that the CAM‐ICU‐T had very high accuracy for nondelirium and severe to profound delirium; in contrast, it had very low accuracy for mild and moderate severity. This may suggest that the CAM‐ICU‐T is not effective for the detection of mild to moderate delirium. As a result, the overall sensitivity of the CAM‐ICU‐T was low in our study population because of the more diverse severity of delirium than that found in ICU patients. We believe that ICU patients have more critical conditions and fewer cases of mild to moderate delirium than our population. Therefore, we do not suggest the use of the CAM‐ICU‐T for delirium screening in the PC population.

As to the median assessment times, although the DSM‐5 was done in 6 minutes (just a minute longer than for the MDAS‐T), it requires expertise in psychiatric assessment and is quite subjective, making it difficult for nonpsychiatrist personnel to use 16. Even though the CAM‐ICU‐T was mostly performed in 3 minutes, we support using the MDAS‐T because of its much better screening properties and scoring system, which allow us to follow up patients' delirium severities.

This study may have some limitations. For one thing, we excluded patients with a high symptom score, which might have been the result of the disinhibition feature of delirium. Excluding those patients may have resulted in a lower prevalence of delirium in this study population. Nevertheless, in our usual clinical assessments of patients with a high physical symptom score (due to pain or dyspnea, for example), we routinely give an instant treatment such as an oxygen supplement or immediate‐release opioids. This routine procedure resulted in only 15 patients out of 308 (4.87%) being excluded, which we believe would not lead to a significant selection bias. Another concern is that we excluded eight patients with dementia, which affected the size of our noncancer population. The results of this research might therefore not be applicable to patients with dementia. Further studies using MDAS‐T in palliative care settings at other than a university hospital and with patients without cancer (especially dementia) are needed.

Conclusion

This study established and validated the MDAS‐T. It has high reliability and yielded the best sensitivity and specificity for delirium at a cutoff score of 9. Our study supports the position that MDAS‐T is a good, highly objective, and feasible test for delirium screening and severity monitoring in PC settings.

Author Contributions

Conception/design: Watanachai Klankluang, Supakarn Tayjasanant, Arunotai Siriussawakul

Collection and/or assembly of data: Watanachai Klankluang, Supakarn Tayjasanant, Panate Pukrittayakamee, Wanlop Atsariyasing, Pratamaporn Chanthong

Data analysis and interpretation: Sasima Tongsai, Watanachai Klankluang, Supakarn Tayjasanant

Manuscript writing: Watanachai Klankluang, Supakarn Tayjasanant, Sasima Tongsai

Final approval of manuscript: Watanachai Klankluang, Supakarn Tayjasanant, Panate Pukrittayakamee, Wanlop Atsariyasing, Arunotai Siriussawakul, Pratamaporn Chanthong, Sasima Tongsai

Disclosures

The authors indicated no financial relationships.