Validation of the Korean Version of the Delirium Diagnostic Tool-Provisional (K-DDT-Pro).

OBJECTIVE: The Delirium Diagnostic Tool-Provisional (DDT-Pro) was designed to detect the presence and severity of delirium briefly and objectively regardless of psychiatric expertise. We translated the DDT-Pro into Korean and validated it in elderly Korean patients.
METHODS: To validate the translation and evaluate inter-rater reliability, a psychiatric trainee and a research nurse independently assessed the same patients referred to the Department of Psychiatry. The results were compared with the reference evaluations performed by a psychiatrist using the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5).
RESULTS: We enrolled 42 elderly patients. The Cronbach's alpha coefficient were 0.839 (the trainee), 0.822 (the nurse). The Cohen's weighted κ between the trainee and nurse, ranged from 0.555±0.102 to 0.776±0.062. The Pearson correlation coefficients (Korean version of the DDT-Pro [K-DDT-Pro] and Korean version of the Delirium Rating Scale-Revised-98 [DRS-R98-K] total score) were γ=-0.850 (the trainee), and γ=-0.821 (the nurse). The areas under the ROC curves (AUCs) were 0.974 (the trainee) and 0.893 (the nurse).
CONCLUSION: The K-DDT-Pro exhibited high internal consistency and relatively substantial inter-rater reliability. The correlation with the DRS-R98-K was strongly negative. The accuracy of the K-DDT-Pro was excellent, regardless of expertise. In conclusion, the K-DDT-Pro is a brief and simple tool that usefully screens for delirium in elderly patients.

INTRODUCTION

Delirium is a disturbance of consciousness, attention, and baseline cognitive function according to the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) [1]. Delirium increases morbidity, the hospitalization time, and mortality [2-5]. Delirium is common (7%–47%) in elderly inpatients [6-8]. As populations age, delirium will become more prominent. It is important to prevent delirium or at least to detect delirium early; this allows rapid medical intervention. However, in clinical studies, about two-thirds of delirium cases were not diagnosed; in emergency settings, 57%–83% of delirium cases were missed, because it is difficult for medical staff lacking expertise with psychiatric interviews and differential diagnosis to identify and evaluate the characteristic neuropsychiatric symptoms of delirium and to distinguish them from those of other diseases [9-11]. There is a need for a simple screening tool that can be used by medical staff who are not specialists in psychiatry or neurology. However, the validity of a delirium evaluation must be verified, as must the capacity of the tool to distinguish delirium from dementia. Also, the various aspects of delirium must be revealed, and, finally, the tool must be practical [12].

The Confusion Assessment Method algorithm (CAM-A) is one of the most commonly used tools that enables nonpsychiatric clinicians to detect delirium quickly and easily. The CAM-A evaluates the presence of four important symptoms of delirium: acute onset and fluctuating course, inattention, disorganized thinking, and altered level of consciousness. The CAM-A can be completed in less than 5 minutes. Because of its simplicity and convenience, CAM-A is widely used by nonexpert clinicians for delirium detection. However, some training is required for optimal use of the CAM. The CAM allows stupor or coma to be diagnosed as delirium and is a dichotomous tool not allowing it to be a continuous measure even with adaptations [13-16]. The 4A’s Test (4AT), another brief assessment tools for delirium and cognitive impairment, developed in the UK. 4AT screens patients with four items that assess the alertness, orientation, attention, and history of a patient’s acute change or fluctuation course in cognition or other mental function. It is short (<2 minutes), simple screening tool for delirium and shows good diagnostic test accuracy for delirium. In the recent meta-analysis study, the pooled sensitivity was 88% and the pooled specificity was 88%. It does not require special training and has the advantage of being easy to implement. However, the test results are affected by whether dementia is accompanied [17]. In addition to CAM-A and 4AT, many tools to detect delirium are being developed and verified.

The Delirium Diagnostic Tool-Provisional (DDT-Pro) developed by Kean et al. [18] is an excellent screen allowing a provisional diagnosis of delirium; the tool is delirium-specific, sensitive, and reliable. The DDT-Pro evaluates the three core symptom domains of delirium: cognition (vigilance in DDTPro), higher-level thinking (comprehension in DDT-Pro), and circadian rhythm (sleep-wake cycle in DDT-Pro). The presence of these core domain symptoms could accurately reflect delirium while other brief tools do not evaluate all the three core symptom domains. In comparison to the CAM-A, because of its highly structured, quantitative items, the DDT-Pro would be more useful to provisionally diagnose delirium. The previous validation study of the DDT-Pro in general hospital patients shows a significant advance over the commonly used CAM. Although the English and Spanish versions of DDT-Pro have been validated with excellent results, validation in other languages employed by specific patient populations is also required [16,19,20]. A Korean version would be useful as Korean hospitals lack such tools. We translated the DDT-Pro from English to Korean (Korean version of the DDT-Pro, K-DDT-Pro). We compared the performance of the K-DDT-Pro to that of the Korean version of the Delirium Rating Scale-Revised-98 (DRS-R98-K), which is validated and is commonly used in Korea [21,22]. We explored the inter-rater reliability of the K-DDT-Pro between an expert and a non-expert; the reference diagnosis was that of a psychiatrist using DSM-5 criteria.

METHODS

Patients

From April 2021 to March 2022, we enrolled inpatients aged 60 years or older of the Department of Psychiatry referred because of suspected delirium or a risk of delirium from various departments at Chungnam National University Hospital. The exclusion criteria included a refusal to participate, any severe communication problem (e.g., hearing loss), stupor, and coma. The Chungnam National University Hospital Institutional Review Board (IRB) approved this study (IRB no. 2021-03-058), and written informed consent was obtained from patients or if necessary, their surrogates.

The Delirium Diagnostic Tool-Provisional description

The three items of DDT-Pro (comprehension, vigilance, and the sleep-wake cycle) are derived from two well-validated delirium diagnostic tools, the Cognitive Test for Delirium (CTD) (items 1 and 2), and the Delirium Rating Scale-Revised-98 (DRS-R98) (item 3). These items represent the core symptoms of delirium. Item 1 (comprehension) and item 2 (vigilance) are expressed in two equivalent forms (A and B), which can be used alternatively used during repeated assessments. Both forms directly reflect the patients’ performance. Item 3 (sleep-wake cycle) is evaluated by the patient interview or by another source for the previous 12–24 hours. Item 3 taken from the DRS-R98 has its scoring reversed to align with that of the CTD items. Each item on the DDT-Pro is scored on a 4-point scale (0–3). Thus, the total score ranges from 0 to 9 points. The lower the score, the higher the severity of delirium, where a score of 8–9 is normal.

Translation and back-translation

We contacted Dr. Trzepacz, the original author of the DDT-Pro, via e-mail and got permission to produce a K-DDT-Pro. The DDT-Pro was translated using the guidelines of the Translation and Cultural Adaptation Group. The initial translation was prepared by a psychiatric trainee and then revised and supplemented by two psychiatrists working independently. Then, all researchers met to discuss progress. When translating item 2, we took care to ensure that patients could easily distinguish the words. The final Korean version was given to a professional translator who speaks both Korean and English for back-translation; we did not provide any information on the original version. The back-translated version of K-DDT-Pro was sent to Dr. Trzepacz by e-mail and was approved. Next, in April 2021, the psychiatric trainee conducted a preliminary study on five delirious patients.

Inter-rater reliability and validation of delirium assessment

Prior to the full-scale evaluation, all raters met to discuss the assessment method. One experienced psychiatrist evaluated all patients using DSM-5 delirium criteria. Then, one psychiatric trainee; a psychiatry resident who had graduated from medical school (who used form A) and one research nurse (who used form B) independently performed delirium assessments on the same subjects using the K-DDT-Pro. The trainee was the psychiatric expert, and the nurse was the non-expert. During the evaluation, the trainee also evaluated patients using the DRS-R98-K with information from their family members and caregivers. The patient assessments of the three raters were completely independent (Figure 1). All evaluations were conducted between 14:00 and 18:00 to minimalize any possible bias arising from fluctuations in each patient’s condition.

Figure 1.

The process of the assessment. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition; K-DDTPro, Korean version of the Delirium Diagnostic Tool-Provisional; DRS-R98-K, Korean version of the Delirium Rating Scale-Revised-98.

Statistical analysis

We classified the patients into the delirium group or non-delirium group based on their DSM-5 diagnoses. The two groups were compared. As the sample sizes were small (both n<30), we subjected all demographic and clinical variables to normality analysis. We then compared the means between the two groups. Age, which satisfied the normality requirement, was compared using the independent t test. Years of education, which was not normally distributed, was compared employing the Mann–Whitney U test. We used the chi-squared test to compare all other nominal variables. To assess the internal reliability of each K-DDT-Pro item, we compared the trainee and nurse ratings by deriving the Cronbach’s α coefficient. The Cohen’s weighted kappa (κ) coefficient was used to assess the inter-rater reliability of the K-DDT-Pro forms administered by the psychiatric trainee (form A) and the nurse (form B). We evaluated all item scores and the overall scores of the two independent raters.

The Criterion validity of the K-DDT-Pro (versus the DRS-R98-K total and severity scores) was evaluated via Pearson correlation analysis. We drew receiver operating characteristic (ROC) curves to assess the accuracy, sensitivity, and specificity of the K-DDT-Pro scores. The references were the psychiatrist DSM-5–based diagnoses. We then determined a K-DDT-Pro cutoff that balanced sensitivity and specificity. All analyses were performed using the SPSS ver. 26.0 software (IBM Co., Armonk, NY, USA).

RESULTS

Baseline characteristics

The patient flow diagram is shown in Figure 2. We interviewed 95 patients who met the inclusion criteria, but 53 also met the exclusion criteria. The ‘communication problem’ of the exclusion criteria is when the patient has severe hearing loss or can neither talk nor motion a response. Therefore, we enrolled 42 patients. The sociodemographic characteristics of the subjects are summarized in Table 1. Using the DSM-5 criteria, 25 were diagnosed with delirium by the psychiatrist. Eight had previously been diagnosed with dementia by psychiatrists or neurologists. The independent t test showed that patients with delirium were older than others. However, no other parameter differed significantly between the two groups.

Figure 2.

Flow diagram of the study participants. *when the patient has severe hearing loss or can neither talk nor motion a response. DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition.

Table 1.

Baseline characteristics of participants

	Total (N=42)	Delirium group (N=25)	No delirium group (N=17)	p value (t, U, χ2)
Age (yr)	73.69±7.41	76.08±7.36	70.18±6.13	0.01[*]
Sex, female	20 (47.6)	11 (44.0)	9 (52.9)	0.569[†]
Education level (yr)	7.45±4.16	7.04±3.81	8.06±4.67	0.264[‡]
Marital status, married	31 (73.8)	19 (76.0)	12 (70.6)	0.366[†]

Values are presented as mean±standard deviation for age, education level, and as N (%) for sex (female) and marital status (married). Delirium group, participants who diagnosed to delirium with Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) by a psychiatrist.

significant at p<0.05 for independent t test;

p value by chi-square tests;

p value by Mann–Whitney U test

Internal and inter-rater reliability of the Korean version of the Delirium Diagnostic Tool-Provisional

The Cronbach’s α coefficients for the psychiatric trainee and the nurse were 0.839 and 0.822, respectively. The internal reliability of the K-DDT-Pro was very good. The Cohen weighted κ coefficients for the K-DDT-Pro items were (item 1) 0.555±0.102 (p<0.001), (item 2) 0.776±0.062 (p<0.001), (item 3) 0.647±0.081 (p<0.001); and that for the total score 0.718±0.048 (p<0.001) (Table 2). The inter-rater agreement was moderate to substantial.

Table 2.

Inter-rater reliability of K-DDT-Pro (psychiatric trainee and nurse)

	Kappa (κ)±standard error (95% CI)
	Cohen’s simple kappa	Cohen’s weighted kappa
K-DDT-Pro total[†]	0.318±0.085[*]	0.718±0.048[*]
K-DDT-Pro item 1	0.421±0.115[*]	0.555±0.102[*]
K-DDT-Pro item 2	0.601±0.094[*]	0.776±0.062[*]
K-DDT-Pro item 3	0.512±0.101[*]	0.647±0.081[*]

p<0.001;

total score of all items in K-DDT-Pro.

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; CI, confidence interval

Validity of the Korean version of the Delirium Diagnostic Tool-Provisional and selection of the cutoff score for delirium

The K-DDT-Pro and the DRS-R98-K total and severity scale scores were inversely related as revealed by the Pearson rho values (Table 3). The K-DDT-Pro assessments of the psychiatric trainee correlated with the DRS-R98-K severity score (-0.850 [p<0.001]) and with the DRS-R98-K total score (-0.857 [p<0.001]). The figures for the nurse were -0.821 (p<0.001) and -0.815 (p<0.001), respectively. Thus, the correlations were very high regardless of the rater’s psychiatric expertise.

Table 3.

Correlations among K-DDT-Pro and DRS-R98-K

K-DDT-Pro total[‡]	Pearson’s r
	DRS-R98-K severity[*]	DRS-R98-K total[†]
Trainee	-0.850 (p<0.001)	-0.857 (p<0.001)
Nurse	-0.821 (p<0.001)	-0.815 (p<0.001)

severity item score for DRS-R98-K;

total score of all items in DRS-R98-K;

total score of all items in K-DDT-Pro.

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; DRS-R98-K, Korean version of the Delirium Rating Scale-Revised-98

The areas under the ROC curves (AUCs) for K-DDT-Pro diagnostic accuracy using the DSM-5 criteria as the reference were 97.4% (95% confidence interval [CI]=93.3%–100%) for the psychiatric trainee and 89.4% (95% CI=80%–98.6%) for the nurse (Figure 3). The sensitivity and specificity of the K-DDT-Pro diagnoses compared to the of DSM-5 diagnoses as revealed by ROC analysis are listed in Table 4. The ascending sensitivity and descending specificity gradients were determined for various cutoff values. K-DDT-Pro scores ≤6 and ≤7 both seemed to balance sensitivity and specificity. Scores ≤7 balanced sensitivity and specificity only in the assessments of the psychiatric trainee. Therefore, the optimal cutoff may be ≤6 or ≤7.

Figure 3.

AUC for the K-DDT-Pro for delirium diagnosis (DSM-5 independent diagnosis). K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; AUC, area under the receiver operating characteristic curve; CI, confidence interval.

Table 4.

Sensitivity and specificity analysis using patient operating characteristic for the K-DDT-Pro for delirium diagnosis in accordance with DSM-5 (total N=42)

Cutoff	K-DDT-Pro by a psychiatric trainee		K-DDT-Pro by a nurse
	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)
1	8	100	4	100
2	12	100	32	100
3	36	100	44	100
4	48	100	52	100
5	60	100	56	100
6	84	94.1	80	82.4
7	100	82.4	92	52.9
8	100	41.2	100	29.4

K-DDT-Pro, Korean version of the Delirium Diagnostic Tool-Provisional; DSM-5, Diagnostic and Statistical Manual of Mental Disorders, 5th Edition

DISCUSSION

We prepared a K-DDT-Pro and evaluated its reliability and validity in terms of diagnosing delirium. In terms of reliability, all three items evidenced high internal consistency, with Cronbach’s α coefficient of 0.839 (the trainee) and 0.822 (the nurse). In terms of the Cohen weighted κ coefficients for each item, the tool evidenced moderate (0.555±0.102, item 1) to substantial (0.776±0.062, item 2) inter-rater reliability between the psychiatric expert and the non-expert. Note that the inter-rater reliability of item 1 was lower than that of others. For comparison, when the item 1 scores of the two raters were averaged, that of the psychiatric trainee (2.4286) was higher than that of the nurse (2.2143). One possible reason is that form A (used by the trainee) could be more intuitively understood by the elderly patients than form B (used by the nurse). Also, the possibility that the evaluation methods differed (even slightly) between the raters cannot be excluded.

The nurse ratings in this study were a little less strong than the trainee. In our opinions, because the research nurse can’t observe the patient’s conditions bedside unlike clinical nurses (representing non-experts in previous studies), it must have been more difficult for research nurse to acquire information about patients than clinical nurses or the trainee. Of course, it can be explained with absolute results of the present study, but further research on clinical nurses (instead of the research nurse) may be needed to explain that the DDT-Pro can be used as useful in non-experts (e.g., nurses, rehabilitation staff) as physicians.

A significant inverse Pearson correlation was apparent between the K-DDT-Pro and DRS-R98-K scores. As the DRS-R98-K is well-validated, this constitutes indirect evidence that the K-DDT-Pro will also usefully screen for delirium. We then confirmed the validity of the K-DDT-Pro.

To ensure that the tool would be clinically useful, the optimal cutoff for delirium diagnosis was obtained via ROC analyses; this was ≤6 or ≤7, as for the English and Spanish versions [16,18,19]. Considering subsyndromal delirium (SSD), the scores of 6 and 7 indicate SSD and full delirium is denoted by the lower score range.19 Compared to the original validation studies of DDT-Pro (sensitivity 100% and specificity 94% in Kean et al. [18], 2010; sensitivity 90% and specificity 87% in Franco et al.,16 2020), the K-DDT-Pro exhibited similar sensitivity and specificity values.

Our work had several limitations. First, the sample size was too small (n=42) to compare the outcomes of patients with and without dementia. Given the small sample size, we sought to render the samples homogeneous, but this was difficult. In the present study, a research nurse was the non-psychiatric expert. Clinical nurses who observe the patient’s conditions bedside should be evaluated as raters.

In conclusion, the K-DDT-Pro exhibited high validity and reliability in terms of delirium screening regardless of the rater’s psychiatric expertise. Thus, the K-DDT-Pro will find clinical applications in Korean hospitals. However, additional studies with larger sample sizes are required. Also, clinical nurses who observe patients’ conditions bedside should be evaluated as raters.

We hope that all medical staff, regardless of their psychiatric expertise, will use the brief K-DDT-Pro to improve the prognosis of elderly inpatients with delirium. Early detection of delirium will trigger rapid medical intervention.