Development and validation of the Thai version of the 4 'A's Test for delirium screening in hospitalized elderly patients with acute medical illnesses.

BACKGROUND: The English version of the 4 'A's Test (4AT) is a rapid screening tool for delirium with a high sensitivity and specificity among hospitalized elderly patients.
OBJECTIVE: To develop the Thai version of the 4AT (4AT-T) and assess its validity. SUBJECTS AND
SETTING: A total of 97 elderly patients aged 60 years or above who were admitted to the general medical wards were included.
METHODS: Both authors independently translated the English version of the 4AT into Thai and thereafter developed a single reconciled forward translation by consensus. Back translation was performed by a bilingual native English speaker and it was then reviewed to ensure its agreement with the original one. After 24 hours of admission, subjects were enrolled and clinical data collected. Definite diagnosis of delirium was made by a psychiatrist using the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, Text-Revision criteria and the 6-item Thai Delirium Rating Scale; the 4AT was then administered to participants by nurses within 30 minutes. A 4AT score ≥4 was considered positive for delirium screening. The optimal cut-off point of the 4AT-T was identified by Youden's index.
RESULTS: In all, 24 out of 97 participants met the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, Text-Revision criteria for delirium. At a cut-off score of 4 or greater, the 4AT-T exhibited satisfactory diagnostic performance with a sensitivity of 83.3% (95% confidence interval (CI): 62.6%-95.3%) and specificity of 86.3% (95% CI: 76.3%-93.2%). The area under the receiver operating characteristic curve was 0.92. The specified score provided maximal Youden's index, suggesting an optimal criterion value for delirium screening.
CONCLUSION: The 4AT-T is a valid delirium-screening instrument for hospitalized elderly patients with acute medical illnesses.

Introduction

Delirium is a neurocognitive disorder usually caused by multiple medical conditions that result in generalized brain dysfunction. The essential features include disturbance in attention and other cognitive functions, disorientation to the environment, and fluctuation in severity during the course of the day.1,2 The condition is common, especially among older people, and associated with high mortality rates ranging from 22% to 76% among hospitalized patients.3 Moreover, delirium significantly increases functional disability, costs of care, length of stay, rates of institutionalization, and rates of nursing home placement.2,4,5

In general, delirium can be resolved after the underlying etiological factors are promptly corrected;1,6 however, the condition is frequently underrecognized,5,7 resulting in a delay for appropriate management and probably leading to persistence of the symptoms.6,8 A recent systematic review reveals that the rates of persistent delirium among hospitalized elderly patients were indeed as high as 44.7% at discharge from hospital and 21% 6 months later.8 Delayed recognition and persistence of delirium also appear to be associated with worse outcomes and higher mortality rate in elderly patients.8–10 Hence, routine screening for delirium is a key strategy for early detection of the condition and avoiding delayed management.4,6,10,11

A large number of instruments for screening, diagnosis, and assessment of delirium have been developed over the past decades.12–14 Among these, the Confusion Assessment Method is one of the most extensively used tools for delirium screening according to its high sensitivity and specificity (95% and 89%, respectively) as well as its brevity in administration.10,11,15,16 However, optimal training is required to achieve such high accuracy. To improve practicality of delirium screening, the 4 ‘A’s Test (4AT; www.the4at.com) has been developed as a rapid assessment test for delirium, which requires no specific training. Its satisfactory diagnostic test accuracy has been proved in 234 hospitalized elderly patients evaluated by a geriatrician without prior specific training.17 The 4AT score ranges from 0 to 12, with a score of 4 or above indicating delirium. The area under the receiver operating characteristic (ROC) curve for delirium detection was 0.93, with a sensitivity of 89.7%, specificity of 84.1%, positive likelihood ratio of 5.62, and negative likelihood ratio of 0.12.17 As compared to other brief screening tools, the 4AT also showed greatest sensitivity (100%) with reasonable specificity (82%) for delirium detection in an acute stroke unit.18

In Thailand, the prevalence of delirium has been reported to be as high as 40.4% among older patients admitted to general medical wards.19 In in-patient settings, nurses are health care personnel holding great opportunities to detect subtle changes in mental status of patients. Nevertheless, underrecognition of delirium frequently occurs,20 which could be partly attributed to excessive workloads of attending staff.21 A study of 5,247 nurses who provided direct care for patients in 39 public hospitals in Thailand found that each nurse was responsible for 7–13 patients.22 To improve detection of delirium, a simple but accurate screening instrument is required particularly in clinical settings with a disproportionately large number of patients, such as in our country. The objective of this study was to evaluate the validity and reliability of the Thai version of the 4AT (4AT-T) as a screening instrument for delirium in general medical settings.

Methods

We applied the reporting standards for studies of diagnostic test accuracy in dementia in our study.23 This study was approved by the Ethics Committee on Human Experimentation of the Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Bangkok, Thailand. Written informed consent was obtained either directly from the participant or from his/her legally authorized representative. This study project continued from April 2014 to September 2015.

Study design and participants

This study was a cross-sectional study conducted in Ramathibodi Hospital, which is a tertiary hospital with 1,300 beds located in the center of Bangkok, Thailand. We defined “elderly” as an age of 60 years or above according to the National Statistical Office of Thailand. The elderly patients who were admitted to the general medical wards were recruited from December 2014 to February 2015. The medical wards provide care for adults of all ages who were admitted with acute medical conditions, while each staff nurse usually takes care of six to eight patients on an average. The service includes patients transferred from intensive care and intermediate units when critical medical conditions had subsided, and also those transferred from secondary health care centers due to complex medical conditions. We excluded patients with conditions that prohibit adequate communication, including severe hearing impairment, inability to speak, intubation, and coma.

Measures

Development of the 4AT-T

After obtaining permission from the copyright holder, the English 4AT version 1.1 was translated into Thai by one psychiatrist (SK) and one psychiatry resident with PhD (TP). These forward translations were carried out independently, and thereafter these were reconciled into a consensus version. Then, the final Thai version was delivered to a professional translator (IH), a bilingual native English speaker fluent in Thai language for back translation without prior knowledge of the original version. Finally, SK and TP reviewed the back-translated version to ensure concordance between the translated and the original versions.

The 4AT-T consists of four items. First, the alertness item assesses the altered level of alertness.24 Second, the 4-item Abbreviated Mental Test (AMT4)25 assesses patients’ cognition by asking about their age, date of birth, the present place, and the current year. Third, the attention item tests patients’ attention using the months backwards test.26 Lastly, the acute change or fluctuating course item evaluates the onset and course of the altered mental status. The total score ranges from 0 to 12 which is the same as the original version.

6-item TDRS

The 6-item Thai Delirium Rating Scale (TDRS) is an instrument used for diagnosing and severity grading of delirium.27 The evaluated items include temporal onset of symptoms, psychomotor behavior, cognitive status, physical disorder, sleep–wake cycle disturbance, and variability of symptoms. The total score of TDRS ranges from 0 to 20 and positively correlates with the severity of delirium symptoms.27 At a cut-off score of 10 or above, the tool showed excellent sensitivity and specificity for the diagnosis of delirium (97.5% and 97.8%, respectively) as well as good internal consistency (Cronbach’s alpha =0.89).27

Procedure

Demographic and clinical characteristics were derived from a direct interview and clinical records. Charlson comorbidity index, ranging from 0 to 11, was calculated to determine the seriousness of comorbid diseases of each participant.28 After being admitted to the wards for at least 24 hours, each participant was interviewed and assessed by one of the two experienced psychiatrists blinded to 4AT scores; definite diagnosis of delirium was made in accordance with the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, Text-Revision (DSM-IV-TR) criteria.29 Additionally, the 6-item TDRS was used to confirm the diagnosis and determine the severity of the symptoms. To be included in delirium group, the DSM-IV-TR diagnosis by the psychiatrist must be delirium and the total score of TDRS must be 10 or above. To evaluate inter-rater reliability, five participants were co-rated with the TDRS and were diagnosed independently by both psychiatrists. All psychiatrists who participated in this study were certified by the Thai Board of Psychiatry and had experience in taking care of patients with delirium.

Within 30 minutes of the evaluation by the psychiatrists, the 4AT-T was administered to each participant by one of the three nurses who work regularly at the wards with no specific training provided. Scoring of the 4AT-T was performed in a blinded fashion to the diagnosis made by the psychiatrists. The inter-rater reliability of the 4AT-T was established in ten participants using a similar process applied to the 6-item TDRS. After completing the evaluation, all the three nurses were interviewed briefly for some feedback about the usage of this new assessment tool.

Statistical analyses

Data analyses were performed using PASW Statistics for Windows, Version 18.0.0 (SPSS Inc., Chicago, IL, USA) and MedCalc for Windows, Version 15.6.1 (MedCalc Software, Ostend, Belgium). Characteristics of patients with and without delirium were compared using chi-square test for categorical variables and Student’s t-test for continuous variables. The internal consistency of the 4AT-T was measured by Cronbach’s alpha coefficient. Correlations between each item and the total scores of the 4AT-T were determined. The inter-rater reliability for the 4AT-T and 6-item TDRS was assessed with intra-class correlation coefficient. The inter-rater agreement for diagnosis of delirium according to DSM-IV-TR criteria was measured by Cronbach’s alpha coefficient.

To evaluate validity of the test, sensitivity, specificity, likelihood ratios, and predictive values of the 4AT-T were calculated using the DSM-IV-TR criteria as a reference standard. The area under the ROC curve was determined. The performance of the 4AT-T against the reference standard was also re-evaluated at various cut-off scores. Youden’s index was calculated to determine the optimal criterion value of the test.

Results

From a total of 237 elderly patients aged 60 years or above who were admitted to the general medical wards during a period of 3 months, 97 were included in this study. Among these, 24 patients had delirium according to DSM-IV-TR criteria as well as 6-item TDRS. Demographic and clinical characteristics of the study population are shown in Table 1. The mean age was 73.6 years (standard deviation (SD) =8.17). There were approximately equal proportions of males and females in both groups of patients. About 60% of participants had 0–6 years of education. No significant difference of mean age, sex, total years of education, and comorbidity index was observed between the delirium and nondelirium groups.

Table 1

Demographic and clinical characteristics of the study population

	Delirium (N=24)	Nondelirium (N=73)	P-value
Age in years, mean (SD)	75.3 (8.1)	73.1 (8.2)	0.265
Sex, female N (%)	12 (50)	36 (49.3)	0.954
Years of education, N (%)
>12 years	3 (12.5)	14 (19.2)	0.812
7–12 years	1 (4.2)	2 (2.7)
1–6 years	15 (62.5)	39 (53.4)
0	5 (20.8)	18 (24.7)
Presenting symptoms, N (%)
Acute fever	5 (20.8)	9 (12.3)	0.012
Dyspnea	6 (25.0)	31 (42.5)
Alteration of consciousness	5 (20.8)	2 (2.7)
Others	8 (33.3)	31 (42.5)
CCI median (IQR)	3 (1.25–5)	4 (2–5.5)	0.403

Abbreviations: CCI, Charlson comorbidity index; IQR, interquartile range; SD, standard deviation.

Reliability and item analysis

Cronbach’s alpha for the diagnosis of delirium based on DSM-IV-TR criteria was 0.998. Intra-class correlation coefficients for the 6-item TDRS and 4AT-T were 0.973 (95% CI: 0.771–0.997; P=0.001) and 0.998 (95% CI: 0.994–0.999; P<0.001), respectively.

Cronbach’s alpha for the internal consistency of the 4AT-T was 0.75. Mean (SD) and corrected item-corrected correlation are shown in Table S1.

Validity

Twenty out of 24 delirious patients were correctly classified as delirious using the 4AT-T. As compared to the reference standard as well as the 6-item TDRS, the 4AT-T falsely detected ten nondelirious subjects but missed detecting four delirious patients (Table 2). Out of ten patients misclassified by the 4AT-T as delirious, five were found to have dementia according to clinical history from reliable informants, and another two were diagnosed to have organic brain syndrome. The instrument, at a cut-off point of 4 or greater, exhibited a sensitivity of 83.3% (95% CI: 62.6%–95.3%), specificity of 86.3% (95% CI: 76.3%–93.2%), positive predictive value of 66.7% (95% CI: 47.2%–82.7%), negative predictive value of 94.0% (95% CI: 85.4%–98.4%), positive likelihood ratio of 6.1 (95% CI: 3.3–11.1), and negative likelihood ratio of 0.2 (95% CI: 0.1–0.5). The ROC curve demonstrated efficient performance of the 4AT-T in the detection of delirium (Figure 1). The area under the curve was 0.92 (95% CI: 0.86–0.97, P<0.001), suggesting high accuracy.30

Table 2

Diagnostic performance of the 4AT-T compared with the reference standard

		Reference standard*
		Delirium	Nondelirium	Total
4AT-T	Delirium (N)	20	10	30
	Nondelirium (N)	4	63	67
	Total	24	73	97

Note:

Reference standard = DSM-IV-TR diagnosis and TDRS score 10 or above.

Abbreviations: 4AT-T, Thai version of the 4 ‘A’s Test; DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders, 4th edition, Text-Revision; TDRS, Thai Delirium Rating Scale.

Figure 1

ROC curve demonstrating the efficient performance of the 4AT-T in the detection of delirium.

Abbreviations: 4AT-T, Thai version of the 4 ‘A’s Test; ROC, receiver operating characteristic.

When each subtest was focused on, the highest sensitivity was observed for the attention item and the AMT4 (95.8% at a score of 1 in both items). The attention test also exhibited high specificity (94.5% at a score of 2). The alertness item showed the highest specificity and positive likelihood ratio but the lowest sensitivity as compared to the others. The diagnostic performance of the 4AT-T and each subtest are shown in Table 3.

Table 3

The performance of the 4AT-T and each subtest in diagnosing delirium

	4AT total	Alertness	AMT4		Attention		Acute change/fluctuation
Cut-off	≥4	4	1	2	1	2	4
Sensitivity (%)	83.3	37.5	95.8	87.5	95.8	50.0	75.0
Specificity (%)	86.3	97.3	67.1	80.8	41.1	94.5	87.7
Positive predictive value (%)	66.7	81.8	48.9	60.0	34.8	75.0	66.7
Negative predictive value (%)	94.0	82.6	98.0	95.2	96.8	85.2	91.4
Positive likelihood ratio	6.1	13.7	2.9	4.6	1.6	9.1	6.1
Negative likelihood ratio	0.2	0.6	0.1	0.2	0.1	0.5	0.3
Area under ROC curve	0.919	0.674	0.867		0.806		0.813
P-value	<0.001	0.011	<0.001		<0.001		<0.001

Abbreviations: 4AT, 4 ‘A’s Test; 4AT-T, Thai version of the 4AT; AMT4, 4-item Abbreviated Mental Test; ROC, receiver operating characteristic.

To determine an optimal cut-off score of the 4AT-T for delirium screening, the diagnostic performance of the test was re-evaluated at various cut-off values. As shown in Table S2, maximal Youden’s index was derived at the cut-off point of >3, suggesting the most appropriate diagnostic criterion for delirium.

Feedback from assessors

All participating nurses provided us some positive feedback about the ease of use and practicality of the 4AT-T. As one nurse mentioned: “All questions of the tool are quite simple and easy to understand. I can complete the assessment within five minutes per case.” Another comment was “I think it could be included in our routine works and I feel no burden using it.”

On the other hand, there was feedback about the difficulty inquiring the date of birth in the AMT4 section: “A few patients gave me the answer of the year of birth in Chinese zodiac, like the Year of Tiger, and then I had to find out more what the right year was.”

Discussion

This study aims to develop and validate a rapid, easy-to-use, screening instrument for delirium in hospitalized elderly patients, a group of the population with a high risk for delirium. The 4AT-T performed well in delirium detection with high sensitivity and specificity. Although the sensitivity of the 4AT-T was less than the original one17 (83.3% and 89.7%), the 4AT-T had a higher specificity (86.3% and 84.1%). Moreover, there is a slight difference in the areas under the ROC curves (0.92 and 0.93). It also demonstrated a high negative predictive value, which greatly facilitates exclusion of delirium in negative cases. The optimal cut-off value of the test for delirium screening was also consistent with the original version.

Despite its satisfactory performance profile, four out of 24 delirious patients were not detected. These subjects appeared to have TDRS scores ranging from 10 to 13, implying that the 4AT-T might be less sensitive to detect mild form of delirium. Considering that symptom severity of delirium tends to fluctuate during the day, serial assessment may improve the sensitivity of the test.

An earlier study of the 4AT in hospitalized elderly patients revealed that its specificity is compromised in demented population.17 Even though baseline cognitive function was not evaluated in the present study, it seems that cognitive impairment that is not caused by delirium, for example, dementia or organic brain syndrome, could contribute to false positive results. The finding is not surprising because it is generally known that clinical expertise is required to discriminate delirium from behavioral and psychological changes due to impaired cognitive function. Besides, such over-detection should be considered acceptable for screening of a serious condition such as delirium.

In Thailand, a heavy workload due to the small number of health care personnel has long been a key public health problem. Concerning delirium, this might be an important factor leading to underrecognition of the syndrome due to lack of time. In the present study, the 4AT-T was administered by attending nurses without prior specific training provided and demonstrated its brevity and ease of use. Moreover, in the medical wards where each nurse has to provide care for up to eight patients with complex conditions, the 4AT-T was still compatible with routine work confirmed by feedback from the participating nurses. From all the aforementioned findings, we propose that the 4AT-T could be a practical delirium-screening tool for routine use in similar clinical scenarios in Thailand.

However, the 4AT-T also has some limitations. First, the test is not suitable for patients having difficulty in communication because such conditions could interfere with scoring of the AMT4 and the attention items. Second, according to the scoring system of the 4AT, solely the presence of changes in alertness or a fluctuating course of the mental status would already generate a score of 4 points, which is the cut-off score for possible delirium. Therefore, any errors in scoring of these two items, probably due to insufficient information from reliable informants or lack of evaluators’ experience in assessing the alertness level, could lead to a misclassification of the condition. Moreover, considering that this study included only patients with general medical illnesses, further validation studies are required for application of the test to patients with other specific conditions such as those in postoperative or critical care settings.

Conclusion

The 4AT-T is a reliable and valid delirium-screening instrument for hospitalized elderly patients with acute medical illnesses. The tool appears practical even in high-workload settings without a need for specific training. Further validation studies in patients with other specific clinical conditions are required.

Table S1

Item level values and item–total correlation of 4AT-T

Items	Mean	SD	Corrected item–total correlation
Alertness	0.454	1.27	0.549
AMT4	0.845	0.93	0.607
Attention	0.845	0.68	0.680
Acute change/fluctuation	1.113	1.80	0.649

Abbreviations: 4AT-T, Thai version of the 4 ‘A’s Test; SD, standard deviation; AMT4, 4-item Abbreviated Mental Test.

Table S2

The performance of the 4AT-T at various cut-off values in diagnosing delirium

Criterion	Sensitivity (%)	Specificity (%)	LR+	LR−
≥0	100.00	0.00	1.00
>0	100.00	39.73	1.66	0.00
>1	95.83	64.38	2.69	0.065
>2	91.67	73.97	3.52	0.11
>3	83.33	86.30	6.08	0.19
>4	79.17	87.67	6.42	0.24
>5	79.17	89.04	7.22	0.23
>6	75.00	93.15	10.95	0.27
>7	45.83	95.89	11.15	0.56
>8	33.33	97.26	12.17	0.69
>9	33.33	98.63	24.33	0.68
>11	29.17	98.63	21.29	0.72
>12	0.00	100.00		1.00

Note: Bold values represent maximal Youden’s index J =0.70.

Abbreviations: 4AT-T, Thai version of the 4 ‘A’s Test; LR+, positive likelihood ratio; LR−, negative likelihood ratio.