Assessing the validity of the Triage Risk Screening Tool in a third world setting.

Background Social risk is rarely evaluated in older emergency department (ED) patients, although its identification can reduce re-attendance. Objectives This study assessed the diagnostic accuracy of the Triage Risk Screening Tool (TRST) in the ED of a developing country. Methods The diagnostic accuracy of the TRST to detect elderly adults in need of social service intervention was compared with routine clinical evaluation, using comprehensive evaluation by an experienced social worker as the "gold standard". The inter-rater reliability of the TRST was assessed on a separate cohort of patients prior to the main study. Results The sensitivity of the TRST was 94.7% versus 55.6% for physician assessment. The TRST had good inter-rater reliability (Cohen's kappa = .882), and physicians found it easy to use. Conclusion The TRST provides a rapid means of assessing risk in older ED patients. This study confirmed the validity of this screening tool in a third world setting.

Introduction

Psychosocial assessments are not routinely performed among older people attending emergency departments (EDs) in Trinidad, although this is routine in many developed countries.[1-3]

Adverse outcomes, including higher mortality and re-attendance rates in older ED patients are higher in those with inadequate social support. For example, Dramé et al.[4] demonstrated that social isolation and increased caregiver burden were associated with higher 6-month mortality rates in older hospitalised patients, and Foo et al.[5] demonstrated that identification and addressing the social support needs of older ED patients significantly reduced their re-attendance rates.

Few screening tools assess the need for social support among older ED patients.[6] The Comprehensive geriatric assessment (CGA) has been in use for this purpose since the 1980s. However, the CGA is time-consuming and, while used in EDs, is not always suitable for use in that environment.[7] The Triage Risk Screening Tool (TRST) is a screening questionnaire that has been validated in North America and parts of Europe and is moderately accurate, with good inter-rater reliability.[8,9] Other questionnaires, such as the Identification of Seniors at Risk (ISAR), are also used to screen older ED patients. However, the ISAR is self-administered and may not be appropriate in third world countries, where literacy rates among older people are likely to be relatively high. Both the TRST and ISAR identify seniors at risk of ED re-attendance and in need of social service intervention.

As populations in developing countries age, screening tools such as the TRST are becoming more relevant. By 2050, it is estimated that older adults (age ≥ 60 years) will account for 25% of the population in lower and middle income countries, such as those of the Commonwealth Caribbean, and that 75% of the world’s older population will live in developing countries by this time.[10]

This study was conducted at the Eric Williams Medical Sciences Complex, a tertiary teaching hospital in Trinidad. This Caribbean island has a population of 1.3 million, with health services provided by publicly and privately financed health facilities. As with many Caribbean countries, the population is ageing and patients older than 60 years currently account for approximately 10% to 20% of all ED admissions. Previous studies done among this population have demonstrated relatively high levels of social support for older people; however, some authors suggest that this is declining with time owing to disintegration of the extended family and an increase in the number of households in which both adults are working.[11,12]

This prospective, single-centre diagnostic study tested the predictive validity of the TRST in determining the need for social intervention among older ED patients in Trinidad.

Methods

The aim of this study was to determine whether the TRST was more accurate than routine assessment at detecting patients in need of social service intervention, compared with the “gold standard” of assessment by an experienced social worker.

Patients ≥ 65 years old who were being discharged from the ED were included. Patients were excluded if they were too unwell to take part in the study, refused to participate, or were admitted to the hospital.

A pilot of 20 patients was conducted prior to commencement of the main study. This pilot study assessed the acceptability and inter-rater reliability of the TRST in a local population of older people, when used by local ED physicians. The physicians and patients who participated in the pilot were not included in the main trial. Each patient was assessed independently by two ED physicians using the TRST. The inter-rater reliability of the tool was then estimated and the physicians were asked about the ease of use of the TRST and the time needed to complete it.

Prior to commencement of the main study in May 2012, ED physicians were educated on the benefits of social assessment in older patients. Following this, they were asked to use their clinical judgement to assess whether their older patients needed social service intervention (Phase 1 of the study). The physicians’ notes were written in free form in this phase of the study, which ran for 2 months. In Phase 2, which was undertaken over a 2-month period, patients were assessed using the TRST. There was a 4-week “wash-out” period between phases 1 and 2, to minimise the influence of Phase 1 on Phase 2 assessments. All physicians taking part in the study were fully qualified and registered doctors, working in the ED as emergency medicine residents or consultants.

Two weeks after discharge, each patient was assessed independently by an experienced social worker. The timeframe between physician and social worker assessment was chosen to allow the social worker time to contact patients while minimising the risk of any significant changes to the patients’ circumstances between assessments. The social worker was blinded to the ED physicians’ assessments. Social worker assessments covered the main areas associated with social risk, including lack of social support, financial insecurity, inability to perform daily activities, mental incapacity, loneliness, and depression.

The main outcome was analysed as a categorical dichotomous variable (need for intervention; yes/no). Patients with two or more positive questions on the TRST were assessed as in need of intervention. Using assessment by an experienced social worker as the gold standard, the sensitivity and specificity of the TRST versus routine physician assessment were calculated. P values of <.05 were taken as statistically significant. Sensitivity and specificity were compared using Pearson’s chi-squared test. A minimum sample size of 80 (40 per intervention) was required to detect a difference of 15% in sensitivity between the TRST and clinical assessment (with a power of 0.8 and significance level of 0.05). Data were analysed using IBM SPSS version 21 (IBM Corp., Armonk, NY, USA).

Ethical approval was obtained from the ethics committees of the University of the West Indies. Informed written consent was obtained from all participants.

Results

A total of 99 patients were enrolled in the study, 36 in the intervention arm (TRST) and 63 in the control arm (physician assessment) (Appendix 1). Patients were broadly similar in their age, sex, and clinical presentation (Table 1).

Table 1.

Demographic characteristics of sample.

	TRST assessment	Clinical assessment	Total	P value
Number of patients	36	63	99
Age (95% CI)	73.9 (71.4–76.5)	75.5 (73.6–77.4)	74.9 (73.4–76.4)	0.319
Sex, n (%)
Male	17 (47.2%)	28 (44.4%)	45 (45.5%)	0.836
Female	19 (52.8%)	35 (55.6%)	54 (54.5%)
Presenting complaint n (%)
Medical	24 (47.1%)	59 (60.8%)	83 (56.1%)	0.064
Surgical	1 (2%)	8 (8.2%)	9 (6.1%)
Trauma	5 (9.8%)	4 (4.1%)	9 (6.1%)
Not recorded	21 (41.2%)	26 (26.8%)	47 (31.8%)

Abbreviation: TRST, Triage Risk Screening Tool; CI, confidence interval.

In the pilot study, the TRST demonstrated a high inter-rater reliability (Cohen’s kappa 0.882) and was easy to use; the time needed for completion was less than 2 minutes.

Overall, 37 patients were found by doctors to be in need of social service support; a similar number was found by the medical social worker. This included 19 patients in the TRST arm of the study and 18 in the clinical assessment arm. In comparison, physicians assessed 22 patients in the TRST arm of the study as needing social service support and 15 in the clinical assessment arm. The sensitivity of the TRST was significantly higher than that of physician assessment (94.7% vs. 55.6%, P = 0.025) whereas its specificity was significantly lower (60.0% vs. 88.9%, P = 0.021). The area under the curve (AUC) of the receiver operator characteristic was not significantly different between the TRST and physician assessment (0.752 vs. 0.622, P = 0.311) (Table 2).

Table 2.

Diagnostic accuracy of the TRST compared with clinician assessment.

	TRST assessment	Clinical assessment	Total	P value
Number of patients	36	63	99
Sensitivity (95% CI)	94.7% (74.0%–99.9%)	55.6% (30.8%–78.5%)		0.025
Specificity (95% CI)	60.0% (26.7%–87.4%)	88.9% (76.0%–96.3%)		0.021
AUC (95% CI)	0.622 (0.455–0.789)	0.752 (0.580–0.924)		0.311

Abbreviation: TRST, Triage Risk Screening Tool; CI, confidence interval, AUC, area under the receiver operator characteristic curve.

Appendix 2 shows a subgroup analysis of the diagnostic accuracy of individual items of the TRST against medical social worker assessment. The most frequent positive item on the scale was “Professional Opinion”, which was further broken down into seven subcategories: nutrition/weight loss; failure to cope; sensory deficits; incontinence; medication issues; depression, and other.

Discussion

The TRST was more sensitive than clinical assessment for detecting the need for social service intervention among older people in our study. While this has previously been demonstrated in developed countries, it is the first time the TRST has been assessed in the Caribbean.[8] Whereas the AUC showed no significant difference between physician assessment and the TRST, it reflected a very low sensitivity but high specificity of physician assessment. High sensitivity is of primary importance for a screening tool because such tools attempt to rule out people who do not have the condition (in this case, those who do not need social service input), usually with the intention of providing further assessment for those who have been “ruled in” by the screening tool (in our case, those identified as needing social service input). In this context, the TRST clearly outperformed clinician judgement, with much higher sensitivity and a moderate level of specificity.

Our ED physicians’ ability to identify older patients in need of social intervention was very low, confirming the poor detection of psychosocial problems by ED physicians in general.[1] This is significant, as there is a link between inadequate social support and adverse clinical outcomes, including patient re-admission and mortality.[4] More structured screening for social risk is needed in our hospitals, as family support for older people in Trinidad may be declining with disintegration of the extended family. In addition, both adults in most households now work, leaving dependent older relatives more vulnerable to neglect.[11,12]

Interestingly, one item on the TRST accounted for most of the diagnostic accuracy of the questionnaire. The item entitled “Professional Opinion” had 90% sensitivity and 75% specificity for detecting patients in need of social service input. This aspect of the questionnaire provides an opportunity for physicians to use their clinical judgement to decide whether patients require further social service intervention. There is need for a larger study investigating the utility of this aspect of the TRST as a stand-alone tool to validate these findings.

Our pilot study demonstrated good inter-rater reliability of the TRST. The simplicity of administration is likely to have contributed to this.[8] It took less than 2 minutes to administer the TRST, unlike the CGA, and did not rely on self-administration, as does the ISAR.[13] However, there was a perception among doctors that this screening questionnaire created an additional burden of work, and if it is to be implemented more widely, these perceptions and attitudes will need to be taken into account.

Limitations

The sample size, and single-centre design of this study were major limitations, particularly in light of the high attrition rate of patients in the intervention arm of the study (those in whom the TRST was used for assessment). However, our a priori sample size calculation suggested the need for 40 patients in each arm of the study and our recruitment was very close to this target. While a relatively high proportion of study subjects were lost to follow-up, there was no significant difference in age, sex, or clinical presentation between patients who completed the study and those lost to follow-up (Appendix 3). In light of this, we believe that our findings remain valid, although there is a need to conduct a larger, multi-centre study to confirm the results.

Patients presenting to our hospital are broadly representative of the general population of the island, as it serves just under half of the population (approximately 450,000 people). Therefore, the single-centre study design is unlikely to have produced a significant selection bias. The before-and-after nature of the study may also have potentially biased results; the 4-week “wash-out” period between phases attempted to minimise this.

Other studies assessing the utility of the TRST used hospital re-attendance as the main outcome.[8] We chose the assessment of an experienced social worker as our gold standard because the less formal organisation of the health service in Trinidad made it difficult to verify patient re-attendance.

Conclusion

This study supported the validity of the TRST in a third world setting. We plan to conduct a larger multi-centre assessment of the TRST and to measure re-attendance rates among older ED patients, once the limitations mentioned above can be overcome.

Item	Positive responses n (%)	Sensitivity	Specificity
1. History of Cognitive Impairment	7 (19.44%)	38.6%	100.0%
2. Difficulty Walking/Transferring or Recent Falls	4 (11.11%)	21.1%	100.0%
3. Recent ED use or Hospitalisation	6 (16.67%)	31.6%	100.0%
4. Polypharmacy	1 (2.78%)	5.3%	100.0%
5. Professional Opinion	21 (58.33%)	90.0%	75.0%
Nutrition/Weight Loss	10 (27.78%)
Failure to Cope	4 (11.11%)
Sensory Deficits	4 (11.11%)
Incontinence	0 (0%)
Medication Issues	1 (2.78%)
Depression	7 (19.44%)
Other	6 (16.67%)
6. Lives Alone/No Available Care Giver	9 (25%)	36.8%	83.3%

Abbreviation: ED, emergency department.

	Completed study	Lost to follow-up	Total	P value
Number of patients	99	49	148
Age (95% CI)	74.5 (72.4–76.6)	74.9 (73.4–76.4)	74.8 (73.6–76.0)	0.675
Sex, n (%)
Male	45 (45.5%)	25 (51%)	70 (47.3%)	0.601
Female	54 (54.5%)	24 (49%)	78 (52.7%)
Presenting complaint n (%)
Medical	55 (55.6%)	28 (57.1%)	83 (56.1%)	0.127
Surgical	3 (3%)	6 (12.2%)	9 (6.1%)
Trauma	7 (7.1%)	2 (4.1%)	9 (6.1%)
Not Recorded	34 (34.3%)	13 (26.5%)	47 (31.8%)
Triage-positive patients, n (%)	40 (40.4%)	22 (45.8%)	62 (42.2%)	0.595

Abbreviation: CI, confidence interval.