Validation of the Cipto Triage Method: A Single-Centre Study from Indonesia.

PURPOSE: A national referral hospital in Indonesia developed a three-category triage acuity method called the Cipto Triage Method (CTM) for emergency departments (ED) in developing countries. This was a validation study to assess the performance of the triage method.
METHODS: This cohort, retrospective, single-centre study was conducted in the ED of Cipto Mangunkusumo Hospital that receives approximately 30,000 patient visits per year. The ED medical records throughout the year 2017 were randomly selected as the study sample. Completely written forms of triage and ED initial assessment were included in this study. Validation of the CTM decision was done by using expert panel opinion as reference standard, and also using surrogate conditions such as patient outcome for hospital admission and in-hospital mortality.
RESULTS: There were 1348 samples assigned to the following three categories: resuscitation (14.9%), urgent (63.8%) and non-urgent (21.3%). Overall accuracy was more than 80%, positive predictive value and negative predictive value for resuscitation category were 99% (95% confidence interval [CI], 96.5-99.9) and 96.9% (95% CI, 95.7-97.8), respectively. Resuscitation category had a relative risk (RR) for admission of 1.341 (95% CI, 1.259-1.429) and a RR for mortality of 4.294 (95% CI, 3.180-5.799). Undertriage increases the risk of mortality compared to correct triage (RR, 3.1; 95% CI, 2.11-4.54).
CONCLUSION: CTM has a good criterion and construct validity; it is also easy to understand and can accommodate a simple ED design in the majority of hospitals in Indonesia.

Introduction

Triage is an important process in the emergency department (ED).1 It helps the emergency healthcare providers in prioritising various patients present in the ED with different clinical acuities.2,3 Sorting the patients based on their clinical severity or urgency will increase the safety of the patients and healthcare providers, promote effective utilisation of ED resources, and increase ED patient flow.4,5

Prominent ED triage methods have already been validated and are practised worldwide.6–9 Triage method specifically designed for developing countries is still rare, and some of them have been validated such as the South African Triage Scale (SATS) and Turkey Triage System.10,11

Validation of triage performance in various countries showed a moderate validity and reliability on the five-level triage acuity.12,13 It is also understood that different country settings (developed-, developing-, and under-developed countries), or country policies on healthcare systems contribute to the different characteristics of the patients seeking emergency care.3 Thus, in Indonesia, adjustment of triage method needs to be performed, because triage systems will work better on populations they were developed for.14

The ED of Cipto Mangunkusumo Hospital (CMH) had been practising triage since 1986.15 Initially, the triage decision-making was done using the common sense. Moreover, there were no proper education or training for medical doctors or nurses before they undertook the role of a triage officer. Since 2012, the ED practiced triage based on the ATS.16,17

In mid-2016, Cipto Triage Method (CTM) was developed, consisting the following three categories: resuscitation, urgent and non-urgent. The definitions of these three categories are derived mainly from the ATS categories. ATS categories 1 and 2 were merged into resuscitation category, while categories 4 and 5 were merged into non-urgent category. ATS category 3 remained as a single category (urgent).

To evaluate the sensitivity and specificity of the CTM after 1 year of implementation, we conducted a validation study based on the principles of diagnostic research to assess CTM’s criterion and construct validity. Our hypothesis for this study was that CTM has good sensitivity, specificity, and predictive value to predict ED patients’ clinical severity and urgency in Indonesia.

Methods

Design and Setting

This cohort retrospective, single-centre study was conducted between January 2017 and- December 2017 in the ED of CMH, an academic and referral hospital in Jakarta, Indonesia. The ED receives about 30,000 patient visits annually.16 The characteristics of the patients’ conditions range from simple tropical disease infection to chronic and terminally ill diseases. The ED team consisted of multidisciplinary specialised physicians and was managed by a specialist trained in emergency medicine. The CTM had been used for at least 6 months before the study initiation. The triage officers were nurses who have been practising triage more than 1 year. Triage training of CTM was performed continuously once a month in the ED continuing medical education forum. This study was approved by the institutional ethical board of the hospital. All patient data was treated with confidentiality, in accordance with the Declaration of Helsinki.

Data Collection

Every month, there were 128 ED medical record extracted randomly from the database, the sampling frame was total ED patients per month, and sample selection was done by using randomization application. Stratified random sampling was not performed in order to reflect the distribution of patients admitted to the ED of CMH according to their clinical severity level. All age and medical conditions were eligible for this research. Triage and initial assessment forms were collected from the extracted medical records. Incomplete forms-, or patients who left without being examined were excluded from this study. Formal sample size calculations based on power assumptions for diagnostic modelling cohort research did not exist; thus, we defined the sample size according to the previous diagnostic research in triage by using relative risk assumption = 2, power= 80%, confidence level 95%, and the sample size result was 960.11,18,19

Study Protocol

The triage process of CTM was performed in a dedicated triage room at the main entry of the ED building by a nurse per shift, and one shift lasted for 8 hours. All patients who presented to the ED will undergo single triage process followed by disposition to the treatment zone. The CTM aims to determine patient priority based on their clinical severity and urgency. Assessment of the patients’ clinical severity and urgency was based on syndromic approach (chief complaint, comorbid conditions, or mechanism of injury)-, and vital signs (mental status, peripheral pulse rate and quality, respiratory rate, capillary refill time, temperature). Example of CTM decision making process is available in and . Triage process was not designed to establish a diagnosis or further assess any complications of the patient’s condition.16 No scoring system was utilised in prioritising the patient’s condition; instead, clinical judgement was made qualitatively using the mandatory information to establish triage decision.

Every triage decision has its significance. Resuscitation category means that the patient has a critically ill condition, therefore he/she needs immediate intervention from the emergency medical team. Any delay in diagnostics and treatments could decrease the patient’s survival potential. Urgent category means that the patient does not have obvious critical conditions, but has symptoms or conditions that, if not treated early, can result in morbidity or deterioration. The patient can wait until 30 minutes to get medical assessment and treatment. Non-urgent category means that the patient has signs and symptoms that are not associated with severe or urgent conditions, but he/she could still have the diagnostic study or treatment but could wait up to 1 hour to receive medical treatment.

Triage decision that was made by a triage officer is called index test. Triage decision from the selected medical record was evaluated by the expert panels, in which their decision was the reference standard of the triage decision (criterion validity). Expert panels consist of three specialists (internist, anaesthesiologist, and orthopaedic surgeon). All of them already had additional 1 year of clinical training in emergency medicine after finishing their respective specialist training. They practised daily as emergency physicians in the ED of CMH since early 2016.

The expert panels evaluated retrospectively the triage decision based on chart reviews of triage forms and initial assessment forms.20 Panel consensus was made after individual adjudication based on whether they agreed with the triage decision or not and established clinical diagnosis, and they also provided correction of the triage decision. Expert panels and triage officers were kept blinded from the patient outcome (admission and mortality). The outcome was identified based on information from electronic medical record.

Data Analysis

Medical records were collected by trained general physicians. They extracted the following data: patient demography, the actual triage decision, the corrected triage decision by the expert panels, and patient outcome. This study calculated triage performance (sensitivity, specificity, predictive value, and likelihood ratios) with 95% confidence interval (CI) using the normal approximation. Statistical analysis was performed using chi-squared test to assess the relative risk (RR) of each triage category to the surrogate outcomes, that is, hospital admission and in-hospital mortality. Stata version 15 software was used to perform the analysis. Microsoft Excel was used to perform confidence interval for proportion of accuracy, undertriage, and overtriage. We presented this article following guideline from Standards for Reporting of Diagnostic Accuracy Studies (STARD) ()

Results

There were 1536 ED medical records collected, and 1348 were included in this study since the triage and initial assessment forms were written completely (). Table 1 illustrates the characteristics of the patients. Most patients who presented to the ED had non-traumatic cases, and the most common triage was urgent category. No adverse event was observed during the collection of index test data or during expert panel assessment.

Table 1

Characteristic of ED Patients

Demography	N (%)
Sex
Male	742 (55)
Female	606 (45)
Total	1348
Age interval
0−28 days	21 (1.5)
>28 days−18 years old	323 (24)
>18−60 years old	734 (54.5)
>60 years old	270 (20)
Total	1348
Trauma related
Yes	215 (15.9)
No	1132 (84.1)
Actual triage levels
Resuscitation	201 (14.9)
Urgent	860 (63.8)
Non-urgent	287 (21.3)
Hospital admission
Yes	940 (69.7)
No	404 (29.9)
Dead on arrival	4 (0.4)
In-hospital mortality
Yes	139 (10.3)
No	1205 (89.4)
Dead	4 (0.3)

Distribution of the triage decision is shown in Table 2, the index test was actual triage done by triage officer and reference standard was panel judgement assessment.

Table 2

Distribution of Triage Decision Using Cipto Triage Method Based on Panel Judgement and Triage Officer and Its Accuracy

Categories	Panel Judgement			Total	Accuracy(%)	95% CI
Triage Officer	Resuscitation	Urgent	Non-Urgent
Resuscitation	199	2	0	201	97.2	96.3–98.1
Urgent	34	716	110	860	88.1	86.3–89.8
Non-urgent	2	15	270	287	90.6	89.0–92.1
Total	235	733	380	1348
Overall undertriage					5.6	4.3–6.8
Overall overtriage					8.3	6.8–9.7

Abbreviation: CI, confidence interval

Expert panel review showed that the overall accuracy rate of the triage decision was more than 80% in every category. There was more overtriage compared to undertriage (Table 2). Accuracy was measured from each triage category using 2x2 table () derived from Table 2.

Table 3 displays triage performance compared to the standard reference. Most triage category had good to excellent sensitivity and specificity except for the sensitivity of the non-urgent category.

Table 3

Diagnostic Performance of CTM (Criterion Validity)

Triage Category	Results	95% CI
Resuscitation
Sensitivity (%)	84.7	79.4 −89
Specificity (%)	99.8	99.4 −100
PPV (%)	99	96.5 −99.9
NPV (%)	96.9	95.7 −97.8
LR+	471	118 −1884
LR-	0.153	0.11 −0.21
Non-urgent
Sensitivity (%)	71.1	66.2 −75.6
Specificity (%)	98.2	97.2 −99
PPV (%)	94.1	90.7 −96.5
NPV (%)	89.6	87.6 −91.4
LR+	40.5	25.1 −65.1
LR-	0.29	0.25 −0.34

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; LR-, negative likelihood ratio; CI, confidence interval.

This condition happened because there were considerable number of the non-urgent patients overtriaged to urgent category. Each triage category had excellent PPV and NPV.

Construct validity of each triage category was made by using admission and in-hospital mortality as surrogate markers (Table 4). Risk of admission was similar between resuscitation and urgent category, and risk of mortality was higher in resuscitation category compared to that of urgent and non-urgent (Table 4) categories. Chi squared analysis showed undertriage increased the risk of mortality with a RR of 3.1 (95% CI, 2.11 −4.54; p = 0.000). Overtriage had a RR to mortality of 0.247 (95% CI, 0.089 −0.848; p = 0.000).

Table 4

Construct Validity of Triage Category Compared to the Risk of Admission and in-Hospital Mortality

Triage Category	Hospital Admission	Relative Risk for Admission	In-Hospital Mortality	Relative Risk of in-Hospital Mortality
Resuscitation	176 (18.7%)	1.341(95% CI, 1.259 −1.429)	59 (42.4%)	4.294 (95% CI, 3.180 −5.799)
Urgent	664 (70.6%)	1.354 (95% CI, 1.243–1.475)	76 (54.7%)	0.679 (95% CI, 0.496 −0.930)
Non-urgent	100 (10.7%)	0.438 (95% CI 0.373–0.515)	4 (2.9%)	0.109 (95% CI, 0.041 −0.292)

Abbreviation: CI, confidence interval.

Discussion

The triage systems should be designed based on the population characteristics, and also resource capability in the ED. Triage method created in the developed countries may need to be modified when applied to developing countries or poor resource settings. Development and application of the triage system should be followed by validation to make sure the methods are effective and efficient.21

The validation of triage should follow diagnostic research principles. Twomey suggested that triage validation is conducted by combining the criterion validity (using reference standard such as expert panel decision) and structural validity (eg admission and mortality).14

Consideration to divide CTM priority into three categories as oppose to five in the developed nations was based on its simplicity and easiness to understand for the ED staff. Most of the patients in Indonesia who presented to the ED are not via the ambulance services; hence, condition is fully undifferentiated when triage was performed.

These considerations are also applied by the World Health Organization. The organisation already developed three categories of emergency triage for children in developing country and poor resources settings.3 Research in other developing countries showed that three-triage category such as in Turkey had better performance than 5 categories such as in South Africa.10,11,22 One of the reasons why three-triage category was developed was that it is in line with the simple ED zone design due to space restrictions, conditions that can be found in majority of non-referral hospitals in Indonesia.

Diagnostic performance of CTM was excellent, and its overall accuracy was more than 80%, which is higher compared to Turkey Triage System (inter-rater agreement, 0.725; 95% CI, 0.68 −0.77) and Emergency Severity Index (overall accuracy is 59.2%).11,23

Sensitivity and specificity of the resuscitation category are comparable with the relevant three-level triage category from Turkey (red zone sensitivity, 80.9%; specificity, 92.7%).11 Overall triage accuracy of CTM was better than that of the previous five level acuity triage implemented in CMH before 2016. A study reported in 2016 regarding triage in CMH using ATS method stated that the ATS method had good performance in predicting 24 -hour mortality (area under the curve [AUC] = 0.787; 95% CI, 0.69 −0.88) and fair performance in predicting 7-day mortality (AUC = 0.66; 95% CI, 0.59 −0.72). Positive likelihood ratios of ATS method in CMH regarding mortality were 11.36, 1.11, 1.69, 0.4, and 0.23 for level 1 to 5 categories, respectively.24

Another triage category from developing country such as SATS that has five triage categories showed 91% sensitivity and 54.5% specificity for resuscitation category.22 Compared to that of MTS that showed, 0.47 sensitivity and 0.84 specificity for adult patients and 0.65 sensitivity and 0.83 specificity for children.25

Cipto triage system has more overtriage compared to undertriage. Overtriage leads to overuse of ED resource, but is better for safety. In ESI, undertriage rate was 27.6%, and overtriage rate was 13.2%.23 In Botswana that use SATS, overtriage rates were 38.4%, while undertriage rate was 16%;22 meanwhile, Turkey triage for under- and overtriage occurred at similar rates in 11% and 10.7% of cases.

We also evaluated the construct validity based on admission and mortality in one episode of ED visit. In 2017, there were a total of 69.7% admission rates to ward and intensive care unit (ICU), and 10.3% in-hospital mortality. In Botswana, there were 41.9% admission for non-ICU, and 0.19% of the patients died in the ED. Turkey had 15.5% patient admitted from the ED and 2.8% in-hospital mortality.11,22

Cipto triage showed that the higher the triage level, the higher the risk of mortality. This is relevant to other three triage category in Turkey that showed the RR for admission were 10.2 (95% CI, 4.2–24.9; p < 0.001), and 3.58 (95% CI, 1.45–8.83; p = 0.006) for resuscitation and urgent category, respectively. The resuscitation category had a significantly higher risk of in-hospital mortality with a RR of 30 (95% CI, 1.8–499.2; p = 0.018) compared to the non-urgent category.11

Triage method from developed country such as MTS showed similar pattern according to the admission odds ratio (level 1, 21; level 2, 9; level 3, 4.77) and mortality odds ratios were 75.9% for level 1, 20.7%, for level 2, 3.4% for level 3, and 0 for levels 4 and 5.12,13

Overcrowding and access block contribute to the lower risk of admission in our resuscitation category compared to the urgent category. Some of the patients in the resuscitation zone were unexpectedly boarded in the ED because the ICU is full. While waiting for admission, some of them were dead and were not able to complete the admission process.

Evaluation of triage agreement by using medical record could lead to bias, because evaluators only have information based on what is written in the forms and could not perform visual assessment. Triage assessment was done by nurse, and the expert panel only included physicians, this could contribute to limitation in judgemental process. This study was also not generalisable for all hospitals in Indonesia, since it was performed in a single hospital. Similar practice and subsequent validation studies should be conducted in the ED of various levels of hospital in Indonesia. Further prospective and reliability study should be conducted in a validation study of the CTM for specific age (neonates, paediatric, geriatric).

As a tertiary referral centre in Indonesia, the ED of CMH has different patient characteristics compared to other EDs in non-referral centres or in rural areas in Indonesia. Proportion of exacerbation is significantly higher in chronic non-communicable disease in the ED of CMH compared to communicable disease.

Conclusion

Cipto Triage Method showed excellent criterion and construct validity and can be used as predictive tool for patient’s clinical severity or urgency in emergency department. Larger studies should be conducted to assess its performance in other hospitals in Indonesia.