Development of the Japanese version of the Intensive Care Unit Trigger Tool to detect adverse events in critically ill patients.

AIM: The Intensive Care Unit Trigger Tool (ICUTT) was developed to detect adverse events (AEs) in intensive care unit (ICU) patients. The purpose of this study was to determine the validity and reliability of the Japanese version of the ICUTT (ICUTT-J).
METHODS: The translation of ICUTT was carried out based on the guideline for translation of instruments. Subsequently, two review teams independently reviewed 50 patients' medical records using the ICUTT-J, and agreement regarding the presence and number of AEs was evaluated to ensure reliability.
RESULTS: The ICUTT-J was submitted to the authors of the original ICUTT, who confirmed it as being equivalent to the original version. The item-content validity index and scale-content validity index were 1.00 and 1.00, respectively. Interrater reliability showed moderate agreement of κ = 0.52 in terms of the presence of AEs and linear weighting of κ = 0.49 (95% confidence interval, 0.28, 0.71) in terms of the number of AEs.
CONCLUSION: This study's findings suggest that the ICUTT-J is valid and moderately reliable for use in ICUs.

Background

Adverse events (AEs) are defined as “unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalization, or that results in death.” Improving patient safety requires detection, quantification, and reduction of AEs. Several methods for detecting AEs in health‐care facilities are traditionally used, including medical record review, direct observation, voluntary reporting, morbidity and mortality conference, autopsy, and malpractice claims analysis. However, most of these methods were time‐consuming and costly. Furthermore, these methods focused on medical errors and were unsuitable for detecting AEs because medical errors were often not directly linked to patient harm.

The trigger tool (TT) has been used worldwide as an alternative to traditional methods for detecting AEs. The TT is a retrospective review of randomly selected patient medical records using “triggers” to identify possible AEs. Examples of triggers include new onset dialysis, pneumonia onset, and intubation. When a trigger as a clue to identify AEs is found, a careful reading of the medical records is carried out to confirm if an AE was related to the trigger event. For example, if a “new dialysis onset” trigger is found, there might be a contrast‐induced renal failure event. The TT is an ideal approach as it is both time‐ and cost‐effective and has higher sensitivity than conventional methods. Trigger tools have been modified and used in various types of health care, including general health‐care, surgical health‐care, pediatric care, emergency care, and intensive care. , , , ,

Previous studies reported that patients in the intensive care unit (ICU) were more severely ill and unstable and experienced more AEs than those in general wards. Up to 39.2% of ICU patients experienced an AE, and AEs increased the length of ICU and hospital stays. As the treatment provided in the ICU differs uniquely from that in the general ward, the triggers for detecting AEs are also specific. In fact, the Intensive Care Unit Trigger Tool (ICUTT) was developed specifically for the ICU (Table 1). However, the ICUTT has not been translated into Japanese.

Table 1

Original version of the Intensive Care Unit Trigger Tool

Triggers
Care module
C1	Positive blood culture
C2	Greater than 7 days in ICU
C3	Abrupt drop in Hg > 4
C4	Blood transfusion
C5	Venous Doppler for clot
C6	Radiologic tests to rule out emboli
C7	Code
C8	3 or more consultants
C9	Family complaints
C10	Pneumonia not present on admission
C11	Code status change in the unit
C12	Death
C13	Transfer to higher level of care
C14	Abrupt change of physician in charge
C15	Albumin < 2
C16	Readmission to ICU
C17	Nosocomial infection of any kind
C18	New onset dialysis
Procedure module
P1	Chest tube insertion
P2	Tracheostomy done
P3	Intubation/ reintubation
P4	Procedure associated event

Hb, hemoglobin.

The purpose of this study, therefore, was to translate the ICUTT into Japanese and to evaluate the validity and reliability of the Japanese version of the ICUTT (ICUTT‐J).

Methods

Development of the ICUTT‐J

Translation process

We started to translate the ICUTT on the basis of the World Health Organization proposal and guidelines about the translation of instruments, after obtaining permission from the Institute for Healthcare Improvement, where the ICUTT was originally authored. The translation process involved: (i) forward translation, (ii) expert panel, (iii) back‐translation, and (iv) pretesting (Fig. 1).

Fig. 1

Process of translating the Intensive Care Unit Trigger Tool from English to Japanese.

Forward translation

The guidelines recommend a translation method in a parallel fashion ; thus, forward translation was performed by three clinical researchers. The translators were a primary researcher, a certified ICU nurse, and a nursing teacher with a doctoral degree. After each had independently translated the ICUTT, we matched and integrated the translation to create the provisional Japanese version.

Expert panel

An expert group of four health‐care professionals (a patient safety researcher, a certified ICU nurse, an intensivist, and a nursing teacher with a doctoral degree) was established to evaluate the content validity of the translated ICUTT. Each expert derived the Content Validity Index (CVI) for each item of the ICUTT. To calculate the CVI, a four‐point scale of item relevance (1, not relevant; 2, somewhat relevant; 3, quite relevant; and 4, highly relevant) was used to decide the relevance of the item. An item‐level CVI of 0.78 or higher and a scale CVI average of 0.90 or higher are recommended as the evaluation criteria of content validity. Therefore, when the CVI fell below the evaluation criteria, the expert group discussed the differences between the original English version and the provisional Japanese version to reduce the variability through each expertise. The discussion was repeated until the CVI exceeded the criteria value for all items, and the final Japanese version was created.

Back translation

We used the back translation method to ensure linguistic and cultural equivalence. The final Japanese version was translated into English to make a final English version by a professional translator not familiar with the TT. Next, the final English was checked and commented on by the original author, and the items were corrected. Finally, the original author confirmed that the modified version was equivalent to the original version, and the translation of the ICUTT was completed (Table 2).

Table 2

Final English version of the Intensive Care Unit Trigger Tool, back‐translated from Japanese, and content validity

Triggers		Score †
		4	3	2	1
Care module
C1	Positive blood culture	4	0	0	0
C2	ICU stay >7 days	4	0	0	0
C3	Rapid decrease in Hb levels (by ≥4 g/dL)	4	0	0	0
C4	Transfusion	4	0	0	0
C5	Venous Doppler/venous ultrasound to examine for thrombi	4	0	0	0
C6	Radiography to rule out embolism	3	1	0	0
C7	Code (cardiorespiratory arrest, what is required etc., of the rapid response team)	2	2	0	0
C8	Consultation with several clinical departments (at least three departments/specialties)	1	3	0	0
C9	Complaints from the patient’s family	4	0	0	0
C10	Pneumonia onset following ICU admission	4	0	0	0
C11	Change the code status in the ICU (define or avoid “DNAR” etc.)	1	3	0	0
C12	Death	4	0	0	0
C13	Transition to more advanced treatment	3	1	0	0
C14	Sudden change of the attending physician	3	1	0	0
C15	Albumin <2 g/dL	3	1	0	0
C16	Readmission to the ICU	4	0	0	0
C17	Nosocomial infections (all types)	4	0	0	0
C18	Dialysis initiation	4	0	0	0
Procedure module
P1	Chest tube insertion	4	0	0	0
P2	Tracheostomy	4	0	0	0
P3	Intubation/reintubation	4	0	0	0
P4	Treatment‐related event	3	1	0	0

DNAR, do not attempt resuscitation; Hb, hemoglobin.

1, not relevant; 2, somewhat relevant; 3, quite relevant; 4, highly relevant. The level of agreement was set to no panel member scoring an item at <3.

Evaluation of the ICUTT‐J

Sample selection and data collection

This study was carried out at an ICU with 12 beds at a university academic hospital in Japan. The medical records of 50 patients admitted to the ICU between April and June 2018 were randomly selected with a randomization function. The exclusion criteria were: (i) age <18 years, or (ii) stay less than 48 h.

The patient demographic data were retrospectively collected from the electronic medical records. The demographic data included age, sex, mechanical ventilation status, admission category, length of ICU stay, and Acute Physiology and Chronic Health Evaluation II (APACHE II) score (scale range, 0–71) as a marker of illness severity. The APACHE II score is calculated using the most abnormal values obtained during the first 24 h of an ICU stay.

We formed two review teams (team I and team II) to evaluate the reliability of the ICUTT‐J. Each review team consisted of two intensive care nurses (primary reviewers) and one intensivist (secondary reviewer) (Table 3).

Table 3

Characteristics of reviewers of the Japanese translation of the Intensive Care Unit Trigger Tool

Reviewer	Team	Sex	Age (years)	Academic qualification	Clinical experience (years)	Intensive care experience (years)
Primary (nurses)
A	I	Male	28	Bachelor	5	5
B	I	Male	38	Doctoral	14	12
C	II	Male	33	Masters	10	10
D	II	Female	33	Bachelor	10	6
Secondary (physicians)
E	I	Male	33	Masters	8	6
F	II	Male	34	Masters	8	2

Review process

The review was carried out in two stages following the TT methodology. In stage 1, the primary reviewers independently reviewed patients’ medical records with the ICUTT‐J within 20 min per record. The primary reviewer screened for one or more of the 22 triggers and marked any such triggers on the ICUTT‐J worksheet and documented suspected AEs with a one‐ to two‐paragraph summary. Next, the primary reviewers within the same team compared their findings and reached agreement.

In stage 2, the secondary reviewer reviewed only the primary reviewers’ summaries without the patients’ medical records. The secondary reviewer made the final determinations about the presence, severity, and preventability of suspected AEs. The suspected AEs about which the secondary reviewer disagreed were discussed within each team.

Statistical analysis

Quantitative variables are reported as mean (standard deviation) and median (interquartile range), and qualitative variables as number (%). The incidence rate of AEs was calculated as the number of AEs per 1000 patient days and per 100 admissions.

Agreement between primary reviewers in terms of interrater reliability for each trigger was assessed using the Fleiss kappa statistic. Agreement between team I and team II in terms of interrater reliability for AEs was assessed using the Cohen kappa statistic. We used an unweighted kappa for nominal data and a linear‐weighted kappa for ordinal data and reported the results as coefficients with 95% confidence intervals (CI). The following interpretations were used for the kappa coefficients: poor, <0.0); slight, 0.00–0.20; fair, 0.21–0.40; moderate, 0.41–0.60; substantial, 0.61–0.80; and almost perfect, 0.81–1.00. All analyses were undertaken with IBM SPSS Statistics version 25 (SPSS Inc., Chicago, IL, USA).

Results

Translation

Linguistic and semantic problems arose in the translation process. The first was the items “Code” and “Code status change in the unit.” As it is uncommon to use the word “code” in Japanese, these items were translated into the surrogate words “Cardiopulmonary arrest” and “Change in treatment owing to development of cardiopulmonary arrest,” respectively. However, after discussing with the committee and the original author, we decide to use “code” so as not to lose the meaning of the term.

The final English version of the ICUTT was submitted to the original author, who then approved it. The final ICUTT‐J is shown in Appendix S1.

Patient characteristics

Of the 136 patients admitted to the ICU within the study period, 50 patients were randomly selected and reviewed. The mean (standard deviation) age was 64 (14) years, 32 patients (64%) were men, 34 patients (68%) were on mechanical ventilation, 15 patients (30%) underwent medical therapy, 26 patients (52%) underwent planned surgery, 9 patients (18%) underwent unscheduled surgery, the median length of ICU stay was 2.0 (1.0–6.5) days, and the mean APACHE II score was 18.6 (7.8).

Interrater reliability

Team I identified 13 unique AEs, team II identified nine unique AEs, and both teams identified eight AEs (Fig. 2). Table 4 lists the AEs identified by teams I and II. The incidence of AEs was 63 versus 51 events per 1000 patient days and 42 versus 34 events per 100 admissions, respectively. Table 5 shows the level of agreement between all primary reviewers (four nurses) on the presence of each trigger. The level of agreement on each trigger varied and the mean kappa was 0.463, corresponding to moderate interrater reliability.

Fig. 2

Venn diagram of the number of adverse events, identified by review teams I and II, among 50 patients admitted to a Japanese intensive care unit.

Table 4

Adverse events and agreement between review teams I and II to evaluate the reliability of the Japanese translation of the Intensive Care Unit Trigger Tool

Adverse events	Agreed team
Phlebitis	I
Urinary tract infection	I
Skin tear	I
Bleeding associated with medical device	I
Rash	I
Hemorrhage due to anticoagulants	II
Hypotension due to drug	II
Cardiac arrest	II
Peroneal nerve palsy	II
Renal impairment	II
Pressure ulcer	II
Hemorrhage due to surgery	II
Hypotension due to drug or procedure	II
Hypotension due to sick	II
Contrast‐induced nephropathy	I, II
Cerebral hemorrhage	I, II
Urinary tract infection	I, II
Cardiac arrest	I, II
Skin tear	I, II

Table 5

Interrater reliability in the Japanese translation of the Intensive Care Unit Trigger Tool

Triggers		Fleiss’ kappa	(95% CI)
Cares module
C1	Positive blood culture	0.539	(0.426 to 0.652)
C2	ICU stay >7 days	0.669	(0.556 to 0.782)
C3	Rapid decrease in Hb levels (by ≥4 g/dL)	0.655	(0.541 to 0.768)
C4	Transfusion	0.699	(0.586 to 0.812)
C5	Venous Doppler/venous ultrasound to examine for thrombi	0.662	(0.548 to 0.775)
C6	Radiography to rule out embolism	0.327	(0.213 to 0.440)
C7	Code (cardiorespiratory arrest, what is required etc., of the rapid response team)	0.592	(0.478 to 0.705)
C8	Consultation with several clinical departments (at least three departments/specialties)	0.344	(0.231 to 0.457)
C9	Complaints from the patient’s family	−0.005	(−0.118 to 0.108)
C10	Pneumonia onset following ICU admission	0.150	(0.037 to 0.263)
C11	Change the code status in the ICU (define or avoid “DNAR” etc.)	0.391	(0.278 to 0.504)
C12	Death	0.808	(0.695 to 0.921)
C13	Transition to more advanced treatment	0.590	(0.477 to 0.704)
C14	Sudden change of the attending physician	0.111	(−0.002 to 0.224)
C15	Albumin <2 g/dL	0.672	(0.559 to 0.785)
C16	Readmission to the ICU	0.198	(0.085 to 0.311)
C17	Nosocomial infections (all types)	0.411	(0.298 to 0.524)
C18	Dialysis initiation	0.715	(0.602 to 0.828)
Procudure module
P1	Chest tube insertion	0.210	(0.097 to 0.324)
P2	Tracheostomy	1.000	(0.887 to 1.113)
P3	Intubation/reintubation	0.291	(0.178 to 0.404)
P4	Treatment‐related event	0.150	(0.036 to 0.263)

CI, confidence interval; DNAR, do not attempt resuscitation; Hb, hemoglobin.

Table 6 displays the level of agreement between teams I and II on the presence and number of AEs. The kappa statistic on the presence of AEs was 0.520 (95% CI, 0.248, 0.792), and on the number of AEs it was 0.490 (95% CI, 0.276, 0.705), corresponding to moderate interrater reliability.

Table 6

Interrater reliability of adverse events between review teams I and II evaluating the Japanese translation of the Intensive Care Unit Trigger Tool

	Kappa coefficient	(95% CI)
Presence of adverse events †	0.520	(0.248 to 0.792)
Number of adverse events ‡	0.490	(0.276 to 0.705)

CI, confidence interval.

Unweighted kappa analysis.

Linear weighted kappa analysis.

Discussion

We developed the ICUTT‐J to detect AEs of critically ill patients. Complementary translation by multiple translators in a parallel fashion and an ICUTT‐J equivalent to the original version obtained by using the back‐translation method were appropriately produced in this study. The content validity of the triggers in the ICUTT‐J was supported by experts. The level of agreement on individual triggers varied with the kappa coefficient from low to high, and the mean kappa coefficient of the triggers was 0.463, with moderate interrater reliability.

Although there is no gold standard for identifying AEs, we were able not only to reinforce the validity of the translation but also to mitigate any cultural differences by assessing the content validity in an expert panel. However, there was a moderate interrater reliability for overall triggers, the presence of AEs, and the number of AEs, and only 26% (8 of 31) of the same AEs could be identified by the two teams. A previous study assessed the interrater reliability of the presence and number of AEs among five teams using the TT, and the reliability was similar to that of the present study with kappa = 0.45 (95% CI, 0.26, 0.63). Taken together, these results indicate that our translated ICUTT is the first Japanese version that has been tested and confirmed for validity and reliability. This ICUTT‐J can be used to quantify AEs in ICUs in Japan and to evaluate the effectiveness of interventions to improve patient safety. Moreover, this is an easy‐to‐use and low‐tech tool, so it can be implemented in clinical practice.

The interrater reliability regarding the triggers and AEs was not high because the criteria for determining AEs were unclear and depended on intrateam discussion. Among the triggers, C9, C14, and P4 were suggested to be particularly unreliable, for the following reasons: for C9, “Complaints from the patient’s family,” it was difficult to distinguish between a complaint and a comment from the patient's family; for C14, “Sudden change of the attending physician,” it was difficult to detect the trigger because it is common for patients to be seen by multiple departments in the ICU; and for P4, “Treatment‐related event,” the concept “treatment” is broad, so the sensitivity of the trigger was likely low. However, the interrater reliability could be improved by establishing detailed criteria for AE decisions and by training reviewers. Furthermore, in reviewing medical records for the detection of overall triggers and AEs, we must look at a vast amount of information, including vital signs variability, medication data, nursing records, and infection reports. Hence, triggers are likely to be missed when depending on the human eye alone to explore such vast and varied information. Electronic automatic detection of triggers or AEs is more accurate but would require purchasing power and equipment investment.

In comparison with the Global Trigger Tool, which is commonly used around the world, the ICUTT‐J does not have triggers for detecting adverse drug events (e.g., naloxone administration, or oversedation/hypotension). Thus, adverse drug events, which account for the majority of AEs, may be missed, and the incidence of AEs underestimated. Therefore, the use of the ICUTT and the TT for measuring adverse drug events together could allow the detection of more AEs.

Our study has some potential limitations. First, we were unable to detect undocumented AEs because the data were collected on the basis of the medical records according to the study design. Second, because the cause of harm had to be explored in the review process, hindsight bias could have occurred. Ideally, staff members not affiliated with the unit would have evaluated the AEs. Finally, this study was undertaken at a single ICU of one university hospital as an initial reliability test. In the future, research in multiple ICUs will offer basic data for testing of the ICUTT‐J.

Conclusion

We used the parallel translation method and back‐translation method to develop the ICUTT‐J for detecting AEs in intensive care patients. The content validity of the ICUTT‐J was supported, and its interrater reliability was moderate.

Disclosure

Approval of the research protocol: The institutional review board of the University of Tsukuba Hospital approved this study.

Informed consent: N/A.

Registry and the registration no. of the study/trial: H30‐108.

Animal studies: N/A.

Conflict of interest: None.

Ethics approval and consent to participate

The institutional review board of the University of Tsukuba Hospital approved this study (H30‐108). Informed consent from each patient was waived because the study design was retrospective and we used anonymized patient data.

Appendix S1. The Japanese version of the Intensive Care Unit Trigger Tool.

Click here for additional data file.