The ability of triggers to retrospectively predict potentially preventable adverse events in a sample of deceased patients.

Several trigger systems have been developed to screen medical records of hospitalized patients for adverse events (AEs). Because it's too labor-intensive to screen the records of all patients, usually a sample is screened. Our sample consists of patients who died during their stay because chances of finding preventable AEs in this subset are highest. Records were reviewed for fifteen triggers (n = 2182). When a trigger was present, the records were scrutinized by specialized medical doctors who searched for AEs. The positive predictive value (PPV) of the total trigger system and of the individual triggers was calculated. Additional analyses were performed to identify a possible optimization of the trigger system. In our sample, the trigger system had an overall PPV for AEs of 47%, 17% for potentially preventable AEs. More triggers present in a record increased the probability of detecting an AE. Adjustments to the trigger system slightly increased the positive predictive value but missed about 10% of the AEs detected with the original system. In our sample of deceased patients the trigger system has a PPV comparable to other samples. However still, an enormous amount of time and resources are spent on cases without AEs or with non-preventable AEs. Possibly, the performance could be further improved by combining triggers with clinical scores and laboratory results. This could be promising in reducing the costly and labor-intensive work of screening medical records.

Introduction

Unintentional medical harm received increased attention during the past years (Rutberg et al., 2014, Najjar et al., 2013, Kennerly et al., 2013, Zegers et al., 2009, Mull et al., 2015, Kurutkan et al., 2015, Farup, 2015, Doupi et al., 2013, Goodman et al., 2011, Baines et al., 2015).

Several years have passed since the report “to err is human” was published, in which the need for a safer health care system was emphasized. Fifteen years after this initial report, a recent update stressed the importance of continuing efforts to improve patient safety (Anon., 2015). Also, a recent Dutch paper (2013) showed that an average of 12% of patients who died in the hospital still experienced care related injury, which sometimes even contributed to the death of the patient (Langelaan et al., 2013a). It is, therefore, important to identify AEs and to determine the risk factors related to their occurrence, in order to reduce harm to patients and improve the quality of care (Hwang et al., 2014).

It is time-consuming to screen all records for the presence of AEs. Therefore, “triggers” that can be easily identified in the medical records by well-trained nurses in a relatively short time, have been developed. Several trigger systems were created to screen medical records of hospitalized patients for AEs. These triggers are indicators or characteristics of the disease course, known to be often associated with AEs (Resar et al., 2003). The fact that cases can be missed, is generally accepted because investigating all records would be too time and cost-consuming in relation to the positive effect of screening. A well-known trigger system is the Global Trigger Tool (GTT), developed by the Institute for Healthcare improvement (IHI). Also, the system from the Harvard medical practice study (HMPS), with a smaller set of triggers, is often used (Griffin and Resar, 2009, Brennan and Leape, 1991). For the aforementioned trigger systems, the positive predictive value has been determined in several studies (Kennerly et al., 2013, Unbeck et al., 2013). However, the part of the quality cycle where medical records are scrutinized is still time-consuming and costly. Therefore, it is important to minimize the number of false positive results without increasing the number of false negative results. Because it is too labor-intensive to screen the records of all patients, usually the screening is performed in a sample.

Our sample consists of records of all patients who died during their stay. Therefore, in this study, we used a slightly adapted list of triggers. Examples of cases are illustrated in appendix I, to explain some of the most used triggers and the ones which needed extra explanation.

It closely resembles the trigger list from the HMPS, but adjusted to be applicable to medical records of deceased patients. Admittedly, AEs in diseases with negligible mortality but with an unfavorable outcome or hospitalization in departments with low mortality (e.g. ENT, ophthalmology, obstetrics, pediatrics etc.) would escape the opportunity for improvement of care using this sample. Although there are conflicting reports, the most recent and largest study concerning detection of preventable AEs showed that this is particularly effective in deceased patients (Baines et al., 2015, Hogan et al., 2012, Dunn et al., 2006). However, patients who die in hospitals are usually older with more comorbidities and therefore studies in these patients are not generalizable to the average hospital patient. In this study, we assumed that the probability of detecting (serious) AEs was highest in this subset of patients. This would then result in a manageable number of cases to be scrutinized by the committee, but still acquiring a fair overall estimation of the quality of treatment and causes of treatment failure. We wondered whether the positive predictive value (PPV) of the trigger system in deceased patients was acceptable compared to other study samples. Therefore, we analyzed our database with information on triggers and AEs of all in-hospital deaths in the past years. In addition to this, we performed supplementary analyses in an attempt to optimize the current trigger system.

Methods

This study was performed at the Maastricht University Medical Centre (MUMC +), a teaching hospital in the south of the Netherlands. The medical records used in this study included all inpatient wards including children's. The study protocol was approved by the Ethics Committee of our hospital. We also checked whether patients ever expressed objections against the use of their data for research (this is recorded in a special database in the hospital). If so, their data were excluded. However, none of the patients that were in this sample, did so.

The medical records of all patients who died in our hospital between January 1st, 2012 and January 1st, 2015 were explored by a team of trained nurses for the presence of triggers. Subsequently, a committee consisting of medical specialists from all major disciplines analyzed the records to search for AEs. Both the screeners and the specialists were not time restricted. All results were saved using software provided by Medirede®, Clinical File Search version 3 (Mediround BV, 2015). This software was designed to store these data in a clear and easily accessible way. An AE was defined as an unintended outcome arising from the (non)-action of a caregiver and/or the health care system with damage to the patient resulting in temporary or permanent disability or death of the patient (Wagner, 2007). If a potentially preventable AE was suspected, this was discussed with the involved medical department. Finally, the committee decided on the definite presence of an AE and its potential preventability. For the purpose of this study, we used the committee result as a gold standard for AEs. We did not evaluate the effect of hindsight bias, inter- and intrarater reliability.

The starting point of our trigger system was the HPMS list, and we hypothesized that this list would be redundant in deceased patients (Brennan and Leape, 1991). Trigger 1 (patient was admitted before (< 12 months) for a reason related to the current admission) was adapted to a shorter period (< 3 months) because analysis of previous years showed this trigger was not discriminative for potentially preventable AEs. The 12-month cut-off contained a large number of patients with planned chemotherapy or planned second stage operations. Two other triggers were not applicable in a deceased population.

To create a simplified method of triggering, we calculated the positive predictive value (PPV) for the combination of triggers that can be detected by a computer search of the medical records (trigger 1, 4 and 5) and a combination of three triggers that generate the highest number of potentially preventable AEs (trigger 4, 7 and 8). Here, we only looked at the PPV for potentially preventable AEs as the outcome. The PPV of individual triggers was calculated as the rate at which a trigger was associated with an AE, both potentially preventable and not preventable (Naessens et al., 2010). Furthermore, we calculated risk scores for an AE in patients with a trigger taking the patient characteristics into account. These risk scores could then be used, to generate cut-off points leading to a smaller selection of records with a varying number of AEs depending on the chosen cut-off point.

Statistical analysis

Descriptive statistics are used to describe the general characteristics of the screened medical records and the triggers used in this retrospective analysis.

Chi-square tests and independent t-tests were performed to determine the differences between the groups of patients who experienced an AE during their stay, compared to the group of patients who did not develop an AE.

Furthermore, multivariable backward logistic regression analyses (with classification cut-off 0,5) were performed for three scenarios, the first one to detect only computer detectable triggers. The second model contains all 15 triggers to identify the trigger with the highest odds for AEs and potentially preventable AEs. The last model was used to determine the contribution of patient characteristics to the occurrence of AEs to identify possible additional factors that could improve the selection of cases with AEs.

The presence of an AE was used as the dependent variable. Independent variables were: origin (coming from another hospital yes/no), emergency admission, age, gender, admission specialism, and length of stay (in days). Referred by emergency admission was applicable when the patient was admitted via the emergency ward. Admission specialties were divided into surgical (e.g. urology, vascular surgery, gynecology etc.) and medical departments (e.g. internal medicine, gastroenterology, cardiology, pulmonology, rheumatology, pediatrics etc.). For evaluating the additional value of including the patient characteristics in this last logistic regression model (model 3), we have calculated the probability of every individual of having an AE, given the fact, one or more triggers would be positive. In this model, the following patient characteristics were included: urgent admission, origin, age, gender, length of stay and admission specialism. The logistic regression model yields a continuous outcome, i.e. the predicted probability ranging from 0.0 to 1.0. However, the model will likely be used to classify patients into high risk versus low risk, or positive versus negative. To aid in choosing the right cut-off point for classification, we evaluated 6 different cut-off points. By computing test characteristics for each cut-off point, one cut-off point can be chosen that fits the need for either ruling in or ruling out an adverse event.

Analyses were executed using IBM SPSS Statistics version 23 (IBM Corporation, 2015), a p < 0.05 was considered statistically significant.

Results

The medical records of 2182 patients were investigated (shown in Fig. 1). The general characteristics of these patients are shown in Table 1. Men were significantly younger than women (p = 0.004) and they had a significantly higher chance of experiencing an AE (p = 0.021). The length of stay is significantly longer in patients with an AE compared to patients without an AE (p < 0.001), whereas preventable and non-preventable AEs don't differ concerning the length of stay (p = 0.911).

Fig. 1

Flowchart of the medical record analysis in this study.

Table 1

General characteristics of the studied population (patients deceased during hospitalization).

Variable	Total n = 2182 (%)	Trigger present N = 1262 (%)	AE present N = 589 (%)	Potentially preventable AE present N = 217 (%)	Preventable AE potentially contributing to patient death N = 206 (%)
Gender
Male	1220 (56)	743 (59)	353 (60)	120 (55)	115 (56)
Female	962 (44)	519 (41)	236 (40)	97 (45)	91 (44)
Age	69.6 (95%CI 68.8–70.5)	69.2 (95%CI 68.1–70.2)	69.6 (95%CI 68.2–70.9)	71.5 (95%CI 69.3–73.6)	71.5 (95%CI 69.4–73.7)
Length of stay (average days)	12.9 (95%CI 12.2–13.6)	17.1 (95%CI 16.0–18.2)	20.9 (95%CI 19.0–22.9)	21.1 (95%CI 17.6–24.5)	21.3 (95%CI 17.7–24.9)
Urgent admission
Yes	1546 (71)	1184 (94)	524 (89)	196 (90)	185 (90)
No	636 (29)	78 (6)	65 (11)	21 (10)	21 (10)
Transferred from another hospital
Yes	186 (9)	104 (8)	47 (8)	19 (9)	18 (9)
No	1996 (91)	1158 (92)	542 (92)	198 (91)	188 (91)
Admission specialism
Surgical	455 (21)	351 (28)	257 (44)	107 (49)	106 (51)
Medical	1727 (79)	911 (72)	332 (56)	110 (51)	100 (49)

The number of patients admitted to the medical departments is higher than to surgical departments. However, the percentage of patients with an AE is significantly higher at the surgical departments. (p < 0.001).

The PPV of our trigger system is 47%, 589 of the 1262 positively triggered cases had an AE. 217 of the 1262 (17%) triggered cases were considered potentially preventable. Table 2 shows the distribution of the individual triggers and their PPVs.

Table 2

Results of individual triggers and AEsc.

Triggers	Number of records (% of total)b	Non preventable AE	Potentially preventable AE	Total number of AE combined (preventable and non- preventable)d	Percentage of total number of AEs	PPV potentially preventable AE (95%CI)a	PPV AE combined (preventable and not preventable) (95%CI)a
1. Unplanned readmission < 3 months	379 (17.4)	97	47	144	23.0	0.33(0.25–0.40)	0.38(0.33–0.43)
2. Hospital incurred patient injury	116 (5.3)	44	32	76	12.2	0.42(0.31–0.53)	0.66 (0.57–0.74)
3. Adverse drug reaction	77 (3.5)	34	13	47	7.5	0.28(0.14–0.41)	0.61(0.50–0.72)
4. Unplanned transfer to ICU	441 (20.2)	158	107	265	42.4	0.40(0.34–0.46)	0.60(0.56–0.65)
5. Unplanned return to the operating room	173 (7.9)	75	67	142	22.7	0.47(0.39–0.55)	0.82(0.76–0.88)
6. Unplanned removal or damage to an organ during surgery	76 (3.5)	29	29	58	9.3	0.50(0.37–0.63)	0.76(0.67–0.86)
7. Healthcare related infection or sepsis	509 (23.3)	176	99	275	44.0	0.36(0.30–0.42)	0.54(0.50–0.58)
8. Other complications such as CVA/lung embolism/acute myocardial infarction/TIA	350 (16.0)	131	77	208	33.3	0.37(0.30–0.44)	0.59(0.54–0.65)
9. Development of neurological deficit	129 (5.9)	54	22	76	12.2	0.29(0.19–0.39)	0.59(0.50–0.68)
10. (Initial) unexpected and/or sudden death, absence of terminal care	266 (12.2)	84	56	140	22.4	0.40(0.32–0.48)	0.53(0.47–0.59)
11. Cardiac or respiratory arrest	197 (9.0)	72	50	122	19.5	0.41(0.32–0.50)	0.62(0.55–0.69)
12. Injury related to abortion or delivery	–	–	–	–	–	–	–
13. Dissatisfaction with care	59 (2.7)	19	11	30	4.8	0.37(0.18–0.55)	0.51(0.38–0.64)
14. Documentation indicating litigation	14 (0.6)	3	6	9	1.4	0.67(0.28–1.05)	0.64(0.36–0.93)
15. Other patient complications	224 (10.3)	57	35	92	14.7	0.38(0.28–0.48)	0.41(0.35–0.48)

CI is confidence interval.

Total number of records is 2182.

The same AE can be found with different triggers, it was not possible to determine which trigger in this case was related to the AE.

Total number of AEs is 625.

In the triggered cases, the number of unique triggers occurred with a mean of 2.39 per patient (95%CI 2.31–2.46). Finding more triggers gradually increased the likelihood of finding a (potentially preventable) AE. In total 625 AEs were found in 589 records. 33 records showed two or more AEs and 35% of the AEs were potentially preventable. This is shown in Table 3.

Table 3

Number of triggers and (potentially preventable) AEs with corresponding PPV.

Number of triggers (n)	Records (n)	AE present (n)	PPVb for AE (95%CI)a	Potentially preventable AE present (n)	PPVb for potentially preventable AE (95%CI)a
1	440	134	0.30(0.26–0.35)	45	0.34(0.25–0.42)
2	330	136	0.41(0.36–0.47)	52	0.38(0.30–0.47)
3	224	127	0.57(0.50–0.63)	38	0.30(0.22–0.38)
4	156	106	0.68(0.61–0.75)	42	0.40(0.30–0.49)
5	74	55	0.74(0.64–0.85)	26	0.47(0.34–0.61)
6	26	22	0.85(0.7–0.99)	9	0.41(0.19–0.63)
7	9	7	0.78(0.44–1.12)	4	0.57(0.08–1.07)
8	2	1	c	1	c
9	1	1	c	0	c

CI is confidence interval.

PPV is positive predictive value,

Could not be calculated due to the small number of records with 8 or 9 triggers.

Detecting AEs with a simple computer algorithm (model 1)

For this analysis, only computer detectable triggers were selected (trigger 1, 4 and 5). 777 cases were positively triggered and contained 391 AEs (this is 63% of the AEs (625) found with the original trigger set). The PPV for an AE with this selection of triggers is therefore 50%. This set of triggers found 147 potentially preventable AEs (66% of potentially preventable AEs (222) found with the original complete trigger set).

When we combine triggers that generate the highest number of potentially preventable AEs (trigger 4, 7 and 8), 162 AEs were found. The PPV for a potentially preventable AE with this system is therefore 20% (this is 73% of potentially preventable AEs (222) found with the original complete trigger set).

Logistic regression with all fifteen triggers (model 2)

In Table 6A and 6B (appendix II), the results of individual triggers and (potentially preventable) AE is shown. The OR was highest for trigger 5 (OR = 5.055) and trigger 6 (OR = 3.501).

Although not statistically significant, trigger 1 (OR = 0.884) and trigger 15 (OR = 0.826) suggest a lower risk for finding an AE. For preventable AEs, the OR was the highest for trigger 14 (OR = 2.795), trigger 5 (OR = 1.671) and trigger 6 (OR = 1.588).

Logistic regression with all fifteen triggers and patient characteristics (model 3)

The patient characteristics which were included in the logistic regression combined with the triggers were: urgent admission (yes/no), origin (coming from another hospital yes/no), age, gender, the length of stay (days) and admission specialism (surgical/medical).

In Table 7A and 7B (appendix II), the results of the combination of all triggers and these additional characteristics are shown.

To find out which combination could identify the highest number of patients having an AE, we chose several cut-off points, which are shown in Table 4. A cut-off point of 0,3 would mean that 532 of the 589 cases with AE would be found and that fewer records were selected for review than in the original review (PPV is therefore 90%). With a cut-off point of 0.3, 194 of the 217 (PPV 89%) possibly preventable AEs will be detected. Higher cut-off points detected less AEs.

Table 4

Cut off points.

Cut-off point	Number of medical records selected	AE found	Cases with AE missed	Potentially preventable AE found	Cases with potentially preventable AE missed
> 0.1	1262	589	0	217	0
> 0.2	1259	589	0	217	0
> 0.3	1061	532	57	194	23
> 0.4	537	360	229	134	83
> 0.5	437	310	279	118	99
> 0.6	386	281	308	110	107

Discussion

This study showed that the trigger system had an average PPV for AEs of 47% and for potentially preventable AEs of 17%. The more triggers found in a case, the higher the probability of finding an AE. Adjustments to the trigger system slightly increased the PPV for AEs and potentially preventable AEs but fail to identify around 10% of cases (cut-off point 0.3) compared to the complete original trigger system.

The triggers of the “Harvard Medical Practice Study” and the “Global Trigger Tool” have an overall PPV of 40.3% and 30.4% respectively (Unbeck et al., 2013). This matches well with the results in our sample. Looking at the individual triggers we found that unplanned removal, damage or repair of an organ (PPV = 76.3% for total AE) and unplanned return to the operating room (PPV = 82.1% for total AE) had the highest predictive value for an AE. A study by Naessens et al. (2010) also showed that the trigger with the highest yield was ‘return to the operating room’ where 80.6% of these patients suffered from an AE (Naessens et al., 2010). Hwang et al. (2014) analyzed the global trigger tool. They found that only six triggers had positive predictive values of > 50% (Hwang et al., 2014). Two of these PPV's could be reproduced by the triggers in our data. The definitions of the other triggers were not comparable. 12 of our triggers had a PPV higher than 50%. Possibly this is caused by a difference in patient selection (we only investigated deceased patients) or in the expertise of the committee that investigated the records and adjudged AEs.

Clinical and patient characteristics associated with increased occurrence of AEs were admission through the emergency room, transfer from another hospital, a higher number of triggers and admission for a surgical specialism. Although the first three seem logical the latter suggests a higher risk in surgical wards that has no easy explanation. This has been noticed in other studies (Zegers et al., 2009). Freund et al. (2013) also found no significant difference in terms of age and sex. However, the data were not corrected for comorbidities or the condition of the patients. Furthermore, the complications are usually closely related to a surgical intervention making the allocation of an AE to the intervention easy, whereas in medical specialties the complications of, for instance, pharmacological interventions, are less predictable or occur later making the detection of a link more difficult. Moreover, surgeons are ahead with registration of complications compared to medical departments and this might simplify the finding an AE in these patients (Marang-van de Mheen and Kievit, 2003, Anon., 2010).

One would expect that cases with multiple triggers or a longer duration of stay in the hospital experience more AEs. Our study indeed shows a higher risk for AEs as the number of triggers per case increases. Although there seems to be a trend in more risk for an AE with longer hospitalization, this was not statistically significant.

A report published by NIVEL (The Netherlands) in 2013 showed that on average 12% of all patients who die in the hospital experience an AE. In academic hospitals, this was 15.1% (95%CI 11.8–19.0). Four percent of the patients who died during their stay experienced a preventable AE according to this report (2.3% in academic hospitals) (Langelaan et al., 2013b).

Our study found that 27% of all patients who died in the hospital, experienced an AE of which 37% (10% of all patients) was considered potentially preventable. This might partially be explained by the fact that we, in contrast to NIVEL, also incorporated gynecology, psychiatry and neonatology cases. Although the subjects in the NIVEL study were older (75 vs. 69.6 years) they had a shorter length of stay (10.4 vs. 12.9 days) indicating this sample might not be completely comparable to ours. Furthermore, internal reviewers are known to find more AEs than external reviewers and NIVEL only recruited external reviewers which had a medical, surgical, or neurological background (Langelaan, 2013, Landrigan et al., 2010). A certain level of bias in judging single cases cannot be excluded and might also increase the number of AEs found. Internationally preventability of AEs ranges from 43 to 70% (Wang et al., 2016, Aranaz-Andres et al., 2008, von Laue et al., 2003). This indicates that there are considerable differences in the way judgments about preventability, or even the presence of an AE, are made. Although some studies use different grades of preventability, there is no international consensus on how to specifically apply these grades. Therefore, comparisons can hardly be made in view of different methods used.

Combining triggers and clinical characteristics seems promising in reducing the review of cases without an AE. However, some (potentially preventable) AEs will be missed. Quality and safety departments in hospitals have to decide on the optimal cut-off point. This could then result in less medical records necessary to be screened by the specialist, saving a fair amount of time and costs.

Possibly entering additional variables into the system might increase the gain of this system. For example, clinical scores of vital functions (like modified early warning score) or laboratory results (like albumin, creatinine, hemoglobin level etc.). These might be combined with the existing triggers to improve the PPV.

The strength of our study is the specific and reliable recording of triggers and AEs using software specifically developed for this purpose. Also, the number of cases from a single hospital over a period of 3 years is large enough to generate reliable results with small confidence intervals. Furthermore, there was a stable trigger team during the period selected for this study and the presence of an AE was decided on after discussion within the committee generating broad support for the final decision. Preventability was judged by the consensus of several doctors with a variable background. In the face of a lacking international consensus on this concept of preventability, this seems an optimal and acceptable method.

Clearly, there are also points for improvement. We compared formerly reported PPV's of these two commonly used trigger sets in different patient samples with the result of a slightly adapted HPMS trigger set in our sample of deceased patients. We have a single measurement of our PPV over a period of three years whereas the PPV of the other trigger systems is based on an average of several studies presented in the literature. Furthermore, we have no information on the negative predictive value of our trigger system. It is possible that some of the cases that were not triggered contain AEs or potentially preventable AEs.

We realize that we looked into a subset of patients which makes our results not generalizable to the average hospital patient. He or she might be younger with fewer comorbidities and other diseases in different departments. Therefore, important potentially preventable AEs in these patients could have been missed. Our data also gave us no information about the reproducibility of the trigger system. For the HMPS method, Kappa values are reported between 0.53(Soop et al., 2009) and 0.76 (Wilson et al., 2012) (moderate to good agreement), for the IHI method between 0.20 (Schildmeijer et al., 2012) and 0.78 (O'Leary et al., 2013) (slight to good agreement). Lastly, it is likely that in a confirmation study that the PPVs will be lower than we have found in this derivation study.

In our opinion, it is disappointing that trigger systems select over 50% of cases without an AE. Even after combining several triggers the PPV does not significantly improve. This method remains, therefore, labor-intensive until we can define triggers or trigger sets with a higher PPV. Further research to optimize these systems concerning the combination of triggers with patient characteristics or possible even laboratory results seems warranted.

Due to the expected higher number of AEs in deceased patients, we expected this tool to perform better in this subsample of patients. However, we think that the PPV of the HPMS in this sample is disappointing but compares well to results from general inpatient samples using the HPMS or IHI trigger system.

Funding sources

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflict of interest

The authors declare there is no conflict of interest.