Trauma models to identify major trauma and mortality in the prehospital setting.

BACKGROUND: Patients with major trauma might benefit from treatment in a trauma centre, but early identification of major trauma (Injury Severity Score (ISS) over 15) remains difficult. The aim of this study was to undertake an external validation of existing prognostic models for injured patients to assess their ability to predict mortality and major trauma in the prehospital setting.
METHODS: Prognostic models were identified through a systematic literature search up to October 2017. Injured patients transported by Emergency Medical Services to an English hospital from the Trauma Audit and Research Network between 2013 and 2016 were included. Outcome measures were major trauma (ISS over 15) and in-hospital mortality. The performance of the models was assessed in terms of discrimination (concordance index, C-statistic) and net benefit to assess the clinical usefulness.
RESULTS: A total of 154 476 patients were included to validate six previously proposed prediction models. Discriminative ability ranged from a C-statistic value of 0·602 (95 per cent c.i. 0·596 to 0·608) for the Mechanism, Glasgow Coma Scale, Age and Arterial Pressure model to 0·793 (0·789 to 0·797) for the modified Rapid Emergency Medicine Score (mREMS) in predicting in-hospital mortality (11 882 patients). Major trauma was identified in 52 818 patients, with discrimination from a C-statistic value of 0·589 (0·586 to 0·592) for mREMS to 0·735 (0·733 to 0·737) for the Kampala Trauma Score in predicting major trauma. None of the prediction models met acceptable undertriage and overtriage rates.
CONCLUSION: Currently available prehospital trauma models perform reasonably in predicting in-hospital mortality, but are inadequate in identifying patients with major trauma. Future research should focus on which patients would benefit from treatment in a major trauma centre.

Introduction

Trauma remains one of the major causes of premature death and disability worldwide1. Studies suggest that patients with major trauma, defined as life‐threatening or potentially life‐changing injury, benefit from treatment in a level 1 trauma centre2. Identification of patients with major trauma in the prehospital setting is therefore important, as certain patients could potentially benefit from bypassing the nearest hospital and a longer primary journey to a more distant level 1 trauma centre.

However, early identification of major trauma remains difficult. Undertriage, whereby patients with major trauma are treated at a non‐trauma centre, could cause avoidable mortality and morbidity3. On the other hand, overtriage, whereby patients without major trauma are transported to a trauma centre, could overwhelm level 1 trauma centres, adversely affecting patient outcomes, and transporting patients longer distances for no benefit. The American College of Surgeons – Committee on Trauma4 stipulates that an undertriage rate of 5 per cent and an overtriage rate of 25–35 per cent is acceptable.

Major trauma is usually defined in anatomical terms by an Injury Severity Score (ISS) above 15. The ISS is based on postimaging anatomical and clinical findings, and cannot be calculated in the prehospital setting5. Currently, there are no consensual criteria for prehospital identification of major trauma. A previous systematic review6 found that prognostic models developed to predict mortality could be useful for identifying major trauma. However, a direct comparison of the performance of existing models in the same validation cohort is needed to assess which models are useful in the prehospital setting to predict major trauma (ISS over 15) and mortality7.

The aim of this study was to undertake an external validation of existing prognostic models for injured patients, and to assess their discriminative ability to predict mortality and major trauma in the prehospital setting.

Methods

Identification of prognostic models

A systematic search was undertaken to identify existing prognostic models aimed at improving early trauma care, based on a systematic review published in 20116. Five prognostic models that were included in the original study were included in this study, and the search was updated using the original published search strategy in the MEDLINE database (Appendix , supporting information). Each study was assessed against the original inclusion criteria of the systematic review, which comprised: a tool for clinicians that includes two or more predictors obtained from the history and physical examination of a suspected trauma victim; predictors collected in the field or in the emergency department up to 12 h after injury; prognostic models developed for adult patients defined, for the purpose of this review, as older than 15 years of age or if the patients were described by the authors as adults; and studies published within the last 20 years. The original exclusion criteria of the systematic review were: models that required complex information such as paraclinical diagnostic tests; models pertaining to burns, drowning, strangulation, isolated proximal femur fractures, isolated traumatic brain injury, pregnancy or medical conditions; and studies not written in English. Prognostic models identified from the systematic review and the updated search were validated externally when the variables used could be retrieved in the validation cohort.

Validation cohort

To validate the selected prognostic models externally, retrospectively collected data from the Trauma Audit and Research Network (TARN) were used. The TARN database is the national trauma registry of England, Wales, Northern Ireland and the Republic of Ireland, with some members in continental Europe. TARN includes patients with significant injury presenting to hospital who were subsequently either admitted for at least 72 h or to a critical care area, or required interhospital transfer for acute care, or who died in hospital. Isolated fractures of the hip/pubic ramus in patients aged 65 years or more and isolated closed limb injury (except for femoral shaft) were excluded. Injured adult patients (aged over 15 years) who were transported by the Emergency Medical Services (EMS) or Helicopter Emergency Medical Services (HEMS) and admitted to hospitals in England between 2013 and 2016 were included; secondary referrals of patients for whom first hospital details were not available were excluded. TARN has UK Health Research Authority Approval (PIAG Section 251) for research on anonymized patient data (TARN extraction date: 23 May 2017).

Statistical analysis

Baseline characteristics of patients in the validation cohort are presented as median (i.q.r.) for continuous variables and as percentages for categorical variables. To account for missing data, multiple imputation was performed8, 9. The pattern of missingness was assumed to be missing at random. This means that the missing values correlate with other patient characteristics, which makes it possible to use multiple imputation. The imputation model contained all relevant predictor and outcome variables: the model predictors (systolic BP, respiratory rate, prehospital intubation, pulse, oxygen saturation, consciousness or Glasgow Coma Scale (GCS) score, mechanism of injury (blunt versus other), number of serious injuries and age), combined with extra predictor variables that could add information to predict missing values (in‐hospital intubation, ISS, New Injury Severity Score, duration of hospital stay, duration of critical care stay, trauma team activation, head injury, Charlson Co‐morbidity Index, GCS components at emergency department, sex and in‐hospital mortality).

The models were first validated to assess their ability to predict in‐hospital mortality, and then to assess their ability to predict injury severity (ISS over 15). Patients were scored retrospectively using the published score chart for each model. The published score charts were used instead of the regression formulas, as the performance of the models in the prehospital setting was assessed and ambulance personnel would use the score charts. When a variable was not available in the TARN cohort, a proxy was used if possible. The intercept was refitted on the TARN registry, which resulted in a recalibrated model that was slightly adjusted to the TARN population. The performance of the models was assessed in terms of discrimination, which is the ability of the model to distinguish between low‐ and high‐risk groups of patients. Discriminative ability was expressed as the C‐statistic, which is equivalent to the area under the receiving operating characteristic (ROC) curve (AUC). A C‐statistic of 1 implies perfect discrimination, whereas a value of 0·5 implies that the performance of the model is equal to chance10. Additionally, overtriage and undertriage rates were calculated. Undertriage is the proportion of patients with major trauma who were not classified by the model as having major trauma, and was calculated as 1 – sensitivity, where sensitivity is defined as the proportion of patients actually with major trauma who were identified as having major trauma by the model. Overtriage is the proportion of patients with minor trauma who were classified by the prediction model as having major trauma, and was calculated as 1 – specificity, where specificity was defined as the proportion of patients actually with major trauma who were identified as having minor trauma by the model.

Finally, the net benefit was calculated for all models at different thresholds. The net benefit represents the potential gain of using the prediction models under study for triage of injured patients compared with sending all patients to a major trauma centre. Net benefit is defined as the proportion of true‐positives – proportion of false‐positives × weight. For example, a threshold of 0·2 means that a trauma centre would accept four patients wrongly classified as having major trauma (false‐positives) to identify one correctly classified as having major trauma (true‐positive, defined as ISS over 15). The weight is defined as the odds of the threshold (maximum number of patients wrongly classified as having major trauma (false‐positives) to correctly classify 1 patient with major trauma (true‐positive)). For the threshold of 0·2, the weight is 1 : 411.

Data were analysed using R software version 3.2.2 or higher (R Foundation for Statistical Computing, Vienna, Austria).

Results

Selection of risk prediction models

The study by Rehn and colleagues6 identified five prognostic models: Circulation, Respiration, Abdomen, Motor, Speech (CRAMS)12, Prehospital Index (PHI)13, Triage Revised Trauma Score (T‐RTS)14, Physiologic Severity Score (PSS)15 and Mechanism, Glasgow Coma Scale, Age and Arterial Pressure (MGAP)16; all were developed for predicting mortality in injured patients (Table , supporting information).

The updated electronic search identified 6048 new articles until October 2017 (Fig. , supporting information). After screening of titles and abstracts, 106 potentially eligible articles were identified for full‐text review. After full‐text assessment, 104 articles were excluded because they did not develop a prediction model or the prediction model included predictors that could not be measured in the prehospital setting. Finally, two extra prediction models were identified: the modified Rapid Emergency Medicine Score (mREMS)17 and the Kampala Trauma Score (KTS)18, also developed for predicting mortality in injured patients (Table , supporting information).

The CRAMS model included a variable that needs prehospital assessment of abdominal tenderness and the presence of rigid or flail chest. There was no proxy available for this predictor in the validation data, so this model was excluded. All other models could be validated in the TARN data set. In total, six prehospital models were included for validation (Table , supporting information).

For KTS, the variable ‘number of serious injuries’ was not directly available in the present validation cohort. Therefore, the number of serious injuries was based on the number of reported Abbreviated Injury Scale (AIS) codes greater than 2.

Some 246 301 patients were included in the TARN registry between 2013 and 2016. A total of 154 476 adult injured patients who were transported by the EMS or HEMS to a hospital were included in the validation cohort (Fig. 1). Overall, 101 658 patients (65·8 per cent) had an ISS of 15 or below. The median age of these patients was 68 (i.q.r. 50–84) years, 48 511 patients (47·7 per cent) were male, median ISS was 9 (5–9), 0·9 per cent of these patients were comatose when presenting at the emergency department, and the in‐hospital mortality rate was 3·7 per cent (Table 1). Among patients who had major trauma (ISS over 15), the median age was 61 (39–81) years, 65·3 per cent were male, the median ISS was 25 (17–27), 16·3 per cent were comatose when presenting at the emergency department, and the in‐hospital mortality rate was 15·5 per cent. The proportions of missing values were low: 0·1 per cent for in‐hospital variables, 6·6 per cent for prehospital variables and below 0·01 per cent for outcome variables in the group with an ISS of 15 or less. Respective proportions in the group with an ISS above 15 were 0·1, 9·1 and less than 0·01 per cent.

Figure 1

STROBE flow diagram of included and excluded patients TARN, Trauma, Audit and Research Network; EMS, Emergency Medical Services; HEMS, Helicopter Emergency Medical Services.

Table 1

Baseline characteristics according to severity of trauma and survival status

	ISS ≤15 (n = 101 658)	ISS > 15 (n = 52 818)	Alive at discharge (n = 142 594)	Died in hospital (n = 11 882)	Total (n = 154 476)
Patient characteristics
Age (years)*	68 (50–84)	61 (39–81)	64 (46–82)	81 (62–88)	66 (47–83)
Male	48 511 (47·7)	34 468 (65·3)	76 272 (53·5)	6707 (56·4)	82 979 (53·7)
ISS*	9 (5–9)	25 (17–27)	9 (9–17)	25 (10–26)	9 (9–17)
GCS score in emergency department*	15 (15–15)	15 (12–15)	15 (15–15)	13 (4–15)	15 (15–15)
GCS score < 9†	944 (0·9)	8516 (16·3)	5087 (3·6)	4373 (37·5)	9460 (6·2)
Penetrating injury	2867 (2·8)	1392 (2·6)	3970 (2·8)	289 (2·4)	4259 (2·8)
Intubated in emergency department	2030 (2·0)	12 357 (23·4)	9819 (6·9)	4568 (38·4)	14 387 (9·3)
Shock (systolic BP < 90 mmHg) in emergency department	17 246 (17·0)	12 445 (23·6)	26 189 (18·4)	3502 (29·5)	29 691 (19·2)
Prehospital measurements
Systolic BP (mmHg)*	139 (122–158)	136 (118–157)	138 (121–157)	139 (113–163)	138 (121–158)
Respiratory rate (per min)*	18 (16–20)	18 (16–22)	18 (16–20)	18 (16–22)	18 (16–20)
Pulse (per min)*	83 (72–96)	84 (71–100)	84 (72–97)	82 (68–100)	84 (72–97)
Oxygen saturation (%)*	97 (95–98)	97 (94–98)	97 (95–99)	95 (91–98)	97 (95–98)
No. of serious injuries
0	961 (0·9)	0 (0)	914 (0·6)	47 (0·4)	961 (0·6)
1	70 989 (69·8)	12 076 (22·9)	78 587 (55·1)	4478 (37·7)	83 065 (53·8)
≥ 2	29 708 (29·2)	40 742 (77·1)	63 093 (44·2)	7357 (61·9)	70 450 (45·6)
Outcomes
Duration of hospital stay (days)*	10 (5–18)	9 (4–20)	10 (5–19)	4 (1–10)	9 (5–19)
Duration of critical care stay (days)*	0 (0–0)	0 (0–1)	0 (0–0)	0 (0–1)	0 (0–0)
Primary referral to MTC	33 349 (32·8)	27 220 (51·5)	55 298 (38·8)	5271 (44·4)	60 569 (39·2)
In‐hospital mortality	3711 (3·7)	8171 (15·5)	–	–	11 882 (7·7)

Values in parentheses are percentages unless indicated otherwise;

values are median (i.q.r.).

Missing values. ISS, Injury Severity Score; GCS, Glasgow Coma Scale; MTC, major trauma centre.

External validation

First, the discriminative ability of the prediction models to predict in‐hospital mortality was assessed. The discriminative performance (C‐statistic) for the models varied between 0·602 (95 per cent c.i. 0·596 to 0·608) for the MGAP model and 0·793 (0·789 to 0·797) for the mREMS model (Table 2). For comparison, the C‐statistic of the ISS for predicting in‐hospital mortality was 0·728 (0·723 to 0·733).

Table 2

Discriminative ability (C‐statistic) for in‐hospital mortality (11 882 patients) and major trauma (52 818) among 154 476 patients registered in TARN, 2013–2016, with overtriage and undertriage rates for predicting major trauma using the optimal cut‐off for models in the validation data set

Model	C‐statistic for in‐hospital mortality	C‐statistic for ISS > 15	Cut‐off indicating major trauma of total model score	1 – sensitivity (undertriage)	1 – specificity (overtriage)
PHI	0·734 (0·729, 0·739)	0·708 (0·706, 0·711)	≥ 1 of 20	38·9	23·7
T‐RTS	0·706 (0·702, 0·711)	0·630 (0·628, 0·632)	≤ 11 of 12	66·8	8·1
PSS	0·741 (0·736, 0·746)	0·710 (0·708, 0·713)	≤ 11 of 12	40·5	21·3
MGAP	0·602 (0·596, 0·608)	0·659 (0·657, 0·662)	≤ 28 of 29	31·0	51·2
mREMS	0·793 (0·789, 0·797)	0·589 (0·586, 0·592)	> 3 of 26	23·1	72·4
KTS	0·769 (0·764, 0·773)	0·735 (0·733, 0·737)	≤ 15 of 16	3·6	82·8
ISS	0·728 (0·723, 0·733)	–	> 15	–	–

Values in parentheses are 95 per cent confidence intervals. An Injury Severity Score (ISS) above 15 indicates major trauma. According to American College of Surgeons – Committee on Trauma guidelines, an undertriage rate of 5 per cent and an overtriage rate of 25–35 per cent is acceptable. TARN, Trauma and Audit Research Network; PHI, Prehospital Index; T‐RTS, Triage Revised Trauma Score; PSS, Physiologic Severity Score; MGAP, Mechanism, Glasgow Coma Scale, Age and Arterial Pressure; mREMS, modified Rapid Emergency Medicine Score; KTS, Kampala Trauma Score.

The ability of the prehospital prediction models to predict major trauma (ISS over 15) was then assessed. The C‐statistic the models varied between 0·589 (0·586 to 0·592) for the mREMS model and 0·735 (0·733 to 0·737) for the KTS model (Table 2).

Overtriage and undertriage rates were calculated. The optimal cut‐off values were tested and the undertriage rate (1 – sensitivity) varied from 3·6 per cent for KTS to 66·8 per cent for T‐RTS. The corresponding overtriage rate (1 – specificity) varied from 82·8 per cent for KTS to 8·1 per cent for T‐RTS (Table 2).

Clinical usefulness

Compared with treating all patients as having major trauma, the prediction models showed benefit with a threshold above 0·2, corresponding to a weight of at least 1: 4 (Fig. 2). Of all models, KTS showed the greatest benefit. For thresholds below 0·2, it was most beneficial to treat all patients as having major trauma (so transporting all patients to a major trauma centre).

Figure 2

Net benefit curves for different prehospital strategies for patients with suspected major trauma The graph shows the net benefit of using specific prehospital models to detect major trauma, based their optimal cut‐off in the validation data. Plots for the strategy of treating none of the patients as having major trauma (transporting no patients to a major trauma centre) or all patients as having major trauma are also shown. The

Discussion

This study comprised an external validation of models developed to predict mortality and major trauma in injured patients in the prehospital setting. Most models were found to perform reasonably in predicting in‐hospital mortality but less well in identifying major trauma. None of the prediction models was able to obtain overtriage and undertriage rates conforming to the guidelines of the American College of Surgeons4. The present results are consistent with those for paediatric triage tools for injured children19, which were also unable to meet the recommended criteria.

Net benefit analysis showed that for thresholds between 0·2 and 0·5, meaning that trauma centres would be willing to accept a maximum of four patients wrongly classified as having major trauma for one correctly classified as having major trauma, KTS was the best model of those analysed and compared with treating all patients within the TARN registry as having major trauma. For thresholds below 0·2 (accepting more than 4 patients wrongly classified as having major trauma for 1 correctly classified as having major trauma), treating all patients as having major trauma seemed most beneficial.

All models, except MGAP, had a more than reasonable score (C‐statistic at least 0·70) in predicting in‐hospital mortality. The three best prediction models for in‐hospital mortality were mREMS, KTS and PSS. These models included the following predictors that seem to be most important for in‐hospital mortality: systolic BP, respiratory rate and level of consciousness or GCS score. The models also performed better in predicting in‐hospital mortality than the ISS. The three best prediction models for predicting major trauma were KTS, PSS and PHI. These models have the predictors systolic BP, respiratory rate and consciousness in common. KTS contains the number of serious injuries, an apparently important predictor that is lacking in all other models. The preferred model to implement would depend on the intended use of the models and the available predictors. The mREMS model showed the best discriminative ability in predicting in‐hospital mortality, whereas the KTS model was the best in distinguishing patients with major trauma from those without.

Most prediction models performed less well in this external validation than in their derivation population and other external validations for the outcome mortality. MGAP had an AUC of 0·90 in its derivation cohort16, compared with 0·602 in the present study; mREMS had an AUC value of 0·9717 compared with 0·793 here; and PSS had an AUC of 0·9315 compared with 0·741 in this study. These AUC values above 0·90 in the development setting imply overfitting of the model and explain the poorer performance in the present study population. External validations of T‐RTS varied between 0·83 and 0·8414, 16, 20, 21 compared with 0·706 in the present study. An external validation of PHI showed an AUC of 0·6613, 22 compared with 0·734 in this study. An external validation of KTS showed an AUC of 0·77 previously18, 23 compared with 0·769 here. The performance of the prehospital trauma scores seems very population‐ and setting‐dependent. Therefore, it might be important to validate models externally before using them in specific settings.

All models were identified through a systematic literature search and it was possible to include most existing prediction models in the validation study. The present validation cohort was very large with wide heterogeneity in patient characteristics, mechanism of injury and outcomes. By evaluating the different models in the same validation cohort, this study provides more information than separate validation studies. The PSS model has never been validated externally before6. The net benefit analysis has shown the importance of looking at different weights as they influence the potential benefit of the models.

One limitation of this study is the use of retrospective data restricted to injured patients who were admitted to hospital. Patients with minor trauma who were discharged immediately from the emergency department were not included in the validation cohort. This could have led to underestimation of the specificity and overestimated overtriage rates of all models, because such patients would have been true‐negatives. This makes the net benefit analyses applicable only to injured patients with higher chances of having major trauma because those who had experienced minor trauma were not included in TARN. A (clustered) RCT of different triage tools might be considered to examine the effect of such tools, for example to assess the effect of KTS in a high‐income country. However, a recent trial24, 25 that investigated the impact of trying to transport patients with head injury straight to neurosurgery showed that it remains difficult to study the effect of certain interventions.

It was not possible to validate the CRAMS model, which had high sensitivity (92 per cent) and specificity (98 per cent) in the original article12, owing to lack of variables and proxy variables. Nor was it possible to assess calibration, because recalibration of the models was necessary owing to missing information about the intercept. Furthermore, there was a lack of cut‐offs in predicted probabilities proposed in the original studies. Some prediction models mentioned only two categories with the corresponding predicted probabilities in their main article, which meant that the intercept and slope retrieved from the validation cohort were not very informative. In addition, there were some missing values in the prehospital variables, with larger percentages of missingness for the severely injured patients. This could have introduced bias, although missing values were handled using multiple imputation with a rich imputation model that included essentially all available information26.

Another limitation of this study was that all included prediction models were developed initially to predict mortality. However, as these prediction models are used clinically for triage in the prehospital setting, their discriminative abilities for predicting both in‐hospital mortality and major trauma were assessed. The model that performed most solidly in predicting both in‐hospital mortality and major trauma was the KTS, which was developed in Uganda23. As this triage model had never been tested in a high‐income environment before, it requires further validation in high‐income countries. As a result of the strict inclusion criteria of TARN, the large number of serious injuries created overtriage rates in KTS. Future studies considering KTS should include the prehospital opinion about the number of serious injuries.

This study has shown that most prehospital trauma models are inadequate in identifying patients with major trauma when this is defined by an ISS of more 15. None of these prediction models met the required undertriage and overtriage rates as defined by the American College of Surgeons (5 per cent undertriage and 25–35 per cent overtriage). These overtriage and undertriage rates imply that avoiding undertriage is considered five to seven times more important, corresponding to a threshold of 0·14. For this threshold, the preferred strategy would be to transport all injured patients to major trauma centres. However, this could cause overcrowding of emergency departments in such centres. The optimal threshold, and thus the cut‐offs used for triage decisions, might differ according to the individual trauma system; different hospitals might be able to tolerate different amounts of overtriage.

However, a universal definition of major trauma is still lacking. The injury severity measure used throughout trauma registries and research is the ISS, with a score of more than 15 defining severely injured or major trauma. Questions about the accuracy of the ISS have been raised. First, an equal AIS in different body regions is assumed to indicate similar injury severity27, 28. Second, the ISS does not account for multiple injuries in the same body region28, 29. This implies that patients with equal ISS scores could benefit differently from treatment in designated trauma centres. Future research should focus on which groups of injured patients could really benefit from designated trauma centres in terms of mortality and quality of life. This is an essential step before new triage tools can be developed.

Future studies should focus on the development of prehospital tools to identify patients who would benefit from treatment in a major trauma centre instead of focusing on the development of new prehospital tools to identify patients with an ISS exceeding 15.

Appendix S1. Search strategy

Fig. S1. Flow chart search strategy

Table S1. Prehospital prediction models

Click here for additional data file.