Discrimination and calibration of a prediction model for mortality is decreased in secondary transferred patients: a validation in the TraumaRegister DGU.

INTRODUCTION: The Revised Injury Severity Classification II (RISC II) score represents a data-derived score that aims to predict mortality in severely injured patients. The aim of this study was to assess the discrimination and calibration of RISC II in secondary transferred polytrauma patients.
METHODS: This study was performed on the multicentre database of the TraumaRegister DGU. Inclusion criteria included Injury Severity Score (ISS)≥9 points and complete demographic data. Exclusion criteria included patients with 'do not resuscitate' orders or late transfers (>24 hours after initial trauma). Patients were stratified based on way of admission into patients transferred to a European trauma centre after initial treatment in another hospital (group Tr) and primary admitted patients who were not transferred out (group P). The RISC II score was calculated within each group at admission after secondary transfer (group Tr) and at primary admission (group P) and compared with the observed mortality rate. The calibration and discrimination of prediction were analysed.
RESULTS: Group P included 116 112 (91%) patients and group Tr included 11 604 (9%) patients. The study population was predominantly male (n=86 280, 70.1%), had a mean age of 53.2 years and a mean ISS of 20.7 points. Patients in group Tr were marginally older (54 years vs 52 years) and a had slightly higher ISS (21.5 points vs 20.1 points). Median time from accident site to hospital admission was 60 min in group P and 241 min (4 hours) in group Tr. Observed and predicted mortality based on RISC II were nearly identical in group P (10.9% and 11.0%, respectively) but predicted mortality was worse (13.4%) than observed mortality (11.1%) in group Tr.
CONCLUSION: The way of admission alters the calibration of prediction models for mortality in polytrauma patients. Mortality prediction in secondary transferred polytrauma patients should be calculated separately from primary admitted polytrauma patients. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Retrospective cohort study without the possibility of verification of each data point.

Patients who died during transportation are not included in this study due to lack of data.

Registry study with big data including more than 120 000 polytrauma patients.

Prediction of mortality match perfectly in primary admitted patients but disagree by 2.3% in transferred patients.

Discrimination depends on the way of admission.

Introduction

Severe trauma counts as one of the leading causes for morbidity and mortality in the working population.1 Numerous studies have investigated pathophysiological parameters that might recommend treatment strategies2 3 or predict mortality.4 Adding measures from different pathophysiologic measures to the prediction model improves the performance of predicting mortality.5 6 To improve outcome analysis, big data registries have been developed on both national and international scales. The German Trauma Society founded the TraumaRegister DGU (TR-DGU) in 1993 and has collected data from polytrauma patients ever since.

Initially, the TR-DGU used the Trauma Injury Severity Score (TRISS)7 for outcome adjustment but then developed its own prognostic instrument, the Revised Injury Severity Classification (RISC).8 The RISC uses patient demographics and injury severity and distribution, as well as physiological measures, at admission to develop a prediction model for mortality. The RISC was updated in 2014 (RISC II),9 which was based on data from 30 866 primary admitted patients from the TR-DGU (years 2010–11) database.

The scoring system has repeatedly been internally and externally validated.10–12 The RISC II score is further used every year when new trauma patients are added to the registry. The results are published in annual reports.13 The score components of RISC II were based on data from primary admitted patients, without prior hospital treatment. However, it is unclear whether the prognosis is also valid for transfer-in patients. It has been shown that mortality is associated with increased prehospital time14; however, secondary transfer of trauma patients appears to not increase the mortality rate.15 Yet, it has been demonstrated that 16.5% of transfers were considered inappropriate, which might mask the effect of secondary transfers.16 These discrepancies might affect the capability of prediction models for mortality in patients with trauma. Therefore, we aimed to assess the discrimination and calibration of RISC II in secondary transferred polytrauma patients.

Methods

Ethical consideration

All data used in this study are derived from an initiative for external quality assessment that is based on routinely available data. In Germany, no additional ethical vote is required for collecting the data. For scientific analyses and publications, the data are further condensed and do not contain data about the date and time of injury or the location of the treating hospital.

Patient and public involvement

No patient involved.

Study population and the TR-DGU

This retrospective cohort study included patients from the TR-DGU and adheres to the TRIPOD statement.17 All patients were treated in participating European trauma centres between January 2015 and December 2019. Furthermore, this study only included patients with an Injury Severity Score (ISS) of at least nine points, representing relevant injury, as defined by the European Union Traffic Statistics (AIS max 3 and higher is defined as a ‘serious injury’).18 We excluded patients who had a written declaration of therapy limitations (such as a do not resuscitate (DNR) order), because this will bias the comparison of observed and predicted outcomes. In Germany, a DNR might include that the patient reject treatment at the intensive care unit (ICU) or even reject mechanical ventilation, which are measures that might be necessary to prevent mortality.

We also excluded patients who were transferred out early (<48 hours), since the final outcome was considered missing in these cases. Patients were grouped according to the way of admission into a primary admission group (group P) and a group with patients who were transferred in after initial treatment in another hospital (group Tr). The time from accident to hospital admission was calculated in minutes if both time points were available or in days if one or both time points were missing. Transferred patients (group Tr) were excluded when the time from accident to admission was less than 60 min (probably not a transferred patient) or patients with late transfers where the time from initial trauma to admission was longer than 24 hours, or on day 2 or later, as this might represent secondary complications after initial surgical treatment.

The TR-DGU of the German Trauma Society (Deutsche Gesellschaft für Unfallchirurgie, DGU) was founded in 1993. The aim of this multicentre database is a pseudonymised and standardised documentation of severely injured patients. Data are collected prospectively in four consecutive time phases from the site of the accident until discharge from the hospital: (1) prehospital phase, (2) emergency room and initial surgery, (3) ICU and (4) discharge. The documentation includes detailed information on demographics, injury pattern, comorbidities, prehospital and in-hospital management, course in the ICU, relevant laboratory findings (including data on transfusions) and outcome of each individual. The inclusion criterion was admission to the hospital via an emergency room with subsequent ICU/Intermediate care unit (IMC) care or reaching the hospital with vital signs and death before admission to the ICU. The infrastructure for documentation, data management and data analysis was provided by the Academy for Trauma Surgery (AUC; Akademie der Unfallchirurgie), a company affiliated with the German Trauma Society. Scientific leadership was provided by the Committee on Emergency Medicine, Intensive Care and Trauma Management (Sektion NIS) of the German Trauma Society. The participating hospitals submitted their data pseudonymised into a central database via a web-based application. Scientific data analysis was approved according to a peer-review procedure laid down in the publication guideline of the TR-DGU. The participating hospitals were primarily located in Germany, but a rising number of hospitals in other countries also contributed data (at the moment Austria, Belgium, Finland, Luxembourg, Slovenia, Switzerland, the Netherlands and the United Arab Emirates). Participation in the TR-DGU is voluntary. For hospitals associated with the TR-DGU, however, the entry of at least a basic dataset was obligatory for reasons of quality assurance.

This study is in line with the publication guidelines of the TR-DGU and registered as TR-DGU project ID 2020-023.

Setting

In Germany, about 650 hospitals provide acute care for severely injured patients. They were certified as level 1 (supraregional, about n=100), level 2 (regional, about n=200) and level 3 (local, about n=350).19 Severely injured patients are examined by a physician on scene. Based on the examination, the receiving hospital is chosen. About 90% of patients do not require further transfer to a higher level trauma centre, while 10% of patients are transferred shortly after admission to another hospital, usually to a level 1 trauma centre. Most of the data are provided from German hospitals (83%), 8% come from Austria and Switzerland and the remaining 9% from the Netherlands, Belgium, Luxemburg, Finland and Slovenia. These trauma systems are comparable and provide similar data to the same registry. We, therefore, believe there is reasonable comparability of trauma systems in these countries.

Prediction model and variables

The development of RISC II has previously been described in detail.8 9 An overview of RISC II variables includes:

Patient: age, sex and pre-existing diseases.

Injury: worst and second worst injury, head injury and mechanism.

Physiology at admittance: coagulation, acidosis, GCS, blood pressure, haemoglobin, consciousness, pupils and cardiac arrest.

The RISC II prediction model has previously been validated and has shown that the quality depends on the design of the registry.10 20 The RISC II was calculated based on admission values of the primary hospital (group P) or on admission values of the hospital where patients were transferred (group Tr).

Prehospital collected data are not included into the calculation of the RISC. In cases where measurements are clearly the result of a medical intervention (eg, GCS of 3 after general narcosis), these measurements were omitted from the RISC calculation.

Outcome

To assess the effect of secondary transfer on the prediction of mortality in polytrauma, the main outcomes were the discrimination and calibration of the RISC II in secondary transferred patient: the comparison of predicted vs observed mortality. We, therefore, defined a threshold of 1% as an acceptable difference between observed and predicted mortality. Further outcomes were the discrimination and calibration of the RISC II prediction model for secondary transferred polytrauma patients.

Sample size consideration and missing data

To provide a sufficiently exact estimation of observed and predicted mortality in transferred patients, the range of a 95% CI around the observed mortality rate in that group should be less than 1%. This would require about 10 000 cases. Therefore, the range of time was set to be 5 years. This is in accordance with recent recommendations.21 22

Age and injuries were the only variables where complete availability is required for calculating the RISC II (compulsory variables in data collection). All other variables may be missing, which is included in the RISC II model. We did not impute missing values. However, the average number of missing data per patient is only 1 out of 13 values required.

Statistical analysis

Continuous variables are presented as means and SDs, while categorical variables are presented as numbers and percentages. In case of considerable skewness of data, median and IQR will replace mean and SD. A 95% CI is provided for selected findings. For all included patients, the RISC II was calculated. The predicted mortality was calculated as the mean value of individual prognoses and then compared with the actual observed hospital mortality. Quality of the prediction model was analysed with comparison of observed and predicted mortality and included sensitivity and specificity analyses of RISC II prognosis. Discrimination of the model was performed with the receiver operating characteristic (ROC) curve and area under the ROC curve (AUC). Discrimination and calibration were further visualised. We plotted expected against observed mortality and grouped the patients in 5% steps. The overall quality of prediction was evaluated with the AUC, with 95% CI. Statistical analysis was performed using SPSS, V.25 (IBM).

Results

Study population

Most eligible patients (n=116 112, 90.9%) were admitted primarily from the scene to the treating hospital, while 11 604 patients (9.1%) were initially stabilised in another hospital before being transferred to the treating hospital. A detailed flow chart is provided in figure 1. The average age of patients in group P was 52.3 years old, while the average of patients in Group Tr was 54.8 years old. Approximately 70% of cases were males in both groups (table 1). The mean ISS was 20.0 points in group P and 21.6 points in group Tr (table 1). More patients in group P suffered road traffic accidents (RTAs) with a car (n=23 466, 20.4%) when compared with those in Group Tr (n=1406, 12.4%). Similarly, in group P, more patients suffered from an RTA with a motorbike (n=15 233, 13.3%) when compared with those in group Tr (n=807, 7.7%). Yet, more patients in group Tr suffered from low falls (<3 m) (n=4647, 41.0%) when compared with those in group P (n=27 672, 24.1%).

Figure 1

Flow chart for the selection of the study population. DGU, Deutsche Gesellschaft für Unfallchirurgie; DNR, do not resuscitate; ISS, Injury Severity Score.

Table 1

Demographic data of included patients

	Group P	Group Tr
n	116 112	11 604
Male sex n (%)	81 383 (70.1)	8141 (70.2)
Age years, mean (SD)	52.3 (22.2)	54.8 (23.4)
ISS points, mean (SD)	20.0 (11.2)	21.6 (10.2)
Time to final admission, minutes, mean (SD)	65.1 (32.3)	334.7 (275.0)

ISS, Injury Severity Score; n, number.

Injury distribution

More patients in Group P suffered from severe chest injuries (AIS≥3, n=52 432, 45.2%) when compared with those in Group Tr (n=3650, 31.5%). Group Tr, on the other hand, included a higher number of patients with severe head injuries (n=7206, 62.1%) when compared with those in Group P (n=44 488, 38.3%) (table 2).

Table 2

Injury distribution

	Group P	Group Tr
n	116 112	11 604
Blunt injury n (%)	107 061 (96.1)	10 768 (97.6)
AIS head ≥3 points, n (%)	44 488 (38.3)	7206 (62.1)
AIS chest ≥3 points, n (%)	52 432 (45.2)	3650 (31.5)
AIS abdomen≥3 points, n (%)	12 992 (11.2)	1311 (11.3)
AIS extremities≥3 points, n (%)	32'193 (27.7)	2401 (20.7)

AIS, Abbreviated Injury Scale; n, number.

Transfer and level of trauma centre

Most of the patients from group Tr were transferred to a level 1 trauma centre (88.7%). Only 10.3% of patients were admitted to a level 2 trauma centre and 1.0% of group Tr was admitted to a level 3 trauma centre. Similarly, 63.2% of patients in group P were admitted to a level 1 trauma centre, 29.0% to a level 2 trauma centre and 7.8% to a level 3 trauma centre.

Mortality and prediction

Group P had a slighter lower mortality rate (n=12 615, 10.9%, 95% CI 10.7% to 11.0%) when compared with group Tr (n=1292, 11.1%, 95% CI 10.6% to 11.7%). The predicted mortality based on RISC II was 11.0% in group P and 13.4% in group Tr (table 3). The AUC was 0.938 in group P and 0.867 in group Tr (figure 2).

Table 3

Comparison of the RISC II in group P vs group TR

	Group P	Group tr
n	116 112	11 604
In-hospital mortality, n (%)	12 615 (10.9)	1292 (11.1)
95% CI for hospital mortality	(10.7 to 11.01)	(10.6 to 11.7)
Expected mortality based on the RISC II (%)	11.0%	13.4%
Difference between observed and predicted mortality	0.1%	2.3%

n, number; RISC II, Revised Injury Severity Classification II.

Figure 2

Area under the ROC curve for the RISC II in primary admitted patients (group P, blue) and transferred patients (group TR, red). AUC, area under the curve; RISC II, Revised Injury Severity Classification II; ROC, receiver operating characteristic.

The visualisation of the observed versus predicted mortality shows a substantial difference when comparing group P with group Tr (figure 3).

Figure 3

Superior calibration of the prediction model in group P when compared with group Tr.

Discussion

This study aimed to assess the discrimination and calibration of the RISC II in secondary transferred polytrauma patients and revealed the following points: (1) observed and predicted mortality match perfectly in primary admitted patients but survival was 2.3% better in transferred cases; (2) discrimination was worse in transferred patients when compared with primary admitted patients and (3) the group of transferred patients consisted mostly of the elderly with head injuries after ground level falls.

RISC was initially developed to optimise the TRISS score by adding new prognostic factors to optimise adjustment for injury severity. RISC II was later developed to adjust for the improved clinical treatment strategies that have led to an overestimation of mortality by the RISC: The observed mortality was more than 2% below the predicted. The present result show further, that the way of admission has a certain effect on predicting mortality. From a pure academic point of view, the inclusion of the way of admission might therefore improve the prediction model for mortality in polytrauma. In clinical practice, a recent study has shown that scores like the RISC II are needed to predict mortality, however, the routine applicability might be hindered by its complexity.23 Clinical decision making based on RISC II should therefore take the way of admission into consideration and reflect the overestimation for mortality in transferred patients.

RISC II includes variables that are time independent, such as injury severity and distribution or patient demographics. It, however, also includes variables that are time dependent. These include physiological responses. The primary hospital might, therefore, have performed some measures to alter the physiologic response that further leads to an increased discrepancy of the predicted versus observed mortality, based on final admission values. Despite this, some studies indicate that the transfer status does not appear to be a significant predictor of survival.24

Secondary transfer might further be indicated in patients with specific medical problems that are not trauma associated. In geriatric trauma, the collaboration of geriatricians and trauma surgeons improves outcome in level 1 trauma centres.25 These factors are not routinely incorporated in prediction models and might therefore lead to less precise prediction models.

The variables that build the prediction model might be chosen by expert opinion and literature review26 or following machine learning algorithms and scoping of data.27 The quality of the prediction model improves with inclusion of an increasing number of variables from different pathophysiologic systems.5 In cases where machine learning algorithms recommend variables that are not routinely used in clinical practice, the expert opinion might provide guidance in treatment and estimating outcome.28

The initial admission of the trauma patient to a regional or local trauma centre might be based on an on-scene underestimation of the trauma.29 The on-scene assessment of intracranial injuries is based on clinical evaluation and clinical tests, including the GCS, that might deteriorate quickly and require secondary transfers.30 The pathophysiological response to trauma not only depends on the trauma energy but also on the patient’s condition prior to the injury. The geriatric patient on oral anticoagulants might suffer a more severe head injury after a ground level fall.31 On the other hand, overtriage in trauma represents a well-known problem.32 Overtriage might lead to the admission of patients to a level 1 trauma centre that do not require a level 1 trauma centre and falsely increase the survival rate.33

Calibration and discrimination analyses of prediction models are necessary, especially when the models aim to support clinical decision making.34 Reasons for poor calibration might include overfitting and measurement errors. For clinical applicability, adequate calibration is recommended and required. The current investigation showed that calibration and discrimination depend on the way of admission in trauma patients. Consequently, it is required to take the way of admission into consideration during the development of new prediction models and to adjust clinical quality control measures for the way of admission.

Strengths and limitations

This is an evaluation of a prediction model based on retrospective analysis from a registry and is thus prone to any pitfalls in this kind of analysis. There is a permanent check of data quality during data entry and selective source data verification, but it is impossible to verify each data point in the registry. However, the cases were representative since documentation and more than 95% of cases have occurred since 2013. Based on the yearly evaluation and the requirement for trauma centres to be involved in the registry to become certified trauma centres, we believe that the data quality is rather high. Furthermore, the availability of an average of 12 of 13 pieces of information per patient that are required to calculate the RISC II supports the data quality.

A further limitation is that we are not able to link treatment episodes for an individual patient from different hospitals. Due to data protection regulations, the scientific dataset is blinded for date/time of accident, hospital and personal identifiers.

The number of patients who died during transportation are not included in the present study. This might falsely increase the survival rate in transferred patients. The same is true for patients who died during primary transport or patients that died on-scene. There is a potential for selection bias based on the inclusion of patients that survived the initial or secondary transportation. Since this bias might be present in both groups and the data does not allow further investigations, we were not able to correct for this potential bias.

Conclusion

The way of admission might impact the prediction of mortality in severely injured patients. The fact that a patient could be transferred seems to be a bias towards better survival. Calibration and discrimination of prediction models alter and depend on the way of admission. The way of admission should therefore be taken into consideration and prediction of mortality of secondary transferred polytrauma patients should be modelled separately from primary transferred patients.