Validation of the artificial intelligence-based trauma outcomes predictor (TOP) in patients 65 years and older.

BACKGROUND: The Trauma Outcomes Predictor tool was recently derived using a machine learning methodology called optimal classification trees and validated for prediction of outcomes in trauma patients. The Trauma Outcomes Predictor is available as an interactive smartphone application. In this study, we sought to assess the performance of the Trauma Outcomes Predictor in the elderly trauma patient.
METHODS: All patients aged 65 years and older in the American College of Surgeons-Trauma Quality Improvement Program 2017 database were included. The performance of the Trauma Outcomes Predictor in predicting in-hospital mortality and combined and specific morbidity based on incidence of 9 specific in-hospital complications was assessed using the c-statistic methodology, with planned subanalyses for patients 65 to 74, 75 to 84, and 85+ years.
RESULTS: A total of 260,505 patients were included. Median age was 77 (71-84) years, 57% were women, and 98.8% had a blunt mechanism of injury. The Trauma Outcomes Predictor accurately predicted mortality in all patients, with excellent performance for penetrating trauma (c-statistic: 0.92) and good performance for blunt trauma (c-statistic: 0.83). Its best performance was in patients 65 to 74 years (c-statistic: blunt 0.86, penetrating 0.93). Among blunt trauma patients, the Trauma Outcomes Predictor had the best discrimination for predicting acute respiratory distress syndrome (c-statistic 0.75) and cardiac arrest requiring cardiopulmonary resuscitation (c-statistic 0.75). Among penetrating trauma patients, the Trauma Outcomes Predictor had the best discrimination for deep and organ space surgical site infections (c-statistics 0.95 and 0.84, respectively).
CONCLUSION: The Trauma Outcomes Predictor is a novel, interpretable, and highly accurate predictor of in-hospital mortality in the elderly trauma patient up to age 85 years. The Trauma Outcomes Predictor could prove useful for bedside counseling of elderly patients and their families and for benchmarking the quality of geriatric trauma care.

Introduction

The Census Bureau estimates that 49 million individuals in the United States were 65 years and older in 2016. By 2034, this number is projected to rise to 77 million, outnumbering the 76.5 million individuals under the age of 18. As the US population continues to age, the number of traumatic injuries in the elderly patient population will inevitably continue to increase. Decreased physiologic reserve, frailty, malnutrition, and frequent comorbidities all contribute to poor outcomes in this patient cohort. , In turn, the high rate of morbidity and mortality in the elderly patient often translates to longer hospital stays and contributes to increased health care utilization.

An important aspect of the delivery of care to the injured geriatric patient and family includes an early discussion regarding the goals of care and treatment plan. To guide discussions and facilitate decision making, accurate knowledge about the patient’s prognosis is crucial. Frequently, discussions are guided by the personal experience of the treating physician. At other times, the surgeon may choose to use validated prognostic calculators after trauma, such as the Trauma Injury Severity Score (TRISS) or the Geriatric Trauma Outcome Score (GTOS). ,

Although these models have predictive power, they share a common methodology whereby every variable in the model contributes the same amount of risk toward the outcome, regardless of other variables in the model. This makes the predictive model linear and thus insensitive to the clinical context to which it is applied. However, the clinical reality is that the interaction among patient demographics, comorbidities, vital signs, and the severity of injury at presentation in determining patient outcome is almost never linear or additive.

To illustrate how the interaction between demographics can be nonlinear, consider a 70-year-old man who fell from standing. The severity of injury at presentation may determine what factors influence his risk of mortality. For instance, if the patient presents alert and cooperative, certain comorbidities such as cirrhosis or heart failure might play significantly into the probability of surviving his hospital stay. Whereas when the injury is more severe and the patient presents with a depressed Glasgow Coma Scale (GCS), the extent of injury becomes a major determinant in survival, and the relative importance of comorbidities wanes.

Novel machine learning (ML) technology has the ability to generate models that capture nonlinear patterns in data. Our group has created the Trauma Outcomes Predictor (TOP), a smartphone-based risk prediction tool for trauma patients that uses ML optimal classification trees (OCTs), and validated the tool for predicting outcomes among all trauma patients. Although the accuracy of TOP in predicting hospital mortality and 9 in-hospital complications has been validated across all trauma patients, it has not been examined specifically in the geriatric trauma population, in which prognostic discussions are of particular importance but risk prediction is historically more challenging. In this study, we aimed to assess the performance of TOP in trauma patients aged 65 years and older.

Methods

Database and patient population

The American College of Surgeons Trauma Quality and Improvement Program (ACS-TQIP) database year 2017 was selected for this validation study. With contributions from more than 800 level I or II ACS-verified or state-designated trauma center centers across the United States, the database consists of more than 100 data items, including patient demographics, comorbidities, type and mechanism of injury, Injury Severity Score (ISS), Abbreviated Injury Scale (AIS), prehospital and emergency department (ED) vital signs, diagnoses, complications, and mortality. All patients older than 65 years included in the 2017 ACS-TQIP database were included. Three age subgroups were also created: 65–74, 75–84, and ≥85 years old. This study was approved by the Mass General Brigham Institutional Review Board.

The OCT methodology and the TOP smartphone application

The OCT methodology used for the development of the TOP calculator has been previously described in detail.10, 11, 12 Briefly, OCTs are a subset of ML methodologies that examine all input variables and select the most important ones for each tree to optimize accuracy. To do this, OCTs reboot themselves with each set of variables to try all conceivable combinations and determine the most impactful ones. In the process, OCTs establish specific cutoffs for each node based on knowledge about downstream and upstream impacts, as opposed to top-down approaches used by other tree-based models such as classification and regression trees. A unique OCT is generated for every outcome of interest. The structure of the mortality prediction OCT with a magnification of one of its terminal nodes is presented in Figure 1 . The example terminal node shows a split on the age of 42. For patients older than 42 years, the next question is whether the patient has liver cirrhosis. If so, the predicted mortality is 9.76%, compared with a patient who does not have cirrhosis, for whom there is an additional question about GCS on presentation. For patients with a GCS greater than 13, the predicted mortality is 1.26%, and for those less than 13, the predicted mortality is 2.82%. The splits on each of these nodes, in addition to the nodes themselves, are not determined by the developers but instead generated by the OCT algorithm in a way that optimizes overall performance of the model. An additional advantage of OCTs is their interpretability, unlike other “black-box” ML models used in risk prediction that risk introducing existing bias and disparities into the models. With OCTs, the clinicians are able to easily follow the algorithms’ reasoning and understand how the final predictions were made.

Figure 1

The optimal classification tree used by Trauma Outcome Predictor (TOP) to predict inpatient mortality, with a magnification on one of the terminal nodes.

The user-friendly TOP smartphone application that is built on the OCTs is presented in Figure 2 . This example demonstrates how the questions generated by the application differ depending on an individual patient’s characteristics. For instance, the patient on the left has a GCS of 15 on presentation, whereas the patient on the right has a GCS of 10, and the subsequent questions differ to ultimately provide individual mortality estimates for each patient.

Figure 2

Example screen shot of the Trauma Outcome Predictor (TOP) application in predicting mortality after blunt injury. TOP is interactive, and the answer to a question dictates the next question. In this case, the value of the Glasgow Coma Scale on emergency department presentation takes the algorithm in a different direction.

Primary and secondary outcomes

The primary outcome was the accuracy of prediction of in-hospital mortality for blunt and penetrating trauma patients. Secondary outcomes included the prediction of combined morbidity (ie, the occurrence of any of 9 in-hospital complications) as well as individual complications across both mechanisms of injury. Individual complications included acute respiratory distress syndrome, cardiac arrest requiring cardiopulmonary resuscitation, organ space surgical site infections, deep surgical site infections, unplanned intubation, severe sepsis, acute kidney injury, deep venous thrombosis, and pulmonary embolism.

Measuring the performance of TOP

The area under the curve (AUC) metric, or c-statistic, was used to measure the ability of algorithms to predict in-hospital mortality and individual and combined morbidity of 9 complications for patients 65 years and older, as well as the 3 age subgroups.

Results

A total of 260,505 patients were included, of which 39%, 40%, and 21% were in the 65–74, 75–84, and ≥85 years age groups, respectively. The median patient age was 77 years (71–84), 56.7% were females, and hypertension (64.5%), diabetes mellitus (25.8%), and chronic obstructive pulmonary disease (13.0%) were the most common comorbid conditions (Table I ). Most injuries resulted from falls (80.6%) and motor vehicle collisions (12.4%). The median ISS was 9 (5–10), and a severe head injury (AIS >3) was present in 18.2% of patients. Most patients were admitted to the floor (54.7%) or ICU (23.4%) from the ED (Table II ).

Table I

Demographic characteristics and comorbidities in geriatric trauma patients

Patient characteristics	Patients ≥65N = 260,505	Age 65–74N = 101,324	Age 75–84N = 104,819	Age ≥85N = 54,362	P value
Demographics
Female	147,828 (56.7%)	50,172 (49.5%)	62,133 (59.3%)	35,523 (65.3%)	<.001
Age, median (IQR)	77.0 (71.0, 84.0)	70.0 (67.0, 72.0)	80.0 (77.0, 82.0)	87.0 (86.0, 88.0)	<.001
Transferred	70,501 (27.1%)	28,120 (27.8%)	28,919 (27.6%)	13,462 (24.8%)	<.001
Race					<.001
White	226,201 (86.8%)	85,430 (84.3%)	91,867 (87.6%)	48,904 (90.0%)
Asian	17,256 (6.6%)	7,232 (7.1%)	6,979 (6.7%)	3,045 (5.6%)
Black or African American	13,886 (5.3%)	7,307 (7.2%)	4,740 (4.5%)	1,839 (3.4%)
Other race	3,162 (1.2%)	1,355 (1.3%)	1,233 (1.2%)	574 (1.1%)
Comorbidities
Bleeding disorder	8767 (3.4%)	2,686 (2.7%)	3,813 (3.6%)	2,268 (4.2%)	<.001
Active chemotherapy	2311 (0.9%)	1081 (1.1%)	958 (0.9%)	272 (0.5%)	<.001
Congestive heart failure	23173 (8.9%)	6547 (6.5%)	10073 (9.6%)	6553 (12.1%)	<.001
Active smoking	22269 (8.5%)	14298 (14.1%)	6475 (6.2%)	1496 (2.8%)	<.001
Chronic renal failure	9359 (3.6%)	3434 (3.4%)	4040 (3.9%)	1885 (3.5%)	<.001
Cerebrovascular accident	15618 (6.0%)	5347 (5.3%)	6785 (6.5%)	3486 (6.4%)	<.001
Diabetes mellitus	67,138 (25.8%)	28,222 (27.9%)	27,948 (26.7%)	10,968 (20.2%)	<.001
Disseminated cancer	3,471 (1.3%)	1,354 (1.3%)	1,471 (1.4%)	646 (1.2%)	.002
COPD	33,865 (13.0%)	13,457 (13.3%)	14,174 (13.5%)	6,234 (11.5%)	<.001
Steroid use	4,402 (1.7%)	1,700 (1.7%)	1,877 (1.8%)	825 (1.5%)	<.001
Cirrhosis	2,833 (1.1%)	1,862 (1.8%)	774 (0.7%)	197 (0.4%)	<.001
Myocardial infarction	5,028 (1.9%)	1,864 (1.8%)	2,129 (2.0%)	1,035 (1.9%)	.006
Peripheral artery disease	3,380 (1.3%)	1,192 (1.2%)	1,461 (1.4%)	727 (1.3%)	<.001
Hypertension	167,941 (64.5%)	59,408 (58.6%)	70,712 (67.5%)	37,821 (69.6%)	<.001

COPD, chronic obstructive pulmonary disease.

Table II

ED and injury characteristics in geriatric trauma patients

Patient characteristics	Patients ≥65N = 260,505	Age 65–74N = 101,324	Age 75–84N = 104,819	Age ≥85N = 54,362	P value
ED parameters
ED pulse >100 bpm	32,017 (12.6%)	14,322 (14.5%)	11,987 (11.8%)	5,708 (10.8%)	<.001
ED SBP <110 mm Hg	22,906 (9.0%)	10,598 (10.8%)	8,492 (8.3%)	3,816 (7.2%)	<.001
ED GCS <8	7,376 (3.0%)	3,470 (3.6%)	2,818 (2.9%)	1,088 (2.2%)	<.001
Injury parameters
ISS, median (IQR)	9.0 (5.0, 10.0)	9.0 (4.0, 11.0)	9.0 (5.0, 10.0)	9.0 (5.0, 10.0)	<.001
AIS Head ≥3	46,965 (18.2%)	17,203 (17.1%)	20,014 (19.3%)	9,748 (18.1%)	<.001
AIS Neck ≥3	8,282 (3.2%)	3,237 (3.2%)	3,347 (3.2%)	1,698 (3.1%)	.7
AIS Face ≥3	421 (0.2%)	231 (0.2%)	139 (0.1%)	51 (0.1%)	<.001
AIS Thorax ≥3	31,707 (12.2%)	15,141 (15.0%)	11,556 (11.1%)	5,010 (9.2%)	<.001
AIS Abdomen ≥3	6,004 (2.3%)	2,896 (2.9%)	2,193 (2.1%)	915 (1.7%)	<.001
AIS Extremity ≥3	68,072 (26.2%)	21,946 (21.8%)	28,790 (27.6%)	17,336 (32.0%)	<.001
Mechanisms of Injury					<.001
Blunt - Fall	210,056 (80.6%)	71,696 (70.8%)	88,556 (84.5%)	49,804 (91.6%)
Blunt - MVT cyclist/pedestrian	5,917 (2.3%)	3,669 (3.6%)	1,820 (1.7%)	428 (0.8%)
Blunt - MVT occupant	32,335 (12.4%)	18,090 (17.9%)	11,070 (10.6%)	3,175 (5.8%)
Blunt - Other	9,062 (3.5%)	5,750 (5.7%)	2,546 (2.4%)	766 (1.4%)
Penetrating - Gunshot wound	1075 (0.4%)	697 (0.7%)	305 (0.3%)	73 (0.1%)
Penetrating - Other/Mixed	416 (0.2%)	283 (0.3%)	102 (0.1%)	31 (0.1%)
Penetrating - Stab wound	1644 (0.6%)	1139 (1.1%)	420 (0.4%)	85 (0.2%)
ED discharge disposition					<.001
Floor	136,932 (54.7%)	51,079 (52.4%)	55,294 (55.0%)	30,559 (58.4%)
Observation unit	8,721 (3.5%)	3,620 (3.7%)	3,459 (3.4%)	1,642 (3.1%)
Telemetry/step-down unit	31,623 (12.6%)	11,607 (11.9%)	12,947 (12.9%)	7,069 (13.5%)
Operating room	14,503 (5.8%)	7.588 (7.8%)	5,027 (5.0%)	1,888 (3.6%)
Intensive care unit	58,487 (23.4%)	23,532 (24.2%)	23,816 (23.7%)	11,139 (21.3%)

AIS, Abbreviated Injury Score; ED, emergency department; GCS, Glasgow Coma Scale; ISS, Injury Severity Score; SBP, systolic blood pressure.

The in-hospital mortality rate was 4.2%, and 3.2% of patients developed 1 or more complications. The most common in-hospital complications were unplanned intubations (1.3%), acute kidney injuries (0.7%), and cardiac arrest (0.7%) (Table III ).

Table III

In-hospital and discharge outcomes of geriatric trauma patients

Patient characteristics	Patients ≥65N = 260,505	Age 65–74N = 101,324	Age 75–84N = 104,819	Age ≥85N = 54,362	P value
Hospital mortality	10,840 (4.2%)	3,426 (3.4%)	4,729 (4.5%)	2,685 (4.9%)	<.001
Combined morbidity	8,275 (3.2%)	3,611 (3.6%)	3,266 (3.1%)	1,398 (2.6%)	<.001
Acute kidney injury	1,759 (0.7%)	678 (0.7%)	745 (0.7%)	336 (0.6%)	.097
ARDS	653 (0.3%)	345 (0.3%)	231 (0.2%)	77 (0.1%)	<.001
Cardiac arrest requiring CPR	1,704 (0.7%)	709 (0.7%)	689 (0.7%)	306 (0.6%)	.006
Deep SSI	93 (<0.1%)	47 (<0.1%)	33 (<0.1%)	13 (<0.1%)	.053
Deep venous thrombosis	1,349 (0.5%)	643 (0.6%)	506 (0.5%)	200 (0.4%)	<.001
Organ space SSI	53 (<0.1%)	33 (<0.1%)	17 (<0.1%)	3 (<0.1%)	<.001
Pulmonary embolism	654 (0.3%)	314 (0.3%)	235 (0.2%)	105 (0.2%)	<.001
Unplanned intubation	3,469 (1.3%)	1,547 (1.5%)	1,387 (1.3%)	535 (1.0%)	<.001
Severe sepsis	950 (0.4%)	434 (0.4%)	361 (0.3%)	155 (0.3%)	<.001
Hospital discharge disposition					<.001
Home (with or without services)	106,823 (43.2%)	52,823 (54.4%)	38,726 (39.1%)	15,274 (29.9%)
Rehab or long-term care	131,549 (53.2%)	41,172 (42.4%)	56,809 (57.3%)	33,568 (65.7%)
Other	8,874 (3.6%)	3,043 (3.1%)	3,575 (3.6%)	2,256 (4.4%)

ARDS, acute respiratory distress syndrome; CPR, cardiopulmonary resuscitation; SSI, surgical site infection.

Prediction of in-hospital mortality

Blunt trauma patients

The c-statistic for predicting in-hospital mortality in blunt trauma patients ≥65 was 0.83. On subgroup analysis, the performance of TOP was the highest among patients aged 65 to 74 (c-statistic: 0.86) and lowest among patients ≥85 (c-statistic: 0.77) (Figure 3 , A).

Figure 3

Trauma Outcome Predictor (TOP) model area under the receiver operator characteristic (ROC) curve; for (A) blunt injury mortality and (B) penetrating injury mortality.

Penetrating trauma patients

The c-statistic for predicting in-hospital mortality in penetrating trauma patients ≥65 was 0.92. On subgroup analysis, the performance of TOP was the highest among patients aged 65 to 74 (c-statistic: 0.93) and lowest among patients 75 to 84 (c-statistic: 0.88) (Figure 3, B).

Prediction of in-hospital morbidity

The c-statistic for predicting in-hospital morbidity in blunt trauma patients ≥65 was 0.69. On subgroup analysis, the performance of TOP was the highest among patients aged 65 to 74 (c-statistic: 0.73) and lowest among patients ≥85 (c-statistic: 0.63) (Figure 4 , A).

Figure 4

Trauma Outcome Predictor (TOP) model area under the receiver operator characteristic (ROC) curve; for (A) blunt injury morbidity and (B) penetrating injury morbidity.

The c-statistic for predicting in-hospital morbidity in penetrating trauma patients ≥65 was 0.71. On subgroup analysis, the performance of TOP was the highest among patients aged 65 to 74 (c-statistic: 0.72) and lowest among patients ≥85 (c-statistic: 0.68) (Figure 4, B).

Prediction of individual complications

Figure 5 is a radar plot that shows the performance of TOP in predicting individual complications. Among patients with blunt trauma, TOP best predicted acute respiratory distress syndrome (c-statistic: 0.75) and cardiac arrest (c-statistic: 0.75). Among patients with penetrating trauma, TOP best predicted organ space (c-statistic: 0.95) and deep surgical site infections (c-statistic: 0.84).

Figure 5

Radar plots depicting the performance (c-statistic) of Trauma Outcome Predictor (TOP) in predicting 9 individual complications in blunt (B) and penetrating (P) trauma patients. The axis extends from the outermost circle, with associated c-statistic (area under the curve) of 1, and each concentric circle moving inward signifies an interval of 0.2.

Discussion

In this study, we examine the ability of TOP to predict in-hospital outcomes in the vulnerable geriatric trauma patient. We found that TOP predicts in-hospital mortality very well in blunt and penetrating trauma patients older than 65 years of age. In addition, TOP has moderate performance in predicting in-hospital morbidity among patients 65 to 74 years old, but its performance drops in older patients, especially those older than 85 years. Despite having good to excellent performance among certain geriatric patients, TOP performed better in the general cohort of trauma patients, when compared to geriatric trauma patients as evidenced by the respective AUCs.

Prognostication of outcome and handling family discussions in the injured geriatric patient is never an easy task. In addition to the physical components of injury, the physician has to take into account the cumulative deficit burden model, which consists of preinjury physical, social, functional, and cognitive factors when counseling family. , To add to the challenge, it has been shown that some families overestimate their loved one’s odds of recovery and are reluctant to accept poor prognoses, whereas others tend to underestimate them and are more inclined to withdraw care. Consequently, surrogate decision makers tend to pursue aggressive treatment options early in the hospitalization course, but often prefer palliative care once the poor prognosis is apparent. This translates into prolonged intensive care unit stays and numerous invasive interventions for the severely injured geriatric patient, which unfortunately do not translate to the survival of hospital stay. Additional evidence suggests that patients typically underestimate their remaining life expectancy, whereas physicians have been found to overestimate patients’ life expectancy. These data reinforce the accuracy of TOP in predicting in-hospital mortality and major complications in the geriatric trauma patient. Compared to relying on personal experience or anecdotal data alone, TOP can substantially help surgeons achieve objective and rapid assessments of risk and set realistic patient and family expectations early in the hospital course. This is mostly applicable to geriatric patients younger than 85 years, where TOP proved to have the best performance.

Two other notable risk calculators that have found application in the prognostication of geriatric mortality after trauma are TRISS and the GTOS. The TRISS method employs a logistic regression model that includes the Injury Severity Score and Revised Trauma Score variables to generate a mortality estimate for blunt and penetrating injuries. The discrimination of TRISS for prediction of mortality as determined by the area under the curve ranges from 0.69 to 0.90 in geriatric patients. , The greatest advantage of TRISS is its durability, where it remains the most recognized and accepted mortality estimator in trauma patients since its creation more than 30 years ago. Although the Dutch Trauma Registry has recently created a modified version of TRISS to extend its use to the elderly population, it has been shown that TRISS has high misclassification rates in patients older than 54 years, which can be attributed to the dichotomization of age at 55 years. TOP, on the other hand, takes into account the nuances of advanced age, where the trees allow splits at various age cutoffs at different levels of the tree. Furthermore, TRISS takes first recorded physiologic parameters, which is problematic because it relies on the effort of emergency medical personnel to capture multiple variables, which might not always be available and might not be missing-at-random in the severely injured patients. To avoid this problem, TOP only includes emergency department variables, which will be readily available for the treating physician.

Conversely, the GTOS was developed as a prognostic tool specifically for elderly mortality during the index hospitalization and has an AUC of 0.84–0.86. , The GTOS is composed of 3 variables: age, Injury Severity Score, and blood transfusion during the first 24 hours of hospitalization. Although simple to use and available online, GTOS requires knowledge of transfusion in 24 hours, which delays an accurate mortality estimation. TOP, meanwhile, available as a smartphone application, can be applied for prognostication as soon as the patient is admitted from the emergency department. Moreover, it is important to note that although the 2 above calculators provide mortality estimates to geriatric trauma patients, they are not capable of predicting the occurrence of a complication during hospitalization, an ability that is possessed by TOP. Additionally, TOP has the potential to be used as a tool for benchmarking the quality of care among geriatric trauma patients, an area that is being currently explored by the authors.

Our study has a number of limitations. First, patients with less severe injuries are not included in the ACS-TQIP database. Second, an AI or ML methodology only performs as well as the data it trains on. For the elderly trauma patient, especially those older than 85, the data points currently collected by the ACS-TQIP are not sufficient. Additional information on patients’ cognitive levels, for example, might considerably improve predictions in the geriatric population. Third, the aggregate of all patients in the United States, on which the model is trained, might not be an accurate representation of any specific patient demographic at a local hospital level. Fourth, TOP focuses solely on in-hospital mortality and morbidity risk, which does not provide a complete prognostic picture, since severe traumatic injuries are known to have an increased risk of mortality beyond the index admission.

In conclusion, TOP is an interpretable and accurate calculator used to predict in-hospital mortality and complications in the geriatric trauma patient up to the age 85 years. TOP could prove useful for bedside counseling of elderly patients and their families regarding the risk of mortality and could guide important goals of care discussions in this population.

Funding/Support

This study was supported by a grant from CRICO/Risk Management Foundation of the Harvard Medical Institutions.

Conflict of interest/Disclosure

Drs Zhou, Dunn, and Bertsimas are founders of Interpretable AI.