Supplementing Existing Societal Risk Models for Surgical Aortic Valve Replacement With Machine Learning for Improved Prediction.

Background This study evaluated the role of supplementing Society of Thoracic Surgeons (STS) risk models for surgical aortic valve replacement with machine learning (ML). Methods and Results Adults undergoing isolated surgical aortic valve replacement in the STS National Database between 2007 and 2017 were included. ML models for operative mortality and major morbidity were previously developed using extreme gradient boosting. Concordance and discordance in predicted risk between ML and STS models were defined using equal-size tertile-based thresholds of risk. Calibration metrics and discriminatory capability were compared between concordant and discordant patients. A total of 243 142 patients were included. Nearly all calibration metrics were improved in cases of concordance. Similarly, concordance indices improved substantially in cases of concordance for all models with the exception of deep sternal wound infection. The greatest improvements in concordant versus discordant cases were in renal failure: ML model (concordance index, 0.660 [95% CI, 0.632-0.687] discordant versus 0.808 [95% CI, 0.794-0.822] concordant) and STS model (concordance index, 0.573 [95% CI, 0.549-0.576] discordant versus 0.797 [95% CI, 0.782-0.811] concordant) (each P<0.001). Excluding deep sternal wound infection, the concordance indices ranged from 0.549 to 0.660 for discordant cases and 0.674 to 0.808 for concordant cases. Conclusions Supplementing ML models with existing STS models for surgical aortic valve replacement may have an important role in risk prediction and should be explored further. In particular, for the roughly 25% to 50% of patients demonstrating discordance in estimated risk between ML and STS, there appears to be a substantial decline in predictive performance suggesting vulnerability of the existing models in these patient subsets.

deep sternal wound infection

logistic regression

machine learning

surgical aortic valve replacement

Society of Thoracic Surgeons

Clinical Perspective

What Is New?

This is a study of the Society of Thoracic Surgeons National Database for isolated aortic valve replacement evaluating the role of supplementing existing risk models with machine learning for improved risk prediction.

Model performance was substantially improved in cases where machine learning and existing societal risk models displayed concordant risk prediction.

What Are the Clinical Implications?

Machine learning can be used independently and as an adjunct to existing societal risk models for improving risk prediction in cardiac surgery.

Further research is needed to identify why discordance exists in these models and leads to vulnerability in risk prediction in these patient subsets.

The Society of Thoracic Surgeons (STS) risk models derived from national STS registry data have long served as the gold standard for risk assessment and prognostication in adult cardiac surgery. Although the risk models have traditionally been associated with excellent calibration, they exhibit only moderate discriminatory capability, findings that are consistent in other population‐level risk models in clinical medicine. , Interest in machine learning (ML) has risen exponentially recently, and the role of ML has been evaluated in several series in cardiac surgery. , , Although improvements in predictive capability of risk models has been demonstrated with ML, what has yet to be explored is the role that supplementing different approaches may have in better understanding risk model vulnerabilities and potential avenues for improvement. The aim of this study was to evaluate the potential role of supplementing ML and STS risk models in predicting outcomes after surgical aortic valve replacement (SAVR).

Methods

Study Cohort

The authors declare that all supporting data are available within the article and its online supplementary files. Adults undergoing isolated SAVR in the STS Adult Cardiac Surgery Database between 2007 and 2017 were included. This corresponded to STS Adult Cardiac Surgery Database data versions 2.61, 2.73, and 2.81. Patients undergoing concomitant coronary artery bypass grafting, other valve surgery, or other major cardiac procedures such that they were excluded from the isolated SAVR category as defined by the STS were excluded from analysis. As this registry contains deidentified data with no direct patient identifiers and was originally collected for nonresearch purposes, the Duke University Health System Institutional Review Board deemed this research exempt from review, as it does not qualify as human subjects research. The requirement for informed consent for this study for each individual subject was waived.

ML Models

The outcomes for which models were evaluated included operative mortality, each major morbidity (acute renal failure, prolonged ventilation, reoperation, stroke, and deep sternal wound infection [DSWI]), and the composite outcome of either operative mortality or major morbidity. The clinical definitions and criteria for these were defined by the STS. The ML algorithm that was used was extreme gradient boosting, or XGBoost. The methodologic approach, which involved randomly dividing the study cohort into training (80%) and testing (20%) cohorts, was detailed previously. The approach to ML model derivation and validation and evaluation of model performance were also detailed previously. In addition, categories of risk (low, intermediate, high) were defined using equal‐size tertiles based on STS predicted risk. Matrices were then created and the observed rates of outcomes in the testing set were evaluated on the basis of concordance and discordance in predicted risk between the STS and ML risk models. Concordance was defined as similar categorization of risk (low, intermediate, or high) by STS and ML, whereas discordance was defined as dissimilar categorization.

Predictive Utility of Concordance and Discordance

For each outcome, patients were stratified according to having concordant versus discordant predicted risk in STS and ML models. The predictive performance of each model was then evaluated and compared between concordant and discordant patients. Comparisons in performance for each outcome for concordant versus discordant patients was evaluated for each ML and STS model separately. In addition, the impact on performance of a combined model that represented the average predicted risk between the ML and STS models was evaluated as well. Performance measures were evaluated in the testing sets and included metrics related to calibration and discriminatory ability of the models. Calibration was assessed using observed‐to‐expected ratios of the outcome, with an optimal value equaling 1; calibration‐in‐the‐large or y‐intercept of the calibration plot, with an optimal value equaling 0; and slope of the calibration curve, with an optimal value equaling 1. Discriminatory ability of the models was measured and compared using the area under the receiver operating characteristic curve, or concordance index.

Multivariable logistic regression models were also generated to identify independent predictors of discordance in predicted risk between ML and STS models for each outcome. These models were generated by evaluating each preoperative variable in the STS registry in univariate logistic regression analysis. Those variables with a significant association (exploratory P<0.05) with the outcome of discordance in predicted risk were then entered into the multivariable model. Variables with >10% missing data were excluded from these models. In addition to models using only preoperative variables, the impact of including intraoperative data to evaluate discordance in predicted risk was examined. Intraoperative variables, which included cardiopulmonary bypass time, aortic cross‐clamp time, intraoperative blood product usage, and use of a mechanical valve, were evaluated in univariate logistic regression analysis, and those variables predictive of discordance were entered into separate multivariable models. The ML models were developed and validated using Python programming software (Python Software Foundation, Wilmington, DE). The remaining statistical analyses were performed with STATA software version 14 (StataCorp, College Station, TX).

De Novo Logistic Regression Models

The STS risk models are updated every few years with adjusted regression coefficients to reflect temporal changes as well as ongoing accrual of data within the registry. Therefore, we conducted a subsequent analysis in which we developed de novo logistic regression (LR) multivariable models for each outcome and used these instead of the STS risk models to ensure that the findings persisted. For these LR models, we evaluated only the preoperative variables that were evaluated for inclusion in the STS and ML models, thus ensuring that no extraneous data were incorporated that were not evaluated for use in the other models. Further, the same derivation cohort of patients from which the ML models were constructed were used to derive these de novo LR models, and the same external validation set of patients was used for validation. Univariate logistic regression was initially conducted on each candidate variable, and those with an exploratory P value of <0.05 were entered into a multivariable model for that particular outcome. Variables with >10% missing data were excluded. Similar analyses comparing discriminatory capability and calibration in cases of concordance versus discordance with ML were conducted but with these LR models instead of STS models.

Results

Baseline Characteristics

A total of 243 142 patients undergoing isolated SAVR were included in this analysis. The baseline characteristics of the overall population as well as the development, validation, and performance of the ML models for each of the 7 outcomes is detailed in prior work. In brief, the ML models were well calibrated and improved the discriminatory ability of the STS models for 5 of 7 outcomes (comparable performance between ML and STS for stroke and DSWI) in the overall study population. The equal‐patient‐size tertile‐based thresholds derived from the STS predicted risk of outcomes were also detailed previously.

Rates and Predictors of Discordance in Predicted Risk Between ML and STS

Rates of discordance in predicted risk between ML and STS in the testing sets were as follows: 26.4% operative mortality, 34.4% composite of mortality and morbidity, 46.0% renal failure, 33.1% prolonged ventilation, 50.6% reoperation, 25.0% stroke, and 63.3% DSWI. A comparison of baseline preoperative characteristics between discordant and concordant patients revealed substantial differences for each of the 7 outcomes evaluated (Tables S1 through S7). With the exception of reoperation and DSWI, the STS predicted risk of the remaining outcomes was significantly lower in discordant patients (Tables S1 through S7). Similarly, several differences in intraoperative characteristics were noted as well (Tables S1 through S7).

In separate multivariable models that were created for each of the outcomes to identify predictors of discordance, there were distinct patterns that were noted (Table 1). For example, diabetes mellitus, hypertension, increasing serum creatinine, increasing ejection fraction, and emergent operative status were each associated with concordance in the vast majority of models (Table 1). Older age, female sex, White race, mild chronic lung disease, cardiogenic shock, and aortic valve insufficiency were each associated with discordance in the majority of models (Table 1). Of the intraoperative variables, mechanical valve placement was associated with concordance, whereas intraoperative blood product transfusion was associated with discordance in predicted risk by the ML and STS models for the majority of outcomes studied (Table 1).

Table 1

Significant Predictors of Concordance and Discordance in Predicted Risk Between the ML and STS Models

Variable	Concordance							Discordance
	Mortality	Comp	Renal Failure	Prolonged Ventilation	Reoperation	Stroke	DSWI	Mortality	Comp	Renal Failure	Prolonged Ventilation	Reoperation	Stroke	DSWI
Age, y (increasing)									x	x	x	x		x
Female			x		x			x	x		x		x	x
White								x	x	x		x	x
BMI (increasing)										x	x
BSA (increasing)					x			x	x				x	x
Hyperlipidemia
Diabetes mellitus	x	x	x	x	x	x
Hypertension	x	x	x	x	x
Chronic lung disease—mild									x	x	x	x	x	x
Chronic lung disease—moderate	x			x								x	x	x
Chronic lung disease—severe	x	x	x	x								x	x	x
Dialysis							x	x			x	x
Creatinine (increasing)	x	x	x	x	x
Immunosuppressed			x								x	x
Infective endocarditis			x	x		x
PAD		x			x			x		x	x
CVD			x	x	x								x
FHCAD									x	x		x
Redo						x	x	x	x	x		x
Prior MI												x		x
Shock								x	x	x	x	x	x
IABP		x		x	x
AV insufficiency									x	x	x	x	x
EF (increasing)		x		x	x	x	x
Urgent status		x		x	x								x
Emergent status	x	x	x	x	x	x
Intraoperative*
CPB time (increasing)
Aortic XC time (increasing)
Blood transfusion								x	x	x	x	x
Mechanical valve		x			x	x

Intraoperative variables were added subsequently after identifying significant preoperative predictors. AV indicates aortic valve; BMI, body mass index; BSA, body surface area; Comp, composite of mortality or morbidity; CPB, cardiopulmonary bypass; CVD, cerebrovascular disease; DSWI, deep sternal wound infection; EF, ejection fraction; FHCAD, family history of coronary artery disease; IABP, intra‐aortic balloon pump; MI, myocardial infarction; ML, machine learning; PAD, peripheral arterial disease; STS, Society of Thoracic Surgeons; and XC, cross‐clamp.

Intraoperative variables were entered into the multivariable model only after fully executing the multivariable models using only preoperative variables.

Performance of the Models in Discordant Versus Concordant Cases

A comparison of calibration metrics demonstrated that the majority of metrics were improved in concordant cases as compared with discordant cases with the STS models (Figure 1 and Table 2). Similar trends in improvements in calibration metrics were observed with the ML models as well (Figure 2 and Table 3). A model that averaged the predicted risk between ML and STS again demonstrated improved calibration in the majority of outcomes with concordant patients (Table 4).

Figure 1

Improvement in calibration for Society of Thoracic Surgeons (STS) risk models for operative mortality in concordant cases.

 

Table 2

Improvement in Calibration Metrics of the STS Models in Cases of Concordance

STS Model
Operative Mortality	Discordant (n=12 615; 26.4%)	Concordant (n=35 191; 73.6%)
Observed‐to‐expected ratio	0.770	0.866
Calibration‐in‐the‐large	−0.267	−0.157
Slope of calibration curve	0.837	0.964

STS indicates Society of Thoracic Surgeons.

Figure 2

Improvement in calibration for machine learning risk models for operative mortality in concordant cases.

 

Table 3

Improvement in Calibration Metrics of the ML Models in Cases of Concordance

ML Model
Operative Mortality	Discordant (n=12 615; 26.4%)	Concordant (n=35 191; 73.6%)
Observed‐to‐expected ratio	0.860	1.017
Calibration‐in‐the‐large	−0.154	0.016
Slope of calibration curve	0.806	0.987

ML indicates machine learning.

Table 4

Improvement in Calibration Metrics of Models Averaging ML and STS Risk in Cases of Concordance

Average Model
Operative Mortality	Discordant (n=12 615; 26.4%)	Concordant (n=35 191; 73.6%)
Observed‐to‐expected ratio	0.812	0.935
Calibration‐in‐the‐large	−0.212	−0.072
Slope of calibration curve	1.325	1.030

ML indicates machine learning; and STS, Society of Thoracic Surgeons.

For all outcomes and all models with the exception of DSWI, the discriminatory capability or concordance index of the models was substantially higher in concordant cases (Figure 3 and Table 5). The greatest improvements in concordant versus discordant cases were in renal failure: ML model (concordance index, 0.660 [95% CI, 0.632–0.687] discordant versus 0.808 [95% CI, 0.794–0.822] concordant), STS model (concordance index, 0.573 [95% CI, 0.549–0.576] discordant versus 0.797 [95% CI, 0.782–0.811] concordant), and combined ML/STS model (concordance index, 0.641 [95% CI, 0.614–0.669] discordant versus 0.807 [95% CI, 0.793–0.821] concordant) (each P<0.001) (Table 5). Excluding DSWI, the concordance indices ranged from 0.549 to 0.660 for discordant cases and from 0.674 to 0.808 for concordant cases (Table 5).

Figure 3

Improvement in area under the receiver operating characteristic (ROC) curve for the Society of Thoracic Surgeons models for operative mortality in (A) concordant vs (B) discordant cases, and for machine learning models in (C) concordant vs (D) discordant cases.

 

Table 5

Improvement in Discriminatory Ability as Measured by Area Under the Receiver Operating Characteristic Curve in Cases of Concordance

Model	Concordance Index (95% CI)	Concordance Index (95% CI)	P Value
Operative Mortality	Discordant (n=12 615; 26.4%)	Concordant (n=35 191; 73.6%)
ML	0.634 (0.595–0.673)	0.774 (0.759–0.788)	<0.001
STS	0.614 (0.576–0.653)	0.769 (0.754–0.784)	<0.001
Average of both models	0.650 (0.614–0.686)	0.775 (0.760–0.789)	<0.001

ML indicates machine learning; and STS, Society of Thoracic Surgeons.

Findings Using De Novo LR Models

There were some differences noted in individual variables that were predictive of concordance or discordance between ML and LR as compared with ML and STS, although the majority remained similar (Table S8). The calibration metrics of the LR models were uniformly improved in cases of concordance with ML (Table S9). Although less pronounced, the majority of calibration metrics were improved in the ML models as well when concordant with LR (Table S10). The same findings held with the models developed by averaging the ML and LR predicted risk (Table S11). Similar to what was observed with the STS models, there was substantial improvement in concordance index in concordant cases across all model types and outcomes with the exception of DSWI (Table S12).

Discussion

Risk modeling plays a vital role in cardiac surgery with important implications in program evaluation, quality improvement, patient prognostication, therapy selection, and clinical trial development. Historically, the STS risk models have demonstrated excellent calibration but only moderate discriminatory capability, findings that are fairly common in most population‐level clinical risk models. Therefore, although the models can accurately assign the rate of occurrence of an outcome for a population at hand, the models are less capable of identifying specific patients in whom that outcome will occur. This limits the ability to provide accurate individual patient counseling and decision making.

Substantial improvements in discriminatory capability of risk models to a point of achieving “state‐of‐the‐art” performance with concordance indices >0.90 can potentially be obtained through several different mechanisms. Foremost, risk models are constrained by the available data points that are available for evaluation and inclusion in the models. If there are highly predictive elements, known or unknown, that are not captured within the data repository from which the model is built, this will likely constrain the performance of the model. Using ML techniques such as natural language processing and automated information extraction from electronic health records can help overcome the constraints of available data. This was demonstrated in a risk model for in‐hospital mortality that achieved “state‐of‐the‐art” predictive capability by taking advantage of the ability to analyze over 46 billion data points. Translating these methods into large national clinical registries such as the STS present unique logistical challenges, and this remains an area of active investigation.

Another approach to improving risk model performance is to use different modeling strategies. Much of the literature describing both logistic regression and ML models for cardiac surgery, similar to other clinical fields, have focused on comparing isolated, singular approaches. , , The concept of supplementing risk modeling approaches to determine potential utility is not necessarily novel in the ML world, but its application in the setting of large clinical registries such as the STS is new.

The main implication of the current analysis, which is the first of its kind to be performed in the STS national registry, is that supplementing ML and STS risk models allows us the ability to identify patient subsets where the STS risk models appear to be vulnerable. Our prior work demonstrated that the observed rates of outcomes in the training set fell within the range of predicted risk 100% of the time when there was concordance between the ML and STS models in predicted risk for each of the 7 outcomes studied in SAVR. Concordance between the models does not appear to augment the predictive performance to “state‐of‐the‐art,” but rather, in those cases that are discordant, there appears to be a drastic decline in area under the receiver operating characteristic curve.

Also of interest is that there appear to be specific patterns and individual variables that predict greater likelihood of being discordant consistently across outcomes. This suggests that we can likely identify clusters of patients for whom existing models are likely to be less reliable. What to do with this information is a matter for debate and requires input from multiple stakeholders, including national societal leaders.

Currently, predicted risks are communicated to patients, clinicians, and to the STS as absolute values. One implication of the current analysis may be to communicate confidence levels in our estimated risks as well. For example, we have strong confidence that the estimated risk is X or that we are 95% confident that the risk will fall between X and Y; or perhaps excluding patients whose estimates are discordant and of low confidence is prudent for hospital and surgeon evaluation. Regardless, further investigation into understanding why and how ML and STS models are calculating risk differently and improving risk prediction in discordant patients is important for improving performance of the models in the overall population.

The STS risk models in addition to the ML models we developed for the current analysis included only preoperative variables. Clinicians appreciate the notion that the postoperative course of a cardiac surgical patient is also largely dictated by intraoperative events. Multiple prior reports have indeed demonstrated strong associations between intraoperative variables such as longer cardiopulmonary bypass times, longer aortic cross‐clamp times, and intraoperative blood product transfusions with increased operative mortality and morbidity risk. , ,

In the context of identifying potential reasons why discordance may exist in predicted risk between ML and STS approaches, we also performed a subanalysis in which we added intraoperative variables reliably coded in the STS registry to determine if any were predictors of discordance. It is conceivable that catastrophic intraoperative events, for example, would dramatically alter the postoperative risk of mortality and morbidity in a patient who was otherwise low risk when considering only baseline preoperative variables. Interestingly, we found that neither cardiopulmonary bypass time nor aortic cross‐clamp time, both of which can be considered surrogates for intraoperative complications and operative efficiency, were predictive of discordance. Intraoperative blood transfusions, however, did reliably predict discordance in predicted risk for the majority of models.

Limitations

The current analysis evaluated only a specific ML algorithm. There are a plethora of other ML algorithms that exist and were not evaluated in this study, and therefore the generalizability of these results is unknown. Furthermore, we examined isolated SAVR using only the STS registry, and therefore whether these results extrapolate to other types of index cardiac operations remains to be elucidated. Other inherent limitations include the retrospective nature of the study design as well as errors in data entry, as is encountered with any multicenter registry.

Conclusions

This study of 243 142 patients undergoing isolated SAVR in the STS national database explored the utility of supplementing ML and STS risk models for operative mortality and major morbidity. The major finding was that in cases of discordant prediction, calibration was less reliable, and discriminatory capability as measured by concordance index was drastically reduced, as compared with cases of concordant prediction. In addition, distinct patterns were identified regarding variables that were reliably predictive of concordance or discordance in the majority of outcomes studied. Further investigation into methods of improving risk prediction in these subsets of patients for whom existing models are vulnerable appears prudent. These data highlight a potentially novel avenue to evaluate and refine risk modeling strategy in large clinical registries that carry profound implications in fields such as cardiac surgery.

Sources of Funding

None.

Disclosures

Dr Kilic is on the Medical Advisory Board of Medtronic, Inc. The remaining authors have no disclosures to report.

Tables S1–S12

Click here for additional data file.