Performance of InsCor and three international scores in cardiac surgery at Santa Casa de Marília.

OBJECTIVE: To apply and to compare the Society of Thoracic Surgery score (STS), EuroSCORE (Eurosc1), EuroSCORE II (Eurosc2) and InsCor (IS) for predicting mortality in patients undergoing to coronary artery bypass graft and/or valve surgery at the Santa Casa Marilia.
METHODS: The present study is a cohort. It is a prospective, observational, analytical and unicentric. We analyzed 562 consecutive patients coronary artery bypass graft and/or valve surgery, between April 2011 and June 2013 at the Santa Casa Marilia. Mortality was calculated for each patient through the scores STS, Eurosc1, Eurosc2 and IS. The calibration was calculated using the Hosmer Lemeshow test and discrimination by ROC curve.
RESULTS: The hospital mortality was 4,6%. The calibration is generally adequate group P=0.345, P=0.765, P=0.272 and P=0.062 for STS, Eurosc1, Eurosc2, and IS respectively. The discriminatory power of STS score 0.649 (95% CI 0.529 to 0.770, P=0.012), Eurosc1 0.706 (95% CI 0.589 to 0.823, P ≤0.001), Eurosc2 was 0.704 (95% CI 0.590-0.818 P=0.001) and InsCor 0.739 (95% CI 0.638 to 0.839, P ≤0.001).
CONCLUSION: We can say that overall, the InsCor was the best model, mainly in the discrimination of the sample. The InsCor showed good accuracy, in addition to being effective and easy to apply, especially by using a smaller number of variables compared to the other models.

INTRODUCTION

The search for quality excellence in the delivery of health services requires, in addition to technical and structural capacity, a broad knowledge of patients regarding the severity and complexity of their disease in order to identify the factors responsible for the results and long-term prognosis. Among medical specialties, cardiovascular surgery is one of the most complex. Its results are dependent of pre-, intra- and post-operative variables and are assessed by risk scores already developed that estimate the morbidity and mortality of surgical procedure.

The use of performance indicators has emerged as an objective way to measure the quality of health services [ . Risk stratification serves to inform patients and healthcare professionals about the potential risk of complications or death for the group of individuals with similar risk profile who had undergone the procedure proposed [ . The creation and validation of local models become increasingly necessary for demographic, socio-economic and cultural differences found in our environment and the need for external validation of existing prognostic models is of paramount importance for their applicability in clinical practice situations [ . For this, every proposed method should be calibrated and discriminated, or that is, presenting good accuracy and distinguish patients at low and high risk, respectively [ . The importance of external validation of the risk model is not only limited in clinical application. It also serves to recognize the limitations and structural deficiencies of different medical institutions and strategies aiming to qualify these medical services.

Among the most commonly used risk scores are the Society of Thoracic Surgeons score (STS score), The European system for cardiac operative risk evaluation (EuroSCORE) and the EuroSCORE II. Recently, Mejía et al. [ , created and validated internally at the Heart Institute of the Clinics Hospital, Faculty of Medicine, University of São Paulo (Incor/HC-USP) a risk model, called InsCor, having meant to be a tool of easy implementation and good accuracy for prognostic analysis of patients undergoing valve replacement with or without CABG in our country. However this model has not been evaluated in any institution outside the place of origin.

The aim of the study was to apply the InsCor in patients undergoing CABG and/or valve surgery at Santa Casa de Marília, a reference center for public patients in the state of São Paulo and compare its performance with the STS, EuroSCORE (Eurosc1) and EuroSCORE II (Eurosc2).

METHODS

The present study is a cohort. It is a prospective, observational, analytical, single-center study and performed at the Center for Cardiac Surgery in Marília. Between April 2011 and June 2013, patients older than 18 years undergoing coronary artery bypass grafting, valve surgery, and the association of the two procedures, including reoperations were included consecutively, totaling 562 patients. The exclusion criterion was patients under 18 years of age and surgical indication for any other reason. Of the 562 patients, 26 patients (4.63%) died. 368 (65.5%) CABG, 160 (28.5%) valve surgery and 34 (6%) associated coronary and valve were performed.

Data were collected from an institutional database stored in an Excel software spreadsheet that already includes all variables of STS score, logistic EuroSCORE, EuroSCORE II and InsCor respecting their definitions (Chart 1 ), which allowed the calculation of the EuroSCORE II and InsCor after their publications. The calculations of the STS score, Eurosc1 and Eurosc2 were performed by own application from a smartphone and in the case of InsCor from a graph with its own scoring system. The follow-up was limited to hospital phase, with the primary outcome of hospital mortality that included the period between surgery and discharge. The estimated mortality was calculated from the scores STS score, Eurosc1 and Eurosc2 and IS.

Chart 1

Description of variables used in the STS score, EuroSCORE, EuroSCORE II and InsCor

STS score	Surgical intervention, age, gender, ethnicity, peripheral disease, cerebrovascular disease and stroke, weight, height, diabetes mellitus, creatinine, dialysis, hypertension, endocarditis, pulmonary disease, immunosuppression, previous cardiac surgery or angioplasty, use of preoperative inotropic, cardiogenic shock, cardiac resuscitation, cardiac arrhythmia, myocardial infarction, coronary symptoms, NYHA functional class, number of coronary arteries involved, LMCD lesion, LV ejection fraction, associated valvular heart disease, number of previous surgeries, procedure status (elective, urgent, emergent), use of intra-aortic balloon.
Euroscl	Age, gender, COPD, peripheral vascular disease, neurological dysfunction, creatinine, endocarditis, previous cardiac surgery, preoperative critical state (use of preoperative inotropic support, cardiogenic shock, cardiac resuscitation, use of intra-aortic balloon), unstable angina , recent myocardial infarction, left ventricular ejection fraction, pulmonary hypertension (PSAP greater than 60 mmHg), emergency, associated surgery, aortic surgery, post-infarction VSD.
Eurosc2	Age, gender, creatinine clearance, peripheral vascular disease, poor mobility, endocarditis, previous cardiac surgery, COPD, critical condition (use of preoperative inotropic, cardiogenic shock, cardiac resuscitation, use of intra-aortic balloon pump), insulin diabetes dependent, NYHA functional class, CCS angina classification, previous myocardial infarction, left ventricular ejection fraction, degree of pulmonary hypertension, procedure status (elective, urgent, emergent), associated surgery, aortic surgery.
InsCor	Age over 70 years, female gender, associated surgery (coronary + valve), recent infarction, reoperation, aortic valve surgery, tricuspid valve surgery, creatinine greater than 2 mg/dl, LV ejection fraction less than 30%, preoperative events (use of preoperative inotropic support, cardiogenic shock, cardiac resuscitation, use of intra-aortic balloon, acute renal failure, cardiac massage, tracheal intubation, tachycardia or ventricular fibrillation).

STS=Society of Thoracic Surgeons; Eurosc1=EuroSCORE; Eurosc2=EuroSCORE II; NYHA=New York Heart Association; LMCD=Left Main Coronary Artery Disease; COPD=Chronic Obstructive Pulmonary Disease; PSAP=Pulmonary Artery Systolic Pressure; LV=Left ventricle; CCS=Canadian Cardiovascular Society; VSD=Ventricular Septal Defect

The calibration was calculated using the Hosmer-Lemeshow test and the P value>0.05 indicates that the model fits the data and predicts mortality appropriately. Discrimination distinguishes patients at low and high risk, and is measured by the area under the ROC curve (Receiver Operating Characteristics). Statistical analysis was performed using SPSS software version 16.0 for Windows (IBM Corporation Armonk, New York). The performance of models was measured by comparing observed and expected mortality in risk groups established by models. The Fisher exact test was used for contingency tables. The P value <0.05 was considered significant. This study was approved by the Research Ethics Committee of the Faculty of Medicine of Marília under number 767 329 on August 27, 2014.

RESULTS

Performance results of the STS score, EuroSCORE, EuroSCORE II and InsCor

Calibration Results

STS score

In the analysis of the whole group, association was observed between the STS score and death (P <0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic=8.9696 with 8 degrees of freedom and P =0.3449. For coronary surgeries, we observe that the STS score was not associated with death (P =0.182) but showed good calibration (P =0.210). For valve surgery, we found that it is associated with death (P =0.009) with good calibration (P =0.460). Associated surgeries (coronary and valve), was not associated with death (P =0.4078), but showed good calibration (P =0.2648).

EuroSCORE

In the analysis of the whole group, association was observed between Eurosc1 and death (P <0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 4.9246 with 8 degrees of freedom and P =0.7656. For coronary procedures, we found that the EuroSCORE is associated with death (P =0.021) with good calibration (P =0.529). For valve surgery, we found that the model is associated with death (P =0.004) in addition to good calibration (P =0.893). For associated surgeries (coronary and valve), although the Eurosc1 was not associated with death (P =0.1033), it showed good calibration (P =0.2911).

EuroSCORE II

In the analysis of the whole group, association was observed between Eurosc2 and death (P <0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 9.8963 with 8 degrees of freedom andP =0.2724. For coronary procedures, we observed that Eurosc2 is associated with death (P =0.040) and demonstrates good calibration (P =0.250). For valve surgery, we found that the model is associated with death (P <0.001) and demonstrates good calibration (P =0.423). For associated surgeries (coronary and valve), although the Eurosc2 was not associated with death (P =0.5159), it showed good calibration (P =0.5659).

InsCor

In the analysis of the whole group, association was observed between InsCor and death (P <0.001). The Hosmer-Lemeshow test showed a goodness-of-fit statistic = 8.9533 with 4 degrees of freedom and P =0.0623. For coronary procedures, we observe that although the InsCor was not associated with death (P =0.059), it showed good calibration (P =0.110). For valve surgery, we observed that InsCor is associated with death (P =0.002) and good calibration (P =0.743). For associated surgeries (coronary and valve), the InsCor was associated with death (P =0.022) but the calibration was not adequate (P≤0.001).

ROC curve for the Eurosc1, Eurosc2, STS-score and InsCor. EuroSc1 ROC=ROC curve presented by EuroSCORE score; EuroSc2 ROC=ROC curve presented by EuroSCORE II score; STS ROC=ROC curve presented by STS-score score; InsCor ROC=ROC curve presented by InsCor score

Discrimination results (Figure 1 )

According to the discriminative power of STSscore, we observe that the area under the ROC curve was 0.649 (95% CI 0.529-0.770, P =0.012) (Table 1 ).

Table 1

Area under the ROC curve for the STS-score, Eurosc1, Eurosc2 and InsCor

	Area	CI 95%	se	P
STS-score	0.649	0.529 - 0.770	0.061	0.012
ES	0.706	0.589 - 0.823	0.060	< 0.001
ES II	0.704	0.590 - 0.818	0.058	0.001
InsCor	0.739	0.638 - 0.839	0.051	< 0.001

ROC=Receiver Operating Characteristic, STS-score=Society of Thoracic Surgeons score, Eurosc1=EuroSCORE, Eurosc2=EuroSCORE II CI=Confidence interval; se=Standard error; P=P value

According to the discriminative power of Eurosc1, we observe that the area under the ROC curve was 0.706 (95% CI 0.589-0.823, P≤0.001) (Table 1 ).

According to the discriminative power of Eurosc2, we observe that the area under the ROC curve was 0.704 (95% CI 0.590-0.818, P =0.001) (Table 1 ).

According to the discriminative power of InsCor, we observe that the area under the ROC curve was 0.739 (95% CI 0.638 to 0.839, P≤ 0.001) (Table 1 ).

DISCUSSION

The present study is the first assessing the performance of InsCor outside the institution that developed it. We apply and compare the most current and popular scores worldwide and found little difference when comparing the EuroSCORE with EuroSCORE II, a result similar to other international studies.

The good current medical practice requires doctors and hospitals proposing to perform cardiovascular surgery to assess their patients adequately and in a responsible manner. In this preoperative evaluation, calculation and estimation of complications or death are required and able to propose strategies in the search for better results.

The prognostic models for risk analysis are inefficient if they serve only to the population where they were developed, thus they must have good performance in other populations, or that is, they must be validated externally [ .

The EuroSCORE system that for nearly 15 years provides an estimate of postoperative mortality in cardiac surgery is widely used to have good accuracy and adequate discriminating power in different countries [ . In our midst, Andrade et al. [ analyzed 804 patients undergoing valve surgery at the Heart Institute of Pernambuco and obtained a good fit with an area under the ROC curve of 0.731 (95% CI 0.660 to 0.793 P <0.001). This same effectiveness has been demonstrated by other authors who studied patients undergoing CABG and found good accuracy, estimated at 69.9% and 89.2% [ . On the other hand, Carvalho et al. analyzed 546 patients undergoing CABG and observed underestimation of lethality expected, with notable differences between the predicted and observed with an area under the ROC curve estimated at 0.62 advising against the use of the model in our midst. This fact is explained by differences in prevalence among risk factors of the model and the differences found in the population studied. An adjustment to the weight of the factors that composes such model could correct its performance and make it an applicable tool [ . In our study, we found that the EuroSCORE was associated with death, with good calibration for CABG and heart valve surgeries and proper ROC curve of 0.706 (95% CI 0.589 to 0.823, P ≤0.001). According to the analysis of discrimination by ROC curve we have a poor discrimination with ROC<0.70, acceptable with ROC<0.75, good with ROC<0.80 and very good with ROC≥0.80.

On October 2011, Nashef et al. showed the remodeled EuroSCORE that became known as the EuroSCORE II, only logistical and started using new variables as: calculation of creatinine clearance, insulin dependent diabetes, NYHA class and CCS class 4, in addition to reclassification of the following variables: ejection fraction, pulmonary hypertension, urgency of the procedure and type of procedure performed. In calibration, the observed mortality was 3.9% and the expected mortality by EuroSCORE II of 3.77%, compared to 4.6% of the original EuroSCORE [ . Barili et al. [validated the EuroSCORE II from the retrospective analysis of the results of a database of three institutions containing 12,325 consecutive patients and reaching a hospital mortality rate of 2.2% and a high discriminative power with an area under the ROC of 0.82 (95% CI: 0.80-0.85) and good calibration curve to a mortality of 30% predicted. However, it did not appear to significantly improve the performance of older versions in the highest tertiles of risk. Paparella et al. [ , studied data on 6191 patients and found hospital mortality rate of 4.85% and EuroSCORE II of 4.40±7.04%. The area under the ROC curve of 0.83 showed good discriminative ability. In the analysis of calibration, there was an underestimation of the expected mortality in high-risk patients [ . In South America, Borracci et al. [ , performed the validation of the EuroSCORE II in 503 patients undergoing cardiac surgery and obtained a hospital mortality rate of 4.17% with predicted mortality of 3.18% (P =0.402). The area under the ROC curve was 0.85 (P =0.0001) and the model showed good calibration in predicting in-hospital mortality (Hosmer-Lemeshow, P =0.082). The EuroSCORE II showed a good overall discriminative ability and calibration in this population, however, the performed model underestimates in-hospital mortality of patients with lower risk [ . In Brazil, the first validation of the EuroSCORE II was performed by Lisboa et al. [ , from the analysis of 1000 consecutive patients undergoing coronary and/or valve surgery.

In calibration, the Hosmer-Lemeshow test was inadequate (P =0.0003). However, in the discrimination, the area under the ROC curve was 0.81 [95% CI (0.76-0.85),P <0.001], concluding that the EuroSCORE II became more complex and similar to the international literature with respect to poor calibration to predict mortality in patients undergoing coronary and/or valve in our midst, reinforcing the importance of a local model [ . In the present study, we found that the EuroSCORE II was associated with death, showing good calibration in CABG and valve surgery performed with proper ROC curve area of 0.704 (95% CI, 0.590 to 0.818, P =0.001).

The model of surgical risk created by the American Society of Thoracic Surgeons (Society of Thoracic Surgeons-STS) allows the calculation of mortality and morbidity, including rates of complications such as the risk of prolonged hospitalization, stroke, prolonged intubation, renal failure, deep wound infection and reoperation in patients undergoing CABG, valve (replacement or repair) and the association between the two types of surgery [ . In our environment, Ikeoka et al. evaluated the STS score and achieved good calibration and discrimination for mortality and morbidity with an area under the ROC curve of 0.76 and 0.75 (P≤0.001) respectively in a group of patients undergoing CABG [ . In study similar to ours, Wang et al. [ , Australia, analyzed four models of risk (EuroSCORE, EuroSCORE II, STS score, and a local score called AusScore) for patients undergoing CABG. The estimated 30-day mortality by EuroSCORE was 2.8% (1.6 to 5.2%), EuroSCORE II 1.6% (from 1.0 to 2.8%), STS Score of 2.3% (1.3 - 4.5%) and the AusScore 0.5% (0.2 to 1.1%). Regarding the EuroSCORE, the EuroSCORE II, the STS score and the AusScore showed a slight improvement in calibration but similar for 30-day mortality discrimination. In our study, although the STS score was not associated with death, it showed good calibration in division by groups and the ROC curve of 0.649 was poor (95% CI 0.529 to 0.770, P =0.012).

In our study, we found overall hospital mortality of 4.6%. Regarding mortality and expected by the scores observed in the three groups the analysis shows an underestimation of mortality estimated by three international scores, especially Eurosc1 and Eurosc2. When comparing the study by Wang et al. [ , we observed a wide variation in the risk of patients seen in our service, which could be explained by delayed access to health care and/or late surgical indication of patients to the surgical procedure, a system of reflexes of poor health. In patients undergoing CABG, mortality was 3.2%, ranging from 1.0% to 4% (0.8%-72%). In valve surgery, mortality was 6.2%, ranging from 1.8% to 9.6% (0.8% -59%). In the combined surgery group, mortality was 11.7%, ranging from 2.8% to 12.5% (from 0.6 to 26%) (Table 2 ).

Table 2

Expected and observed mortality

		Expected mortality (Median)
	Observed mortality	Eurosc1	Eurosc2	STS
Global	4.6%	3.0% (0.8-72)	1.3% (0.5-32)	3.7% (0.2-60)
CABG	3.2%	2.5% (0.8-72)	1.0% (0.5-24)	2.7% (0.2-60)
Valve	6.2%	5.0% (0.8-44)	1.8% (0.5 -32)	9.6% (0.9-59)
CABG + Valve	11.7%	5.8% (1.5-23)	2.8% (0.6-11)	12.5% (1.9-26)

CABG=Coronary artery bypass grafting; STS-score=Society of Thoracic Surgeons score; Eurosc1=EuroSCORE; Eurosc2=EuroSCORE II

In 2013, a model of national risk developed from the analysis of 3000 patients undergoing coronary artery bypass and/or valve at the Heart Institute of the Clinics Hospital, Faculty of Medicine, University of São Paulo, called of InsCor was proposed by Mejia et al. [ , a remodeling product of two models (2000 Bernstein-Parsonnet and EuroSCORE) validated and faced with our reality and has become an important tool for patients treated at InCor- HCFMUSP [ . 10 variables were selected: age >70 years; female; surgical coronary revascularization + valve; myocardial infarction < 90 days; reoperation; surgical treatment of aortic valve; surgical treatment of tricuspid valve; creatinine > 2 mg/dL; ejection fraction < 30%; and events (use of preoperative inotropic support, cardiogenic shock, cardiac resuscitation, use of intra-aortic balloon, acute renal failure, cardiac massage, tracheal intubation, tachycardia or ventricular fibrillation). The Hosmer-Lemeshow test was 0.184, indicating excellent calibration and the area under the ROC curve was 0.79 (95% from 0.74 to 0.83, P <0.001 IC) [ . In our study, the InsCor was not associated with mortality for CABG, presented good calibration for CABG and heart valve surgeries and had adequate ROC of 0.739 (95% CI, 0.638 - 0.839, P≤0,001).

This study assessed and compared three scores of international risk, being an American and two European and one developed in our country. The Brazilian experience in creating risk scores nationwide is completing nearly a decade [ , but without external validation. The search for simple models containing few variables are of wider applicability, since they contain the significant risk factors and have an appropriate degree of prediction because they are easier to be incorporated in patient care [

We can consider limitations of the study the fact that it is a single center and with a small sample. It is an ongoing multicenter study in referral hospitals in the state of São Paulo, aiming at reshaping the InsCor for creation of the SCORE-SP (São Paulo System for Cardiac Operative Risk Evaluation). This project will initiate the São Paulo State Registry of Cardiovascular Surgery, in order to improve the quality of results and the safety of patients undergoing cardiac surgery in the State of São Paulo [ .

CONCLUSION

In conclusion, the InsCor was the best model to predict the final outcome of survival or mortality, followed by EuroSCORE. The EuroSCORE was the best model in adjusted mortality for CABG and InsCor, along with the EuroSCORE II for valve surgery. Therefore, the local model InsCor showed good accuracy, in addition to being effective and easy to apply, mainly by using a smaller number of variables compared to the other models, variables that represent relevant risk factors in our population.

Abbreviations, acronyms & symbols
CCS	Canadian Cardiovascular Society
Eurosc1	The European system for cardiac operative risk
	evaluation
Eurosc2	The European system for cardiac operative risk
	evaluation II
NYHA	New York Heart Association
ROC	Receiver Operating Characteristic
SPSS	Statistical Package for the Social Sciences
SP-SCORE	Sao Paulo System for Cardiac Operative Risk
	Evaluation
STS	Society of Thoracic Surgeons score

Authors’ roles & responsibilities
MGT	Analysis and/or interpretation of data, final approval of the manuscript conception and study design, conduct of operations, and/or experiments, writing of the manuscript or revising it critically for its content
HAB	Performing the procedures and/or experiments
MSS	Performing the procedures and/or experiments
MHB	Performing the procedures and/or experiments
MPBM	Performing the procedures and/or experiments
SMF	Performing the procedures and/or experiments
EAP	Final approval of manuscript
RTB	Final approval of the manuscript, performing the operations and/or experiments, writing of the manuscript or review of its content