Applicability of the commonly used risk scores for coronary bypass surgery in Algeria.

Objectives: The applicability of European system for cardiac operative risk evaluation II (EuroSCORE II) and the Society of Thoracic Surgeons Predicted Risk Of Mortality (STS-PROM) as well as the initial logistic Parsonnet risk score, who have been developed from European and American datasets, is questionable outside these regions. We aimed to assess the performance of these three risk scores for patients undergoing isolated coronary artery bypass grafting (CABG) in Algeria.
Methods: Between June 2014 and June 2016, data from 235 consecutive patients, who underwent isolated CABG at a reference center in Algiers, were prospectively collected and scored according to the EuroSCORE II, STS-PROM and the Parsonnet score. Their discriminative power was evaluated by the area under the receiver operating characteristic curve (AUC) while their calibration was tested by the Hosmer-Lemeshow goodness-of-fit test.
Results: The mean patient age was 59.08 years and 18.3% were female. The mortality at 30 days was 3.40%. The mortality expected by EuroSCORE II, STS-PROM and by Parsonnet risk score was: 1.33%, 0.78% and 3.35%, respectively. Discriminatory ability was fair for the Parsonnet risk score, good for the STS PROM and excellent for EuroSCORE II (AUCs = 0.737, 0.788, and 0.892, respectively). Regarding calibration, EuroSCORE II and STS-PROM under estimated observed mortality (Hosmer-Lemeshow test: P< 0.001 for both scores), while the Parsonnet risk score was well calibrated (Hosmer-Lemeshow test: p = 0.395). Conclusions: EuroSCORE II and STS-PROM had excellent and good discriminating power, respectively, but both underestimated the risk of 30 days mortality following isolated CABG at a reference center in Algiers. The Parsonnet risk score was well calibrated but was moderately discriminating. The development of a local risk score or the recalibration of recent international risk scores is necessary.

1. Introduction

Accurate scoring models for predicting the risk of operative mortality and morbidity are essential for surgical decision making, informed patient consent and healthcare management. When validated in a local population, these risk scores can be used as a benchmark for the assessment of the quality of cardiac surgical services [1]. Although all these score systems are based on patient derived data such as age, gender and co-morbidities, they differ by their design and by the number of clinical and biological parameters used, which necessarily results in a difference in their estimation and in their validation. Selection of appropriate score systems among those commonly used, or the development of a loco-regional risk score, has become an absolute necessity.

Currently, more than 20 models of risk stratification have been developed to predict short-term morbidity and mortality after cardiac surgery. The pioneer in this field was the Parsonnet risk score, which was described in 1989 [2]. This useful score has been rapidly adopted by several cardiac surgery teams, and other authors have confirmed its predictive value on hospital mortality and morbidity [3]. The modified 2000 Bernstein-Parsonnet algorithm, which is a simplified model [4], showed a good correlation between predicted and observed mortality in patients undergoing CABG (on and off pump).

The European System for Cardiac Operative Risk Evaluation (EuroSCORE), in its initial additive version [5] was developed between 1995 and 1999, and has gained wide acceptance in Europe, North America and Asia. This additive model of Euro-SCORE was followed by a logistic version which was developed in 2003 to improve its tendency to underestimate the risk in high-risk patients [6]. After losing its calibration, this score was updated in 2011 and was called EuroSCORE II, which is currently one of the most commonly used scores in the world [1].

In 2008, the American Society of Thoracic Surgeons developed its own logistic model, which has been called the Society of Thoracic Surgeons Predicted Risk Of Mortality (STS-PROM) [7-9]. This risk stratification model is widely used in North America and received a stronger recommendation than that of EuroSCORE II, by the 2018 European Society of Cardiology and the European Association of Cardio-Thoracic Surgery guidelines on myocardial revascularization, to calculate the risk of mortality following CABG surgery [10].

The need to update regularly these risk scoring systems, within the populations on which they were originated, require external validation for other populations, before certifying their applicability, predictability, and accuracy beyond the European and American boundaries [11].

Given the absence of a loco-regional score, we aimed to assess the performance of the Euroscore II; the STS-PROM, and the Parsonnet risk score, for patients undergoing isolated CABG in Algeria.

2. Methods

2.1. Type of the study

This is a monocentric, prospective and observational study. After obtaining approval from the Ethics Committee, we prospectively enrolled, between June 2014 and June 2016, 235 consecutive patients undergoing isolated CABG at a reference center in Algiers. Data acquisition was done from patient records, and the calculation of EuroSCORE II, STS-PROM and the initial logistic Parsonnet risk score has been done on their respective website (Parsonnet risk score: https://sfar.org/scores/parsonnet.php, EuroSCOREII: http://www.euroscore.org/calc.html, STS-PROM: https://riskcalc.sts.org/stswebriskcalc/calculate. No patient was excluded from analysis due to missing data. The outcome of interest was the operative mortality which includes all in-hospital deaths and deaths occurring within 30 days of the procedure.

All procedures performed in this study involving human participants were in accordance with the ethical standards of the Faculty of Medicine of Algiers and with the 1964 Helsinki declaration and its later amendments.

The study was approved in 2014 by the institutional review board of the Faculty of Medicine of Algiers.

Written and informed consent for coronary artery bypass surgery was obtained from all participants.

2.2. Inclusion and exclusion criteria

We included all consecutive patients who underwent isolated CABG (with or without the use of cardiopulmonary bypass) between June 2014 and June 2016.

Exclusion criteria: Association to another cardiac surgical procedure.

2.3. Statistical analysis

Statistical analyses were performed with SPSS version 19.0 (SPSS Inc., Chicago, Illinois). Continuous variables were expressed as mean and standard deviation, and categorical variables were expressed as percentages. A P value of less than 0.05 was considered significant.

The risk models performances were evaluated in terms of discrimination and calibration.

The discriminating power of these three risk scores (statistical accuracy) was tested by calculating the areas under the receiver operating characteristic curve (AUC), which assessed how well the model could discriminate between survivors and non-survivors. The AUC value ranges from 0.5 (no ability to discriminate) to 1.0 (full ability to discriminate). The discriminating power of the model is considered reasonable when the AUC is greater than 0.7 and strong when the AUC exceeds 0.8. AUCs were compared using the method proposed by DeLong’s test.

Calibration (statistical precision) represents the agreement between observed and predicted outcome. It was tested, for these risk scores, by the Hosmer–Lemeshow goodness-of-fit statistics, which performs a χ2 test comparing the observed and expected data, which have been previously divided into deciles of risk. A well-calibrated model gives a P value greater than 0.05.

3. Results

All 235 patients undergoing isolated CABG, between June 2014 and June 2016 were prospectively studied. The mean patients age was 59.1 ± 9.6 (37–82), and 18.3% were female. In this cohort, 68.1% of patients had hypertension, 61.7% had diabetes mellitus, 23.4% presented extracardiac arteriopathy, and 25.5% have had a myocardial infarction within the past 03 months. A left ventricular ejection fraction (LVEF)<50% was noted in 29.4% (n = 69) of patients. Most of the patients (60%, n = 141) had three-vessel diseases and 21.3% (n = 50) of them had left main disease. Bilateral internal thoracic artery grafting was performed in 59.6% (n = 140) of patients and the mean of the distal anastomoses was 2.4 ± 0.8/patient (1–5) (Table 1).

Table 1

Clinical and operative baseline characteristics of CABG patients.

Risk factors	Patients (N = 235)
Male sex. %(N)	81.7 (192)
Age (Year).	59.1 ± 9.6 (37–82)
Hypertension. % (N)	68.1 (160)
Diabetes mellitus % (N)	61.7 (145)
Previous smoker. %(N)	59.1 (139)
Hyperlipidemia. %(N)	57.9 (136)
BMI ≥30 kg/m2	27.7 ± 3.8 (19–38)
Previous Myocardial Infarction. %(N)	75.3 (177)
Myocardial Infarction <3 months. %(N)	25.5 (60)
Previous stroke. %(N)	5.1 (12)
Extracardiac arteriopathy %(N)	23.4 (55)
Chronic lung disease % (N)	4.7 (11)
Creatinine clearance ml/mn	93.8 ± 28.1 (33–190)
Stable angina %(N)	83.8 (197)
Unstable angina %(N)	16.2 (38)
NYHA class I %(N)	43 (101)
NYHA class II %(N)	53.6 (126)
NYHA class III %(N)	3.4 (8)
LVEF %(N)	55.4 ± 12 (24–80)
Pulmonary artery systolic pressure (mmHg)	26.5 ± 7.3 (15–60)
Left main disease %(N)	21.3 (50)
Three-vessel disease %(N)	60 (141)
Number of distal anastomoses	2.4 ± 0.8 (1–5)
BITA grafting %(N)	59.6 (140)
CPB time (minute)	84 ± 33.6 (25–166)
Aortic cross-clamp time (minute)	63.6 ± 24.7 (15–110)

Abbreviations: BITA: Bilateral internal thoracic artery. BMI: Body-Mass Index, CPB: Cardiopulmonary bypass, LVEF: Left Ventricle Ejection Fraction, NYHA: New York Heart Association.

3.1. Overall mortality

There were 8 deaths observed, giving an overall observed mortality at 30 days of 3.40%. Predicted mortality rate for the EuroSCORE II, STS-PROM, and Parsonnet risk score were 1.33% +/− 0.95%, 0.78% +/ 0.96%, and 3.35% +/ 2.44%, respectively (Fig. 1).

Fig. 1

Distribution of mortality rates estimated by EuroSCORE II, STS PROM and Parsonnet score. The red line indicates the average observed mortality.

3.2. Discrimination

EuroSCORE II achieved a strong discrimination with an AUC = 0.893 (95% Confidence Interval: 0.798–0.987), followed by STS-PROM (AUC = 0.788. 95% Confidence Interval: 0.617–0.959) then by Parsonnet risk score (AUC = 0.737. 95% Confidence Interval: 0.520–0.953), whose discrimination was fair (Fig. 2).

Fig. 2

Receiver operating characteristics (ROC) curves of the three scoring models for patients undergoing CABG in Algiers. The area under the curve ROC (AUC) of EuroSCORE II was 0.893 (95% CI: 0.798–0.987), of STS-PROM was 0.788 (95% CI: 0.617–0.959), and of -Parsonnet score was 0.737 (95% CI: 0.520–0.953).

The comparison between the discriminating powers, using the De Long’s test for two receiver operating characteristic curves, showed that the EuroSCORE II outperformed significantly the STS-PROM (0.893 versus 0.788. 95% Confidence Interval: 0.004–0.204, P = 0.041).

3.3. Calibration

EuroSCORE II and STS-PROM achieved bad calibrations and underestimated the observed mortality (Hosmer–Lemeshow statistic was: 37.96, p<0.0001 and 72.14, p< 0.0001, respectively), while the Parsonnet risk score was well calibrated (Hosmer–Lemeshow statistic: 8.40, p = 0.395). The observed/expected ratio of operative mortality was 2.55, 4.36, and 1.01, respectively.

The details of the three risk scores performances are shown in Table 2.

Table 2

Predictive performances of the three risk scores for patients undergoing isolated CABG in Algeria.

	Observed mortality	Predicted mortality	AUC (95% Confidence interval)	Hosmer–Lemeshow test (P value)	Observed mortality/Predicted mortality
EuroSCORE II	3.40%	1.33% ± 0.95%	0.893 (0.798–0.987)	p<0.0001	2.55
STS-PROM	3.40%	0.78% ± 0.96%	0.788. (0.617–0.959)	p<0.0001	4.36
Parsonnet risk score	3.40%	3.35% ± 2.44%	0.737. (0.520–0.953)	P = 0.395	1.01

Abbreviations: AUC: area under the receiver operator characteristics curve; CI: confidence interval.

Calibration curve plots (Fig. 3) showed that all scores had a relatively linear relationship between predicted and observed mortality. While the Parsonnet risk score had good calibration regardless of the patient’s risk profile, the STS-PROM underestimated mortality for all patient’s risk profiles, and the EuroSCORE II only underestimated it for patients whose expected mortality was greater than 2%.

Fig. 3

Calibration plots for the three risk scoring models. The diagonal line represents the perfect calibration line. (Blue) for Parsonnet score. (Red) for EuroSCORE II. (Green) for STS-PROM.

4. Discussion

The main conclusions of this prospective monocentric study, concerning patients undergoing isolated CABG at a reference center in Algiers, are:

EuroSCORE II and STS-PROM, which are the most widely used risk scoring systems worldwide, had excellent and good discrimination, respectively, but both underestimated the risk of 30 days mortality.

The initial logistic Parsonnet risk score had fair discrimination but perfect calibration.

The excellent and good discriminating power of EuroSCORE II and STS PROM, compared to the fair accuracy of Parsonnet risk score, means that the design of these two models and the greater number of variables they contain make them more accurate in identifying patients at high, intermediate or low risk of mortality. However, the lack of precision of these two recent risk scores, reflecting the current and efficient European and united states health systems, in estimating the mortality of Algerian CABG-patients; is due to the higher than expected mortality rate in our cohort which was closer to that expected by Parsonnet risk score.

A calibration failure can be explained by differences in the patients risk profile, but also by a deficiency in the quality of care, which depends closely on socio-economic conditions, surgical technique and quality of postoperative care, or by a poor design of the predictive model.

Compared with EuroSCORE II database patients, our cohort included many more patients with mellitus diabetes (+36.7%), with an average Glycosylated hemoglobin of 8.2 ± 1.7%. This may partly explain the fact that our patients, of which 25.5% had MI within the last 90 days, required coronary bypass surgery at a younger age (−5.5 years).

The stability and quality of isolated CABG outcomes clearly depends on both the annual volume of the center and that of the surgeons. Like other Algerian public cardiac surgery departments (n = 7), our center has a relatively low experience in CABG surgery, with less than 120 patients/year. This is partly explained by the distribution of patients through a larger number of private hospitals (n = 17), but also by the fact that our population is still experiencing an epidemiological transition recording more and more coronary artery diseases without reaching yet the high western incidence of this disease.

Thus, the distribution of patients across many centers will not allow any of them to be a high volume center with a high degree of expertise to reach the western level of performance. This may be one of the reasons limiting the validation of these recent international risk scores in developing countries, where the majority of centers have a low volume of procedures.

Our patients were operated on by 4 surgeons, working in pairs on all procedures. Our surgical team has favored the bilateral internal thoracic artery (BITA) grafting in its different configurations, insitu-BITA, BITA-Y composite grafts, as well as the different types of sequential anastomoses performed in the classic way or in diamond. BITA grafting was performed in 59.6% of patients while total arterial grafting was performed in 35.3% of patients. At the beginning of the study period, 02 surgeons were already experienced and had already performed more than 100 CABG while the others had performed less than 50 CABG. Of note, epiaortic ultrasound, transit-time flow measurement of coronary artery bypass grafts and the various mechanical circulatory supports, which can significantly reduce CABG morbidity and mortality, are still not available in our center or in any other cardiac surgery center in across the country. The lack of modern equipments added to the lurning curve of certain surgeons certainly influenced the operative mortality of our study.

External validation of EuroSCORE II and STS-PROM for patients with isolated CABG has been performed on many ethnically different populations, whose lifestyles and eating habits tend to mimic the western model, such in China [12] and Malaysia [13]. However, in developing countries, these risk scores, which were established on European and North American populations, seem inappropriate for estimating operative mortality following CABG. This is due not only to ethnic and epidemiological factors, but also to differences in the quality of control of cardiovascular diseases risk factors and in the quality of intra-hospital care, which necessarily lead to different outcomes.

Thus in a Brazilian retrospective cohort study of 1,065 patients, 2000 Bernstein-Parsonnet score and EuroSCORE I showed good discriminatory power, but underestimated the in-hospital mortality. This result was in contradiction with what has been observed, during the same period, when applying these two scores in developed countries [14]. Furthermore, in a more recent cohort of 5,222 Brazilian patients, including 60% of patients who benefited from a CABG, the regional score outperformed the STS-PROM and the EuroSCORE II, which also greatly underestimated the observed mortality [15]. The calibration curves of the Euro-SCORE II and STS-PROM of this Brazilian cohort are in perfect adequacy with those generated by our study.

The same conclusion was made for EuroSCORE II in India [16], and Turkey [17]. Even the logistics EuroSCORE, who is known for its overestimation of the risk in European populations, was not applicable in an Iranian study including 2220 CABG procedures [18].

Concerning the predictive performances over time of these scoring systems, it has been noted that Parsonnet risk score, STS-PROM, and Euro-SCORE have experienced a progressive loss of their calibration, even in the populations where they have been established. Indeed, these models overestimated the risk as the results of cardiac surgery have substantially improved with a sustained reduction of risk-adjusted mortality [1]. In addition, EuroSCORE II has been criticized since its publication. Its innate tendency to underestimate observed mortality was considered “acceptable” by authors of the original publication, although its calibration was controversial, since the value of the goodness-of-fit test was near the failure in the original internal validation (p = 0.0505) [1]. This has led to validation concerns of Euro-SCORE II in countries that participated in its development, such as Canada [19] and the United Kingdom [20] where this model was poorly calibrated and failed the Hosmer–Lemeshow test in cohorts of isolated CABG patients (P = 0.002, P < 0.001, respectively).

Since international scores are not applicable in all countries, some national scores have emerged, and were validated on their respective populations, such as the Chinese [12], or the Australian System for Cardiac Operative Risk Evaluation [21] or the São Paulo risk Score for Brazil [15].

5. Limitations

The major limitations of this study are the small sample size and the single-centre design, which may affect the generalizability of results to the entire country. Our relatively low annual volume of CABG procedures may have affected the prediction precision of EuroSCORE II and STS-PROM which also showed good accuracy. A multicentre study conducted over a longer period will make it possible to collect a larger sample size of patients in order to improve the power of the study and to validate or not the applicability of these universal tools of risk scoring at the country scale.

6. Conclusions

EuroSCORE II and STS-PROM had good discriminating power, but both underestimated the risk of 30 days mortality following isolated CABG at a reference center in Algiers. The Parsonnet risk score was well calibrated but was moderately discriminating. The development of a local risk score or the recalibration of recent international risk scores is necessary to better assess the risk.