Validity of Outcome Prediction Scoring Systems in Korean Patients with Severe Adult Respiratory Distress Syndrome Receiving Extracorporeal Membrane Oxygenation Therapy.

Recently, several prognostic scoring systems for patients with severe acute respiratory distress syndrome (ARDS) requiring extracorporeal membrane oxygenation (ECMO) have been published. The aim of this study was to validate the established scoring systems for outcome prediction in Korean patients. We retrospectively reviewed the data of 50 patients on ECMO therapy in our center from 2012 to 2014. A calculation of outcome prediction scoring tools was performed and the comparison across various models was conducted. In our study, the overall hospital survival was 46% and successful weaning rate was 58%. The Predicting Death for Severe ARDS on V-V ECMO (PRESERVE) score showed good discrimination of mortality prediction for patients on ECMO with AUC of 0.80 (95% CI 0.66-0.90). The respiratory extracorporeal membrane oxygenation survival prediction (RESP) score and simplified acute physiology score (SAPS) II score also showed fair prediction ability with AUC of 0.79 (95% CI 0.65-0.89) and AUC of 0.78 (95% CI 0.64-0.88), respectively. However, the ECMOnet score failed to predict mortality with AUC of 0.51 (95% CI 0.37-0.66). When evaluating the predictive accuracy according to optimal cut-off point of each scoring system, RESP score had a best specificity of 91.3% and 66.7% of sensitivity, respectively. This study supports the clinical usefulness of the prognostic scoring tools for severe ARDS with ECMO therapy when applying to the Korean patients receiving ECMO.

INTRODUCTION

Extracorporeal membrane oxygenation (ECMO) has been increasingly used as a rescue therapy for patients with severe acute respiratory distress syndrome (ARDS) (1). Recent promising studies suggested that transfer to the specialized ECMO center led to better outcomes in patients with severe ARDS (12). However, most studies that supported the efficacy of ECMO are cohort studies restricted to a specific group of patients with H1N1 virus infection (234). Nonetheless, the implementation of ECMO is more frequent (5) and a further rise of use is expected in a broader spectrum of patients. Given the need of highly specialized staff, additional hospital costs and complications related to the device, early identification of appropriate and practical use of this limited resource is considered as an upcoming issue. To select the optimal patients who could benefit most from ECMO, a number of outcome prediction scoring systems has been recently proposed (2345). However, the developments of these systems were based on the subsets of patient population in terms of underlying respiratory disease, such as influenza A (H1N1)-associated ARDS (3) and mode of ECMO (6). Therefore, validation of these scoring systems is strongly needed to help us to identify who will benefit most from a trial of ECMO therapy in ARDS patients with various backgrounds. This study was designed to assess validity of the currently proposed prediction scoring systems in Korean patients with severe acute respiratory failure refractory to conventional therapy.

MATERIALS AND METHODS

Study design

Data were extracted retrospectively from a prospectively updated registry of ECMO patients and ICU clinical database of our center. Patients over the age of 18 years who underwent ECMO support between January 2012 and December 2014 were included. Further clinical features were obtained from retrospective review of patient medical records from routine care. This study was approved by the institutional review of our center. The requirement of informed consent from the patients was waived since the aim of this study was to analyze clinical individual data of patients retrospectively.

ECMO protocol

We have a multidisciplinary team composed of intensivists, pulmonologists and cardio-thoracic surgeons and make the decisions for initial ECMO implant based on the ELSO guideline (7). The general indication was either a ratio of partial arterial pressure of oxygen/fraction of inspired oxygen (PaO2/FiO2) < 70 mmHg with FiO2 > 0.8 or a refractory respiratory acidosis (pH < 7.2), despite optimization of mechanical ventilation therapy (8) (PEEP ≥10 cmH2O, VT 6 mL/kg PBW) and the adjunctive therapy including Nitric Oxide inhalation, prone position and steroid. Contraindications for ECMO therapy were malignancies with fatal prognosis within 5 years, intracranial bleeding, or patients with decisions to limit therapeutic interventions. We followed the details of ECMO protocols according to the ELSO guidelines (9). The mechanical ventilation protocol was based on the ARMA trial published in 2000 by ARDSNET (8). ECMO was primary implemented in veno-venous (VV) ECMO. Veno-arterial (VA) ECMO was performed for the patients as following: when VV ECMO was anticipated not to be sufficient, such as severe heart failure, severe hemodynamic instability or septic shock. Later switching of VVA mode was depended on upper body hypoxemia or aggravated desaturation despite conventional ventilator therapy. The configuration for VV ECMO was femoral vein and internal jugular vein, and femoral artery and femoral vein were used for VA ECMO. The ECMO system consisted of a polymethylpentene fiber oxygenator system (Quadrox PLS; Maquet, Hirrlingen, Germany) with simplified Bioline-coated circuits (Maquet, Rastatt, Germany). All patients were supported with centrifugal pumps (Maquet, Rastatt, Germany). VA and VV ECMO were used in all patients with size-appropriate cannulas (10). Patients received an initial unfractionated heparin (UFH) bolus of 50 units/kg body weight when the cannula was implemented, and UFH was infused continuously during ECMO. Continuous renal replacement was performed via the ECMO circuit.

Data collection

Demographics, co-morbidities, main diagnosis, simplified acute physiology score (SAPS) II, which was calculated by the worst value during the first 24 hours in the ICU and sequential organ failure assessment (SOFA) score before ECMO initiation were recorded. The setting of mechanical ventilator and the results of arterial blood gas analysis (ABGA) were collected at ECMO initiation. In addition, days on ECMO, mechanical ventilation, the requirement for vasopressors, neuromuscular blocking agents, nitric oxide (NO) therapy and prone position were recorded. Established mortality prediction tools including PRESERVE score (2), ECMOnet score (3), RESP score (4), and the score as proposed by Roch et al. (5) were calculated as presented in the original articles and applied for the whole study population. All the patients in our study group received ECMO in our center from the first time.

Statistical analysis

Data were conducted using Kolmogorov-Smirnov and Shapiro-Wilk tests for normal distribution. For continuous variables, either student T test or Mann Whitney U test was performed based on their distribution. For categorical variables, either χ2 test or the Fisher exact test was used to investigate comparisons between survivor and non-survivor groups. All P values were two sided and P values ≤ 0.05 were considered statistically significant. Discrimination of each score was evaluated by receiver operating characteristic (ROC) analysis. In addition, sensitivity and specificity for mortality prediction scoring systems were determined and cutoff value was corresponded to maximum of the Youden’s index. The Kaplan-Meier analysis was used to estimate of survival probabilities over time for cutoff value of SAPS II score and RESP score. All date were analyzed using SPSS of version 21.0 (SPSS Inc., Chicago, IL, USA).

Ethics statement

This study was approved by the institutional review board of Pusan National University Yangsan Hospital (approval number 05-2015-107). The requirement of informed consent from the patients was waived since the aim of this study was to analyze clinical individual data of patients retrospectively.

RESULTS

Pre-ECMO characteristics of study population

This retrospective study was based on a total of 50 patients with severe ARDS receiving ECMO support between January 2012 and December 2014 in our center. Among these patients, 30 patients (60%) underwent VV ECMO and 20 patients (40%) underwent VA ECMO. Of these who received VA ECMO, 9 patients were changed to VVA mode due to the risk of upper body hypoxemia or aggravated desaturation despite conventional ventilator therapy. The rest 11 patients were remained the VA ECMO. The mode of cannulation was not significantly different between survivor and non-survivor group (Table 1). The overall hospital survival was 46% and successful weaning rate was 58%. The survival rate of VV ECMO was 53.3% while the survival rate for VA ECMO was 35%. Patient baseline characteristics are presented in Table 1. The study included 34 males (68%), with a patient median age was 57.5 ± 14.3 years. The mean age of the survival group was lower than non-survivor group (50.2 ± 15.6 vs. 63.7 ± 9.8, P = 0.001). The mean BMI was 23.1 ± 3.9 and the difference between survivor and non-survivor group was not significant (23.5 ± 4.1 vs. 22.7 ± 3.7, P = 0.485). The main cause of ARDS was pneumonia followed by postoperative ARDS and extra-pulmonary sepsis. The etiology of ARDS did not reveal any significant difference between the survivor and non-survivor group. However, there were more immunocompromised patients in the non-survivor group (17.4% vs. 59.3%, P = 0.003). Comorbid conditions as expressed by Charlson score was higher in the non-survivor group (1.13 ± 1.01 vs. 2.44 ± 1.55, P = 0.001). In addition, SAPS II score and SOFA score, severity scores and mortality prediction in ICU patients, were higher in the non-survivor group, respectively (50.35 ± 15.19 vs. 68.70 ± 17.98, P < 0.001, 10.52 ± 3.42 vs. 12.74 ± 3.61, P = 0.001). Pre-ECMO mechanical ventilator settings and requirements of rescue maneuvers did not differ between groups (Table 1). Overall mean PF ratio was 76.6 ± 23.5 mmHg and average pre-ECMO ventilator days was 3.1 ± 4.2. The administration of neuromuscular blocker was for 90% of patients, vasopressor for 80% of patients, steroid for 36% of patients, and bicarbonate was infused in 56%, respectively. Otherwise, nitric oxide inhalation was used in 16% of patients and prone position was used in 6% of patients.

Table 1

Pre-ECMO baseline characteristics of all patients

Parameters	All patients (n = 50)	Survivor (n = 23)	Non-survivor (n = 27)	P value
Age, yr	57.5 ± 14.3	50.2 ± 15.6	63.7 ± 9.8	0.001
Men	34 (68.0)	14 (60.8)	20 (74.0)	0.318
BMI, kg/m2	23.1 ± 3.9	23.5 ± 4.1	22.7 ± 3.7	0.485
ECMO mode (initial mode)
VV (veno-venous)	30 (60.0)	16 (69.6)	14 (51.9)	0.203
VA (veno-arterial)	20 (40.0)	7 (30.4)	13 (48.1)	0.158
ARDS etiology
Pneumonia	26 (52.0)	10 (43.5)	16 (59.3)	0.266
Postoperative	15 (30.0)	7 (30.4)	8 (29.6)	0.951
Extrapulmonary sepsis	8 (16.0)	4 (17.4)	4 (14.8)	0.804
Trauma	3 (6.0)	2 (8.7)	1 (3.7)	0.459
Pre-ECMO condition
Immunocompromised*	20 (40.0)	4 (17.4)	16 (59.3)	0.003
Renal dysfunction	14 (28.0)	7 (30.4)	7 (25.9)	0.723
Heart dysfunction	5 (10.0)	1 (4.3)	4 (14.8)	0.357
CNS dysfunction†	5 (10.0)	0 (0.0)	5 (18.5)	0.054
Charlson comorbidity score	1.84 ± 1.48	1.13 ± 1.01	2.44 ± 1.55	0.001
SAPS II score	60.3 ± 18.99	50.35 ± 15.19	68.70 ± 17.98	0.000
SOFA score	11.72 ± 3.67	10.52 ± 3.42	12.74 ± 3.61	0.031
Pre-ECMO ventilator days	3.1 ± 4.2	2.3 ± 2.4	4.2 ± 5.3	0.114
Ventilator parameter
PF ratio, mmHg	76.6 ± 23.5	79.0 ± 25.6	74.6 ± 21.9	0.517
FiO2,%	89.8 ± 15.7	87.2 ± 17.4	92.0 ± 14.2	0.281
PEEP, cmH2O	9.6 ± 3.2	9.22 ± 3.2	9.96 ± 3.2	0.411
Tidal volume/PBW, mL/kg	6.16 ± 1.9	6.08 ± 2.3	6.23 ± 1.4	0.791
RR, /min	23.5 ± 5.9	22.1 ± 5.6	24.6 ± 6.0	0.143
Peak inspiratory pressure, cmH2O	26.5 ± 4.0	25.3 ± 3.2	27.4 ± 4.5	0.061
Pre-ECMO blood gases
pH	7.29 ± 0.15	7.31 ± 0.16	7.27 ± 0.15	0.476
PaO2, mmHg	66.5 ± 14.7	66.4 ± 15.6	66.6 ± 14.2	0.948
HCO3, mEq/L	21.8 ± 9.2	21.6 ± 6.2	21.9 ± 11.3	0.892
PaCO2, mmHg	45.1 ± 25.4	41.4 ± 16.4	48.4 ± 31.1	0.337
SaO2,%	87.7 ± 10.3	87.5 ± 13.0	87.8 ± 7.6	0.936
Pre-ECMO Rescue therapy
Steroid	18 (36.0)	8 (34.8)	10 (37.0)	0.863
Prone position	3 (6.0)	1 (4.3)	2 (7.4)	1.000
Nitric oxide	8 (16.0)	2 (8.7)	6 (22.2)	0.193
Neuromuscular blockade	45 (90.0)	20 (87.0)	25 (92.6)	0.508
Vasopressor	40 (80.0)	17 (73.9)	23 (85.2)	0.321
Bicarbonate infusion	28 (56.0)	9 (39.1)	19 (70.4)	0.027

Values are expressed as median (mean ± standard deviation) or No. (%).

BMI, body mass index; ECMO, extracorporeal membrane oxygenation; ARDS, acute respiratory distress syndrome; FiO2, fraction of inspired oxygen; HCO3, bicarbonate; MV, mechanical ventilation; PaO2, partial pressure of oxygen; PaCO2, partial pressure of carbon dioxide; PBW, predicted body weight; PEEP, positive end-expiratory pressure; SaO2, oxygen saturation.

*Immunocompromised is defined as hematological malignancies, solid tumor, solid organ transplantation, human immunodeficiency virus, and/or cirrhosis; †Central nervous system dysfunction diagnosis combined neurotrauma, stroke, encephalopathy, cerebral embolism, and seizure and epileptic syndrome.

Performance of pre-existing outcome prediction scoring system in our cohort

The ROC curve (Fig. 1) reveals that PRESERVE score significantly discriminated survivors and non-survivors with an AUC of 0.80 (95% CI 0.66-0.90, P < 0.001). Similarly, RESP score and SAPS II score performed fair discrimination for hospital mortality with an AUC of 0.79 (95% CI 0.65-0.89, P < 0.001) and AUC of 0.78 (95% CI 0.64-0.88, P = 0.001), respectively. The score performed by Roch et al. (5) with a fair prediction ability (AUC 0.70, 95% CI 0.56-0.82, P = 0.013), however, ECMOnet score failed to discriminate between survivors and non-survivors with AUC of 0.518 (95% CI 0.37-0.66, P = 0.830).

Fig. 1

Comparison of the receiver-operating curves for all risk prediction tools (n = 50). The ROC curve reveals that PRESERVE score significantly discriminated survivors and non-survivors with an AUC of 0.80. RESP score and SAPS II score expose fair discrimination with an AUC of 0.79 and 0.78, respectively. The score proposed by Roch et al. (5) also had fair prediction ability, however, ECMOnet score failed to discriminate with AUC of 0.518.

AUC, area under the curve; ROC, receiver operating curve; CI, confidence interval; PRESERVE, predicting death for severe ARDS on VV-ECMO; SAPS, simplified acute physiology score; RESP, respiratory extracorporeal membrane oxygenation survival prediction.

Comparison between each outcome prediction scoring systems

The pairwise comparison of ROC curves revealed no statistically significant difference between PRESERVE score, RESP score, Score proposed by Roch et al. (5) and SAPS II score, respectively (Appendix 1). However, ECMOnet score showed worse prediction over the other scoring systems (vs. PRESERVE score; P = 0.003, vs. RESP score; P = 0.001, vs. SAPS II score; P = 0.009 and vs. the score proposed by Roch et al. (5); P = 0.048, respectively).

Predictive accuracy for hospital mortality according to optimal cut-off point in our cohort

As presented in Table 2, RESP score was found to have the best specificity and positive predictive value of 91.3% and 90.0%, respectively. The cut-off value for mortality in our study group was ≤ -1. At this cut-off point, the results of Kaplan-Meier analysis indicated that RESP score above the cut-off point of -1 showed a significantly higher hospital survival rate (P = 0.001, Fig. 2A). The SAPS II score had good agreement of mortality prediction in our study population. The optimal cut-off value for mortality > 58 showed the best sensitivity of 74.1% and was used to classify patients into high and low risk for hospital mortality. According to SAPS II score below the cut-off value of 58 had a significantly higher hospital survival rate (P = 0.005, Fig. 2B).

Table 2

Prediction accuracy of hospital mortality in pre-existing scoring system

Scores	Cut-off value	Sensitivity, %	Specificity, %	PPV, %	NPV, %
RESP score	≤ -1	66.67	91.30	90.0	70.0
Score proposed by Roch et al.	> 3	55.56	82.61	78.9	61.3
PRESERVE score	> 5	66.67	73.91	75.0	65.4
SAPS II score	> 58	74.07	73.91	76.9	70.8

Data are expressed as No. (%).

RESP, respiratory extracorporeal membrane oxygenation survival prediction; PRESERVE, predicting death for severe ARDS on VV-ECMO; SAPS, simplified acute physiology score.

Fig. 2

Kaplan-Meier analysis for hospital survival probability according to cut-off point. (A) The patient group in RESP score above the cut-off point of -1 was indicated significantly higher hospital survival rate that the patient group those who were below the cut-off point of -1 (P = 0.001). (B) The lower Pre-ECMO SAPS II score group (≤ 58 points) showed a significantly higher hospital survival than the higher Pre-ECMO SAPS II score group (> 58 points) in the study population (P = 0.005).

RESP, respiratory extracorporeal membrane oxygenation survival prediction; SAPS, simplified acute physiology score.

DISCUSSION

This single center study was designed for validating the preexisting mortality prediction model in our respiratory ECMO cohort. This study found that the outcome prediction using PRESERVE score and RESP score fit with the observed short-term outcome in our cohort. Both PRESERVE score and RESP score showed a good predictive power for hospital mortality by ROC curve. Moreover, RESP score showed high specificity (91.3%) and positive predictive value (90.0%), which indicated that RESP score was useful tool to predict who could be beneficial by ECMO therapy. SAPS II score also showed the possibility of an alternative prediction model. Therefore, both PRESERVE and RESP scoring systems and SAPS II score could be useful outcome prediction models in our ARDS cohort requiring ECMO support.

Our data was collected since 2012 and was influenced by the result of recent evidences about ARDS management such as early use of neuromuscular blocker and early application of ECMO (1112). In this study, pre-ECMO ARDS management was consistent in both survivor and non-survivor groups in terms of lower tidal volume, moderate level of PEEP, relatively early application of ECMO (before 7 days of mechanical ventilation) and use of rescue therapy (Table 1). Therefore, outcome prediction in our cohort reflects the patient’s chronic health condition such as pre-ECMO comorbidities, immunosuppression and organ dysfunctions. Despite consistency of ARDS management between survivor and non-survivor, PRESERVE score, RESP score and SAPS II score showed the good prediction of short-term mortality.

Our cohort had several different characteristics from previous data which developed ECMO prediction mortality scoring system. First, all the patients were Korean, which BMI were relatively lower than Western population. Comparing our baseline characteristics to patients incorporated in PRESERVE score, the incidence of immunocompromised patients, Charlson comorbidity scores and mean age were higher, while mean BMI of patients was lower in our study population. In a recent report, Asian race and low BMI were associated with poor outcome (13). In PRESERVE score, Schmidt et al. (2) showed that a BMI greater than 30 kg/m2 may be protective and included as one of variables. In this study, despite a lower BMI in our patients compared to the population of PRESERVE score, PRESERVE score showed a good predictive power for hospital mortality. Second, in our study population, most ARDS cases originated from sepsis, and 80% of patients required vasopressor and approximately half of patients required bicarbonate infusion to stabilize metabolic and respiratory acidosis. This septic condition was related to poor outcome prediction of ECMOnet score, which was based on the patients with H1N1 virus infection (3). In the RESP score study, bicarbonate infusion was an independent risk factor for hospital mortality (4). The rate of bicarbonate use was higher in our patients than in the cohort of RESP score study (56% vs. 18%). As well, 40% of patients with hemodynamic instability required arterial cannulation in our cohort. The PRESERVE score was mostly based on ARDS patients on VV ECMO and only 5% of patients applied VA ECMO for hemodynamic support. In a recent external validation study, PRESERVE score and RESP score failed to predict hospital mortality for patients requiring VA ECMO (14). However, despite a large portion of VA mode in our population, this study showed that PRESERVE score and RESP score considerably predicted the hospital mortality. Third, our cohort included only the in-house ECMO patients. The outcome prediction system, which was proposed by Roch et al. (5) was based on ECMO-treated patients retrieved from referring hospital. A calculation of an adapted score developed by Roch et al. (5) for our study group showed weaker prediction power than original data (AUC 0.705 vs. 0.802). Moreover, Roch et al. (5) suggested that SOFA score was independently associated with hospital mortality in ARDS patients. In our study, however, SOFA score did not show better predictive ability compared to RESP score, PRESERVE score and SAPS II score. Overall, it was found that prediction outcome of SAPS II score was favorable in our patients with AUC of 0.782 (95% CI 0.64-0.88). Thus, it is suggested that SAPS II score is useful to be considered as an alternative mortality prediction scoring tool before ECMO initiation. The strength of SAPS II score is that its variables are readily available to clinicians (15). Future studies are necessary to evaluate the SAPS II score for further validation of pre-ECMO mortality prediction.

There are some limitations in our study. This study was a retrospective analysis and a relatively small number of subjects were selected from one single center. The proportion of prone position before ECMO therapy was very low (6% vs. 59%) comparing to PRESERVE score study (2). This may weaken the prediction power of PRESERVE score in our cohort. In addition, pre-ECMO mechanical ventilation settings such as, PEEP and the peak inspiratory pressure, were generally less invasive in terms of the optimal ARDS-therapy prior to ECMO. Despite these limitations, this study showed acceptable outcome prediction using RESP score and PRESERVE score. Up to now there are a few Asian centers which registered in ELSO and pre-existing mortality prediction tools for patients with ARDS requiring ECMO support were established mainly focused on Western population. In order to select appropriate potential candidates for ECMO therapy, validation of these prognostic scoring systems in a more general population would be necessary. Although our patients could not be generalized to all Korean patients with ECMO, our study proposed that Korean, more ideally Asian population if possible, would be included in the future development of tools for mortality prediction. However, these prognostic scoring systems still limit its general applicability and have no ability to predict the long term outcomes such as quality of life and complications after ECMO therapy. Therefore, scoring systems should not be considered a substitute for clinical evaluation in decision for ECMO therapy.