Performance of Multiple Risk Assessment Tools to Predict Mortality for Adult Respiratory Distress Syndrome with Extracorporeal Membrane Oxygenation Therapy: An External Validation Study Based on Chinese Single-center Data.

BACKGROUND: There has been no external validation of survival prediction models for severe adult respiratory distress syndrome (ARDS) with extracorporeal membrane oxygenation (ECMO) therapy in China. The aim of study was to compare the performance of multiple models recently developed for patients with ARDS undergoing ECMO based on Chinese single-center data.
METHODS: A retrospective case study was performed, including twenty-three severe ARDS patients who received ECMO from January 2009 to July 2015. The PRESERVE (Predicting death for severe ARDS on VV-ECMO), ECMOnet, Respiratory Extracorporeal Membrane Oxygenation Survival Prediction (RESP) score, a center-specific model developed for inter-hospital transfers receiving ECMO, and the classical risk-prediction scores of Acute Physiology and Chronic Health Evaluation (APACHE) II and Sequential Organ Failure Assessment (SOFA) were calculated. In-hospital and six-month mortality were regarded as the endpoints and model performance was evaluated by comparing the area under the receiver operating characteristic curve (AUC).
RESULTS: The RESP and APACHE II scores showed excellent discriminate performance in predicting survival with AUC of 0.835 (95% confidence interval [CI], 0.659-1.010, P = 0.007) and 0.762 (95% CI, 0.558-0.965, P = 0.035), respectively. The optimal cutoff values were risk class 3.5 for RESP and 35.5 for APACHE II score, and both showed 70.0% sensitivity and 84.6% specificity. The excellent performance of these models was also evident for the pneumonia etiological subgroup, for which the SOFA score was also shown to be predictive, with an AUC of 0.790 (95% CI, 0.571-1.009, P = 0.038). However, the ECMOnet and the score developed for externally retrieved ECMO patients failed to demonstrate significant discriminate power for the overall cohort. The PRESERVE model was unable to be evaluated fully since only one patient died six months postdischarge.
CONCLUSIONS: The RESP, APCHAE II, and SOFA scorings systems show good predictive value for intra-hospital survival of ARDS patients treated with ECMO in our single-center evaluation. Future validation should include a larger study with either more patients' data at single-center or by integration of domestic multi-center data. Development of a scoring system with national characteristics might be warranted.

INTRODUCTION

Extracorporeal membrane oxygenation (ECMO), a mobile extracorporeal life support device that provides temporary, complete cardiopulmonary function support, is used frequently in Intensive Care Unit (ICU) as a last resort for the treatment of adults with severe acute respiratory distress syndrome (ARDS) who failed to respond to mechanical ventilation.[1] Since ECMO therapy might increase resource requirements and increase hospital costs and can be complicated by a high risk of physical impairments,[2] an accurate survival prediction model to candidates is crucial. Survival prediction models have been developed for bedside use by clinicians,[34567] to guide the selection of appropriate candidates for treatment, provide prognosis information to the patient's family members, and to facilitate risk-adjusted comparison of individual center outcomes.[8]

The utilization of ECMO in the treatment of severe ARDS in China is still in its primary stage, and the overall survival rate has been low.[9] Compared to the use of this treatment method in other countries, the indication, timing of initiation, and sequential management of ECMO in our clinical practice have been less effective. One possible explanation for this difference is that the lack of effective survival prediction model which is suitable for Chinese ECMO clinical practice.

The aim of this study was to perform an external validation of the utility of the survival prediction models ECMOnet,[3] PRESERVE (Predicting death for severe ARDS on VV-ECMO),[4] Roch,[5] Respiratory Extracorporeal Membrane Oxygenation Survival Prediction (RESP)[6] scoring system compared to two classical ICU risk-predicting scores (Acute Physiology and Chronic Health Evaluation II [APACHE II], and Sequential Organ Failure Assessment [SOFA]).

METHODS

Patient selection

All consecutive ARDS adults (>18 years old) who received ECMO therapy in our institute between January 2009 and July 2015 were included. Approval was obtained from our hospital research ethics committee and all individuals provided signed consent forms prior to participation.

Inclusion and exclusion criteria

According to the Extracorporeal Life Support Organization,[10] ECMO establishment can be considered to treat hypoxic respiratory failure when the risk of mortality is 50% or higher as identified by arterial oxygen pressure/inspired oxygen fraction (PaO2/FiO2) below 150 mmHg (1 mmHg = 0.133 kPa) on FiO2 over 90% and/or Murray score 2–3. ECMO can be initiated when risk exceeds 80%, where PaO2/FiO2 is <80 mmHg on FiO2 more than 90% and Murray score is 3–4, or in the case of hypercapnia as indicated by partial pressure of carbon dioxide in artery (PaCO2) over 80 mmHg or inability to achieve safe inflation pressures (platform pressure [Pplat] <30 cmH2O, 1 cmH2O = 0.098 kPa). Patients were excluded if they (1) had been on mechanical ventilation for more than 10 days, (2) showed contraindication to anticoagulation, (3) had severe chronic pulmonary parenchymal disease, (4) were in the terminal stage of cancer, or (5) without survive transport.

Establishment and management of extracorporeal membrane oxygenation

Veno-venous (V-V) mode was the best choice for our hemodynamically stable patients. Transfemoral venous and transjugular percutaneous catheterization were performed under local anesthesia. Veno-arterial (V-A) mode (ipsilateral femoral arterial and femoral venous cannulation) was utilized for selected ARDS patients associated with severe cardiac dysfunction (left ventricular ejection fraction [LVEF] <35%) or who had just undergone cardiopulmonary resuscitation. Systemic heparinization was administered to keep the activated clotting time between 160 s and 200 s. Blood flow was maintained between 3 and 4 L/min. Oxygen concentration and flow were adjusted to maintain arterial oxygen saturation between 90% and 95% and PaCO2 at about 40 mmHg. The respirator parameter settings were gradually reduced to implement a “lung rest” strategy following ECMO and remained unchanged throughout the entire auxiliary process.

Other therapies

“Full coverage” anti-infection modes (e.g., linezolid + imipenem and cilastatin sodium + voriconazole + oseltamivir phosphate) were regularly administered to the patients. Body fluid samples such as sputum and blood were also subjected to pathogen detection to guide the subsequent use of antibiotics. Transpulmonary thermodilution (TPTD) monitoring was established to guide volumetric resuscitation and targeted diuretic therapy (ECMO was suspended for almost 1 min when the measurement of TPTD parameters was conducted). Blood products such as packed red blood cells, fresh frozen plasma, albumin, and platelets were added to allow for hemoglobin 120–140 g/L, prothrombin (international normalized ratio) >1.5, platelet count >75 × 109/L, and an albumin concentration of >30 g/L. Depending on the improvement in hemodynamic indexes, inotropic drug dosage decreased gradually. Ulinastatin was routinely used as an anti-inflammatory. Continuous renal replacement therapy was administered to patients complicated by acute kidney injury (defined by Acute Kidney Injury Network standard).[11]

Extracorporeal membrane oxygenation removal

For V-V ECMO, extracorporeal assistance is reduced as the pulmonary function recovers. Once the flow has fallen to be 1 L/min, the patient might be ready to slowly transition off the ECMO. The oxygen flow decreased to zero and the blood flow and anticoagulation intensity were kept unchanged as the ECMO was intended to remove. Respirator parameters were up-regulated (about PEEP 10 cmH2O, peak airway pressure <30 cmH2O, and FiO2 < 60%) simultaneously and hemodynamic indexes and gas exchange conditions were closely monitored. The ECMO could be removed once oxygenation condition was satisfactory and lasted more than 2 h.

For V-A ECMO, in addition to the improvement of lung mechanics and ventilation, there must be a significant improvement of the left heart function as demonstrated by echocardiography (LVEF >50%) and no obvious pulmonary hypertension or pericardial effusion for ECMO withdrawal. The process of withdrawal is similar to the V-V mode.

Data collection

Data of baseline demographic characteristics, physiologic, respiratory, and other laboratory data immediately prior to initiation of ECMO therapy, the use of precedent adjunctive therapy, ICU admission mode (in-house or transfer to our center), ventilation treatment, hospital survival, and 6-month survival after ICU discharge were recorded. The PRESERVE, ECMOnet, RESP, and Roch scores were calculated using the published descriptions [Table 1].[3456] The PRESERVE score was developed to predict survival 6 months postdischarge, and the other three models were developed to predict in-hospital mortality.

Table 1

Survival prediction models in patients with acute respiratory failure treated with ECMO

Prediction model	Number of centers	Number of patients	Primary disease	Risk factors of mortality	Internal validation c (95% CI)
ECMOnet Pappalardo et al., 2013[3]	14	60	Influenza A (H1N1)- associated ARDS	Pre-ECMO hospital LOS	0.86 (0.75–0.96)
				Bilirubin level
				Creatinine level
				Hematocrit level
				Mean arterial pressure
PRESERVE Schmidt et al., 2013[4]	3	140	ARDS	Age	0.89 (0.83–0.94)
				BMI
				Immunocompromised
				SOFA
				Days of MV
				Prone positioning
				PEEP
				Plateau pressure
Roch et al., 2014[5]	1	85	ARDS retrieved from others hospital	Age	0.80 (0.71–0.89)
				SOFA
				Influenza pneumonia
RESP Schmidt et al., 2014[6]	280	2355	Severe acute respiratory failure	Age	0.74 (0.72–0.76)
				Immunocompromised
				Days of MV
				Diagnosis
				Central nervous system
				Dysfunction
				Acute nonpulmonary infection
				Neuromuscular blockade agents
				Nitric oxide use
				Bicarbonate infusion
				Cardiac arrest
				PaCO2
				Peak inspiratory pressure

The concordance (c) statistic is identical to the area under the receiver operating characteristic curve. ARDS: Adult respiratory distress syndrome; ECMO: Extracorporeal membrane oxygenation; LOS: Length of stay; SOFA: Sequential Organ Failure Assessment; MV: Mechanical ventilation; PEEP: Positive end-expiratory pressure; PaCO2: Partial pressure of carbon dioxide in artery; CI: Confidence interval; BMI: Body mass index; RESP: Respiratory Extracorporeal Membrane Oxygenation Survival Prediction; PRESERVE: Predicting death for severe ARDS on VV-ECMO.

Statistical analysis

SPSS Statistics 16.0 software (IBM, Chicago, IL, USA) was used for data analysis. The quantitative data, in normal distribution, were expressed as a mean ± standard deviation (SD); the comparison between two groups was determined via t-test or t’ test. Abnormally distributed quantitative data were expressed with median and inter-quartile ranges and the Mann-Whitney U- tests were applied for comparison. The qualitative data were expressed by frequency and composition. The differences in constituent ratio among various groups were compared via Pearson Chi-squaretest or Fisher's exact test. All P values were two-sided and considered statistically significant if P < 0.05. The discriminative performances of the models were evaluated by receiver operating characteristic (ROC) curve analysis. Kaplan-Meier curves for cumulative survival after ECMO initiation were constructed and compared with the use of the log-rank test.

RESULTS

Baseline demographic and clinical characteristics

A total of 24 patients suffering from severe ARDS were recommended for ECMO therapy. One patient did not survive transport, thus 23 candidates received treatment and were included in this analysis. V-V and V-A mode were applied in 18 (78%) and 4 (17%) cases, respectively, and there was one case (4%) where V-V and then V-A mode was used. The leading trigger for needing ECMO treatment was severe pneumonia (78%) followed by sepsis (17%). Sixteen (70%) of the patients were transferred by ambulance from peripheral hospitals, of which 7 (30%) received ECMO treatment at an outside hospital before being transported to our institution. Five patients (22%) with influenza A (H1N1) were diagnosed, including two patients who were transferred to our center after undergoing cesarean procedures in another hospital. The overall rates of survival to discharge and 6-month survival postdischarge were 57% (13/23) and 52% (12/23), respectively [Table 2].

Table 2

Comparison of clinical characteristics according to hospital survival

Parameter	All patients (n = 23)	Nonsurvival (n = 10)	Survival (n = 13)	Statistics	P
Age (years)	45.7 ± 18.8	51.7 ± 18.5	40.8 ± 18.4	1.285†	0.180
BMI (kg/m2)	25.1 ± 2.3	25.8 ± 3.0	24.6 ± 1.4	0.930†	0.216
APACHE II score	31.5 ± 9.9	36.1 ± 6.7	27.6 ± 10.6	2.217†	0.035*
SOFA score	10.7 ± 4.8	12.1 ± 3.8	9.7 ± 5.3	1.216†	0.238
Murray score	3.3 ± 0.2	3.4 ± 0.2	3.3 ± 0.1	7.115†	0.011*
Etiology of ARDS, n (%)				1.637‡	0.441
Severe pneumonia	18 (78)	9 (90)	9 (69)
Sepsis	4 (17)	1 (10)	3 (23)
Asthma	1 (4)	0 (0)	1 (8)
Pre-ECMO CPR, n (%)	3 (14)	3 (30)	0 (0)	4.485‡	0.068
Comorbidity, n (%)
Coronary artery disease	3 (13)	2 (20)	1 (8)	0.755‡	0.560
Hypertension	7 (30)	6 (60)	1 (8)	7.304‡	0.019*
Diabetes mellitus	2 (9)	1 (10)	1 (8)	0.038‡	1.000
Stroke	4 (17)	3 (30)	1 (8)	1.958‡	0.281
Duration of ECMO support (h)	114.4 ± 65.4	117.1 ± 87.2	112.2 ± 44.0	0.285†	0.867
Pre-ECMO NEmax (µg·kg−1·min−1)	0.6 (0.3–5.0)	0.8 (0.4–6.3)	0.5 (0.3–3.9)	−1.127¦	0.283
Intervals
MV-ECMO (h)	24.0 (4.0–53.8)	43.3 (4.8–222.0)	12.5 (4.0–24.0)	−1.868¦	0.093
ECMO-MV (h)	164.0 (105.6–259.3)	118.3 (48.4–444.0)	176.0 (123.88–226.75)	−0.868¦	0.456
ICU stay (days)	11.0 (5.0–15.0)	4.5 (2.0–18.8)	12.0 (9.5–17.5)	−1.771¦	0.077
Hospital stay (days)	15.0 (5.0–27.0)	4.5(2.0–18.8)	24.0 (15.0–30.0)	–2.423¦	0.015*
Acute kidney injury, n (%)	13 (57)	9 (90)	4 (31)	8.069‡	0.010*
Complications, n (%)
Oxygenator replacement	4 (17)	4 (40)	0 (0)	6.295‡	0.024*
Systemic embolization	4 (17)	1 (10)	3 (23)	0.673‡	0.604
Catheter related infection	3 (13)	3 (30)	5 (39)	0.178‡	1.000
Coagulation dysfunction	16 (70)	8 (80)	8 (62)	0.910‡	0.405
Multiple organ failure	10 (44)	8 (80)	2 (15)	9.603‡	0.003*
Pattern of hospitalization, n (%)				2.912‡	0.405
Intra-hospital transport	3 (13)	0 (0)	3 (23)
Inter-hospital transport on MV	9 (39)	5 (50)	4 (31)
Inter-hospital transport on ECMO	7 (30)	3 (30)	4 (31)
Admission via EMS	4 (17)	2 (20)	2 (15)

Values are given as a mean ± SD or median (interquartile range) unless otherwise indicated. *P<0.05, comparison between survival and nonsurvival group. †t-test; ‡Pearson Chi-square test; §Mann-Whitney U-tests. BMI: Body mass index; APACHE: Acute Physiology and Chronic Health Evaluation; SOFA: Sequential Organ Failure Assessment; ARDS: Adult respiratory distress syndrome, ECMO: Extracorporeal membrane oxygenation; CPR: Cardiopulmonary resuscitation; NEmax: Maximal dosage of norepinephrine before ECMO; MV: Mechanical ventilation; ICU: Intensive Care Unit; EMS: Emergency medical service; SD: Standard deviation.

Clinical data for patients with different prognoses

The differences in APACHE II scores, Murray score, proportion of hypertension and acute kidney injury, membrane oxygenator replacement, occurrence of multiple organ failures, and length of hospital stay between the groups with different intra-hospital prognoses were statistically significant (P < 0.05) [Table 2]. However, no significant difference for the pre-ECMO laboratory data and ventilator settings was observed for these groups [Table 3].

Table 3

Comparison of pre-ECMO laboratory data and ventilator settings according to hospital survival

Parameter	All patients (n = 23)	Nonsurvival group (n = 10)	Survival group (n = 13)	Statistics (t)	P
White blood cell (×109/L)	12.6 ± 7.1	10.6 ± 7.0	14.2 ± 7.1	−1.208	0.240
Hemoglobin (g/L)	109.6 ± 22.4	109.8 ± 26.1	109.4 ± 20.1	0.043	0.967
Platelet count (×109/L)	137.8 ± 70.2	139.5 ± 74.3	136.5 ± 70.0	0.098	0.923
Albumin (g/L)	29.4 ± 5.6	29.2 ± 5.0	29.5 ± 6.3	−0.133	0.895
Total of bilirubin (µmol/L)	26.1 ± 17.7	19.7 ± 12.9	31.0 ± 19.7	−1.566	0.132
Urea nitrogen (mmol/L)	9.7 ± 6.2	11.5 ± 7.7	8.2 ± 4.6	1.256	0.223
Serum creatinine (µmol/L)	70.0 (51.0–169.0)	101.5 (50.0–301.3)	103.8 ± 85.4	–0.807*	0.446
Arterial PO2 (mmHg)	51.9 ± 24.4	46.9 ± 15.5	55.8 ± 29.6	−0.863	0.398
Arterial lactates (mmol/L)	4.7 ± 3.6	6.2 ± 4.2	3.6 ± 2.7	1.755	0.078
Prothrombin (%)	67.7 ± 26.3	59.7 ± 21.1	71.0 ± 28.4	−0.387	0.703
Fibrinogen (g/L)	4.5 ± 2.3	3.5 ± 1.8	5.5 ± 2.4	−2.007	0.059
Heart rate (beats/min)	112.6 ± 35.8	124.6 ± 29.9	103.3 ± 38.3	1.447	0.163
MBP (mmHg)	75.9 ± 17.8	71.1 ± 22.3	81.2 ± 9.5	1.326	0.201
PEEP (cmH2O)	13.2 ± 1.4	12.8 ± 1.7	14.2 ± 1.7	−1.573	0.136
Respiratory rate (/min)	33.9 ± 6.6	34.3 ± 6.3	33.5 ± 6.7	0.435	0.773
Tidal volume (ml/kg)	5.6 ± 1.9	4.9 ± 1.5	6.1 ± 2.0	−1.541	0.138
Minute ventilation (L/min)	7.6 ± 3.1	6.9 ± 3.5	8.1 ± 2.7	−0.902	0.377
Peak inspiratory pressure (cmH2O)	28.7 ± 3.0	29.7 ± 2.5	28.0 ± 3.2	1.388	0.180
PaO2/FiO2 (mmHg)	76.4 ± 40.5	65.8 ± 48.7	84.5 ± 32.5	−1.102	0.283

Values are expressed as mean ± SD or median (interquartile range) unless otherwise indicated. *Mann-Whitney U-tests. ECMO: Extracorporeal membrane oxygenation; MBP: Mean blood pressure, SD: Standard deviation; PEEP: Positive end-expiratory pressure, PaO2/FiO2: Arterial oxygen partial pressure/inspired oxygen fraction. 1 cmH2O = 0.098 kPa, 1 mmHg = 0.133 kPa.

Discriminate performance of the novel and classic prediction models

The ROC curve analysis of these scoring systems for predicting survival to discharge (all but the PRESERVE model) is shown in Figure 1 and Table 4. Overall, the RESP model showed the best discriminate performance for all patients, with an area under the curve (AUC) of 0.835 (95% confidence interval [CI], 0.659–1.010; P = 0.007). A statistically significant difference was observed in the distribution of patients into RESP groups with different prognoses [Table 5]. The APACHE II score showed weaker but still good discriminating power, with an AUC of 0.762 (95% CI, 0.558–0.965, P = 0.035) for all patients. Next, we evaluated the discriminate performance of these five scoring systems for prediction of survival to discharge for the subgroup with severe pneumonia, as the majority of patients in this group had this diagnosis. Again, the RESP model showed an excellent discriminate performance, with an AUC of 0.858 (95% CI, 0.679–1.037; P = 0.01). The discriminate performance of the APACHE II score was improved in this subgroup, with an AUC of 0.877 (95% CI, 0.716–1.037, P = 0.007). The SOFA score showed good discrimination performance for this subgroup, with an AUC of 0.790 (95% CI, 0.571–1.009; P = 0.038). The ECMOnet and Roch scores did not show significant discriminate performances between survival and nonsurvival to discharge for analysis of the whole group or the subset that had pneumonia. We did not perform a separate analysis of the predictive models for other subgroups such as H1N1-infected cases or patients transferred from another hospital for which ECMO was initiated by our mobile ECMO team due to small sample size (five H1N1 patients and seven patients who received care from the mobile unit).

Figure 1

Comparison of the receiver-operating characteristic curves for all risk-prediction tools (a) the whole population (n = 23). (b) The pneumonia subgroup (n = 18). APACHE: Acute Physiology and Chronic Health Evaluation; SOFA: Sequential Organ Failure Assessment; RESP: Respiratory Extracorporeal Membrane Oxygenation Survival Prediction; ECMO: Extracorporeal membrane oxygenation.

Table 4

ROC Analysis of the RESP score, ECMOnet score, the score published by Roch, APACHE II and SOFA score for predicting survival to discharge of all patients and the pneumonia subgroup

Predicted model	AUC	P	95% CI	Cutoff	Sensitivity (%)	Specificity (%)
Whole population
APACHE II	0.762	0.035*	0.558–0.965	35.5	70	84.6
SOFA	0.700	0.107	0.482–0.918	–	–	–
RESP	0.835	0.007*	0.659–1.010	3.5†	70	84.6
Model from Roch	0.619	0.336	0.384–0.854	–	–	–
ECMOnet	0.554	0.664	0.314–0.794	–	–	–
Severe pneumonia subgroup
APACHE II	0.877	0.007*	0.716–1.037	35.5	77.8	88.9
SOFA	0.790	0.038*	0.571–1.009	8.5	88.9	66.7
RESP	0.858	0.010*	0.679–1.037	3.5†	66.7	100
Model from Roch	0.667	0.233	0.408–0.925	–	–	–
ECMOnet	0.611	0.427	0.344–0.878	–	–	–

*P value represents the discriminate performance of the models for predicting survival to discharge; †The unit of cutoff value is risk class; Null hypothesis: True area = 0.05. AUC: Area under the curve; CI: Confidence interval; APACHE: Acute Physiology and Chronic Health Evaluation; SOFA: Sequential Organ Failure Assessment; RESP: Respiratory Extracorporeal Membrane Oxygenation Survival Prediction; –: No significance; ECMO: Extracorporeal membrane oxygenation.

Table 5

Distribution of all patients and pneumonia subgroup into RESP groups with different intra-hospital prognoses

RESP risk class	All patients (n = 23), n (%)	Status at discharge n (%)		Statistics (χ2)	P	Pneumonia Subgroup (n = 18), n (%)	Status at discharge, n (%)		Statistics (χ2)	P
		Alive (n = 13)	Dead (n = 10)				Alive (n = 9)	Dead (n = 9)
I	2 (8.7)	2 (15.4)	0 (0)	9.775	0.044*	2 (11.1)	2 (22.2)	0 (0)	9.667	0.046*
II	8 (34.8)	6 (46.2)	2 (20)	–	–	6 (33.3)	4 (44.4)	2 (22.2)	–	–
III	4 (17.4)	3 (23.1)	1 (10)	–	–	4 (22.2)	3 (33.3)	1 (11.1)	–	–
IV	4 (17.4)	2 (15.4)	2 (20)	–	–	2 (11.1)	0 (0)	2 (22.2)	–	–

*Comparison between survival and non-survival group. RESP: Respiratory Extracorporeal Membrane Oxygenation Survival Prediction. –: No data.

The optimal cutoff values of the RESP score and APACHE II were risk class 3.5 and 35.5, respectively, when used for evaluation in the overall cohort, and both showed 70.0% sensitivity and 84.6% specificity. The same values (3.5 and 35.5) were obtained in the evaluation of the models for the predictive value of outcomes for the severe pneumonia subgroup, with 66.7%, 77.8% sensitivity and 100%, 88.9% specificity, respectively. The optimal cutoff of SOFA score for prediction of hospital survival was 8.5, with 88.9% sensitivity and 66.7% specificity for the pneumonia subgroup.

Because only one patient (discharged ahead of schedule) died within the 6-month postdischarge period, the PRESERVE model was unable to be tested. Nevertheless, a significant difference in the cumulative probabilities of survival by days after ECMO initiation was identified when groups met the optimal cutoff values of the RESP, APACHE II, and SOFA scores [Figure 2].

Figure 2

Kaplan-Meier estimates of cumulative probabilities of survival for patients with pre-ECMO RESP, APACHE II, and SOFA score (a) RESP score classes <3.5 (n = 14), and >3.5 (n = 9) (log-rank test: χ2= 10.31, P = 0.001), and (b) APACHE II score <35.5 (n = 14), and >35.5 (n = 9) (log-rank test: χ2= 10.41, P = 0.001) in the whole population. (c) SOFA score <8.5 (n = 7), and >8.5 (n = 11) in the pneumonia subgroup (log-rank test: χ2= 6.230, P = 0.013). RESP: Respiratory Extracorporeal Membrane Oxygenation Survival Prediction; ECMO: Extracorporeal membrane oxygenation; APACHE: Acute Physiology and Chronic Health Evaluation; SOFA: Sequential Organ Failure Assessment.

DISCUSSION

The ECMO databases varied in terms of size, etiology diagnoses, and the types of data collected, contributing to the heterogeneity of these survival probability models. In addition, there might be some differences in modern domestic ECMO practice and how ECMO is performed in other countries, possibly rising from differences in economic development, available medical resources, or cultural and ethnic factors between different countries. Therefore, it is important to validate the recently developed survival predicting models developed using data from other countries and determine the most appropriate scoring systems for providing healthcare in China.

This study was a retrospective analysis of 23 patients at our institution evaluating the validity of the recently developed models of RESP, ECMOnet, and the Roch model. The aim was to evaluate the efficacy of these scoring systems to improve identification of those who were likely to gain benefits from ECMO. Our results indicated that the RESP model was the most valuable in the identification of severe ARDS patients who would benefit from ECMO treatment. In contrast with the results of earlier researchers,[12] by incorporating the ICU commonly used severity scoring systems APACHE II and SOFA, we determined that the two classical scoring are valuable tools for the survival prediction of ECMO treatment in severe ARDS patients. This was an important finding because models specific to ARDS patients might be assumed to be more valuable than classical risk assessment tools for decision-making when ECMO has been considered. To the best of our knowledge, there have been few reports examining the validity of these models domestically for the estimation of survival probability of severe ARDS patients with ECMO. Therefore, our results are valuable to enable further stratification of mortality risk for domestic ECMO treatment in severe ARDS.

The RESP model was developed from a large-scale study, constitutes 2355 critical ARDS patients who received ECMO therapy in multiple countries over a 13-year period.[6] This larger sample size allowed creation of a prediction model that is well-calibrated, with extensive application scope, great discriminatory power, and allowed an individual center to compare its risk-adjusted outcome against their international counterparts. Applying this prediction model retrospectively confirmed its excellent performance in discriminating survival probability for our overall cohort (concordance, [c] = 0.835, [95% CI, 0.659–1.010; P = 0.007]) and the pneumonia subgroup (c = 0.858, [95% CI, 0.679–1.037; P = 0.01]). We found much poorer discrimination of SOFA scores (c = 0.700, [95% CI, 0.482–0.918; P = 0.107]), which was also reported in an external validation of RESP score using the PRESERVE data-set.[6] Due to the small sample size of patients in each risk class after grouping by RESP score, the optimal cutoff value obtained from the analysis of ROC curve of the scoring system was used to establish the Kaplan-Meier estimates of cumulative probability of survival. As a result, the overall observed survival was much higher for the group with risk class less than 3.5 compared to the group with a risk class that was >3.5 (log-rank test, P = 0.001). In addition, our findings also suggested that with regard to classical ICU risk-predicting scores, the APACHE II score might have a broader application than SOFA score in the selection of ARDS candidates who would benefit from ECMO therapy.

We compared our baseline characteristics to those of the patients in Schmidt's et al.[6] study and found that the patients were comparable in terms of age, etiology, and composition of ECMO mode, but the incidence of immunocompromised patients, use of pre-ECMO rescue therapy (e.g., high-frequency oscillation ventilation or inhaled nitric oxide), prone positioning before ECMO, and days of mechanical ventilation prior to treatment with ECMO was lower in our cohort. The lower incidence of these conditions could result in an overestimation by up-regulating the score calculation. A shorter length of mechanical ventilation and relative looser standards of respirator settings before ECMO establishment were characterized in our study, indicating that our timing of initiating ECMO treatment was earlier than the conditions applied in the data used for the development of the RESP model. Although the specific timing of ECMO establishment among different centers remains controversial, many reports[46131415] have shown that extended mechanical ventilation prior to ECMO is a strong independent predictor of poor prognosis. Although refractory hypoxemia is used as a main indicator for ECMO, oxygenation status itself was not included in these models, suggesting that alterations in lung mechanics, represented by a higher plateau pressure or lower PEEP,[6] might be more important prognostic factors than the severity of hypoxemia. Therefore, there remains a question about the potential benefits of conventional rescues therapies prior to the initiation of ECMO treatment, previously concluded to not affect mortality reduction.[161718192021]

Compared to the Schmidt et al.'s findings,[6] our study indicated that an earlier application of ECMO resulted in a shorter duration of ECMO treatment and showed comparable survival rate with improved health economics. We believe that this benefit is due to the establishment of a regional cooperative network model for ECMO referral treatment. In the past 5 years, a mode of regional and inter-regional organization of ECMO activity has developed with networks of hospitals and an ECMO referral center with a mobile ECMO unit. This regional ECMO and ventilator treatment center were used for the treatment of critical ARDS patients admitted to the networked hospitals. The rescue network covers a referral radius of 125 km, and satisfactory initial results have been achieved in the intensive treatment of severe ARDS patients via a well-trained mobile ECMO team, positive consultation principle, and standardized referral criteria.

A lack of validation for the predictive value of the ECMOnet and Roch scores for the overall cohort is consistent with the results of Klinzing et al.[12] This might be due to: (1) a small number of patients with H1N1-infection and receiving ECMO during inter-hospital transport in our cohort, (2) no standardized predefined indications for ECMO, (3) different numbers of patients with sepsis or influenza pneumonia, which was closely related to the prognosis. The prognostic discriminate accuracy of these two models might be affected significantly by these confounding factors given the small number of patients in this study. This might also explain the failure of the PRESERVE model in our study. The ECMOnet and Roch scores were developed and validated for a specific ARDS population, which might preclude the widespread use of this score for patients with other diagnoses.

This study has some limitations. The retrospective design and the small number of patients limit the statistical power of the analysis. Although a gradual increase has occurred in the recent decade, the number of domestic ECMO cases is relatively small, especially for nonpostcardiotomy indications. Regarding the indication of ARDS, the transportation of severe ARDS patients to ECMO centers where they receive specialized tertiary care has not become a routine choice for care in China. For this reason, improved techniques for administering ECMO during inter-hospital transportation to an ECMO center are urgently needed. Future evaluation efforts should include a larger study with either more patients at a single-center or by integration of domestic multi-center data. Development of a scoring system with national characteristics might be warranted.

Financial support and sponsorship

This work was partly supported by the grant from the Key Scientific Project of Tianjin Municipal Bureau of Health (No. 12KG106).

Conflicts of interest

There are no conflicts of interest.