Prognostic Value of an Estimate-of-Risk Model in Critically Ill Obstetric Patients in Brazil.

OBJECTIVE: To externally validate the CIPHER (Collaborative Integrated Pregnancy High-Dependency Estimate of Risk) prognostic model for pregnant and postpartum women admitted to the intensive care unit.
METHODS: A retrospective and a prospective validation study were conducted at two reference centers in Brazil. A composite outcome was defined as maternal death or need for prolonged organ support (more than 7 days) or acute lifesaving intervention. To evaluate the performance of the CIPHER model, a receiver operating characteristic curve was used and score calibration was assessed by the Hosmer-Lemeshow test. We conducted a descriptive analysis comparing the results of the current study with the results of the model development study.
RESULTS: A total of 590 women were included. The composite outcome was observed in 90 (15.2%) women. Of these, 13 (2.2%) were maternal deaths and 77 (13%) required one or more component of organ support or lifesaving intervention. The CIPHER model's area under the curve (AOC) did not show significant predictive ability (AOC 0.53, 95% CI 0.46-0.60), and consequently its calibration was poor (Hosmer-Lemeshow test P<.05).
CONCLUSION: The CIPHER model for prediction of mortality and need for interventions in critically ill obstetric patients did not perform well in our Brazilian population. Different predictors of morbidity and mortality may need to be used for patients receiving care in public hospitals in low- and middle-income countries.

Different prediction scores have been useful in guiding clinical decision making in a variety of settings. Some prognostic scores widely used in intensive care units (ICUs) include APACHE II (Acute Physiology and Chronic Health Evaluation II), SAPS II (Simplified Acute Physiology Score), SOFA (Sequential Organ Failure Assessment), and MODS (Multiple Organ Dysfunction Score). Application of these scoring systems to pregnant women with life-threatening conditions has been considered,[1-7] but several factors limit their wide applicability in obstetric populations. First, the physiologic parameters of pregnancy are unique compared with nonpregnant adults. Also, these scoring systems were developed in high-income populations, which may reduce their performance in other situations.[8]

A proposed mathematical prognostic model, the CIPHER (Collaborative Integrated Pregnancy High-Dependency Estimate of Risk) model, was developed in an obstetric population originating from 11 countries, including six low- and middle-income countries.[9,10] Results from the internal validation study[9] showed that CIPHER had good discriminatory power and good calibration in the first 24 hours after ICU admission for maternal death and organ support for more than 7 days or acute lifesaving intervention. This work is of relevant importance considering the need for an internationally useful score for obstetric population.

The aim of this study was to evaluate the predictive capability of the CIPHER prognostic model[9] for pregnant and postpartum women admitted to the ICU at two reference centers in Brazil.

METHODS

This study was conducted at two referral obstetric centers in Brazil: the Woman's Hospital (CAISM) at the University of Campinas, located in the city of Campinas, state of São Paulo, with data collected between January 1, 2013, and December 31, 2015, and at the Instituto de Medicina Integral Prof. Fernando Figueira (IMIP), located in the city of Recife, state of Pernambuco, with data collection occurring between October 29, 2018, and September 30, 2019. Both units are academic tertiary care centers.

The study population consisted of pregnant and postpartum women within 42 days of childbirth who were admitted to the obstetric ICU in the participating institutions for at least 24 hours. Women who stayed in the ICU for less than 24 hours were excluded. Participants at CAISM were collected retrospectively and participants at IMIP were collected prospectively.

At CAISM, a database had been previously built on the Research Electronic Data Capture (REDCap) secure web platform to enter data for all ICU patients. Patients previously entered in this database were included as a retrospective cohort for the current study. This database was originally built for initial CIPHER model development,[9] although only 12 patients were included in that study. The remainder of the sample size was obtained with prospective enrollment at IMIP.

The CIPHER model predicts the risk of prolonged organ support (more than 7 days) or acute lifesaving intervention and maternal death, expressed as a percentage.[9] The CIPHER model was calculated using a specific mathematical formula based on the following 10 variables: maternal age, surgical status in the 24 hours preceding ICU admission (including cesarean deliveries, laparotomies, and hysterectomies), systolic blood pressure, Glasgow Coma Scale (GCS) score, activated partial thromboplastin time, serum creatinine, serum bilirubin, serum potassium, serum sodium, and arterial blood gas pH (Fig. 1).[9] The highest CIPHER score obtained within the first 24 hours of admission was used.

Fig. 1.

Mathematical formula of the Collaborative Integrated Pregnancy High-Dependency Estimate of Risk (CIPHER) prognostic model. Modified from Payne BA, Ryan H, Bone J, Magee LA, Aarvold AB, Mark Ansermino J, et al. Development and internal validation of the multivariable CIPHER (Collaborative Integrated Pregnancy High-dependency Estimate of Risk) clinical risk prediction model. Crit Care 2018; Oct 30;22 (1):278. Doi: 10.1186/s13054-018-2215-6. Copyright 2018, The Author(s). Payne BA et al is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Silva. Prognostic Value of an Estimate-of-Risk Model. Obstet Gynecol 2021.

The main outcome was a composite of maternal death, organ support for more than 7 days, or lifesaving intervention. Organ support and lifesaving interventions included in our primary composite were the same as those used for the original study in which the CIPHER model was developed[9] and are also used as a surrogate for severe maternal morbidity defined by the World Health Organization.[11,12] Specifically, organ support outcomes included 1) respiratory (continuous positive airway pressure, bilevel positive airway pressure, or invasive ventilation), 2) cardiac (inotrope or vasopressor use), 3) continuous renal replacement therapy for acute renal failure (hemodialysis), 4) hepatic (liver transplantation and other management of hepatic failure [for example, ventilatory and circulatory support], management of elevated intracranial pressure, medical therapies for hepatitis B virus [for example, lamivudine], and anticoagulation for Budd-Chiari syndrome), 5) hematologic (transfusion of at least 5 units of blood products), 6) neurologic (GCS score less than 10), or 7) uterine (uncontrollable hemorrhage or infection leading to hysterectomy).

Maternal death was defined as the death of the woman during pregnancy or up to 42 days postpartum, regardless of the duration or location of the pregnancy, from any cause related to or aggravated by the pregnancy or measures related to the pregnancy but not from accidental or incidental causes.[13] If the participant's state of consciousness was compromised by the effect of residual anesthesia or use of sedation, the normality parameter evaluated by GCS score was used.[14]

We collected information on marital status, weight, body mass index (BMI, calculated as weight in kilograms divided by height in meters squared), number of prenatal visits, number of pregnancies, number of deliveries, the reason for admission to the ICU, timing of admission (antepartum or postpartum), gestational age at ICU admission, transfer to external ICU, length of ICU stay, stillbirth, early miscarriage and early neonatal death (from birth up to the 7th day of life).

A single case report form built in the REDCap platform was used at both IMIP and CAISM sites for standardization of data collection. The form was piloted with 10 women who met the inclusion criteria before initiation of data collection. Each patient's clinical and laboratory data were collected from medical records retrospectively for CAISM and prospectively (during the period of hospital stay) for IMIP. Quality-control procedures for the data were performed by double independent abstraction of information from clinical records.

There was ongoing verification of data entry and correction of inconsistencies. Missing information in medical records was searched for in other sources, such as the hospital's laboratory database, prenatal care booklets, and clinical documentation at the time of transfer. Data were obtained for each woman for the calculation of the probability of the primary outcome using the CIPHER model.

For statistical analysis, SAS 9.2 and the Stata 12 programs were used. For comparison of baseline characteristics of groups with and without the composite outcome, the χ2 test was used for categorical variables, and the Fisher exact test was used for expected values less than five. The Mann-Whitney test was used for numerical values given a nonnormal distribution of variables. A significance level of 5% was adopted.

Relationship between each variable in the CIPHER model and the composite outcome was evaluated by performing logistic regression analysis with univariable and multivariable models with reporting of odds ratios and 95% CIs.

A receiver operating characteristic curve was used to show the counterbalance between sensitivity and specificity.[15] Score calibration was assessed by Hosmer-Lemeshow test that compared the predicted risk of outcome based on the CIPHER score with the occurrence of actual outcome.[16]

The performance of the CIPHER model was evaluated according to the power of discrimination and calibration in line with the reporting standard recommended by the TRIPOD statement.[15] The power of discrimination was obtained by the area under curve (AUC),[17] classified as poor (AUC less than 0.69), acceptable (AUC 0.7–0.79), good (AUC 0.8–0.89) and excellent (AUC 0.9 or greater).[18,19] A good calibration implies value of Hosmer-Lemeshow test P>.05.[19,20] We conducted a descriptive analysis comparing the results of the current study with the results of the initial model development study.

Independent research ethics board approval for the prospective portion of the study was obtained at IMIP (CAAE: 97753618.5.1001.5201). All prospectively enrolled patients were informed of the objectives and methods of the study, and only those who provided written consent were included. The retrospective data collection was considered exempt from requiring a written informed consent; however, the protocol had institutional review board approval at CAISM.

RESULTS

Over the study period, 994 women were admitted to the obstetric ICU at both centers. Of these, 404 were excluded because they were gynecologic patients (n=146), stayed for less than 24 hours within the ICU (n=240), or were more than 42 days postpartum (n=18). Therefore, a cohort of 590 women was analyzed.

Characteristics of the study population are presented in Table 1, comparing women with and without the composite outcome, as previously defined. Most of the women with the composite outcome were older, had lower BMIs, and were admitted for obstetric reasons. Furthermore, they more often had a vaginal birth, were admitted to the ICU in the postpartum period for longer and suffered a greater number of perinatal deaths. The composite outcome was observed in 90 (15.2%) women. Of these, 13 (2.2%) were maternal deaths, and 77 (13%) required one or more organ support or lifesaving intervention. The most common lifesaving intervention outcome was a hysterectomy for uncontrollable hemorrhage or infection that occurred before the ICU admission or during the ICU stay (Table 2).

Table 1.

Characteristics of the Study Population

Table 2.

Incidence of Maternal Death and Maternal Morbidities (Intervention Criteria Grouped by the Organ Involved)

Results of univariable and multivariable logistic regression analyses to study the relationship between the CIPHER variables and the composite outcome are shown in Table 3. According to the results of the multivariable analyses, maternal age, surgical history in the preceding 24 hours, systolic blood pressure, the GCS score and arterial pH were significantly associated with the outcome. Women with the highest risk of outcome were those with older age (3.9% increased risk per year), a surgical history (3.6 times increased risk), lower systolic blood pressure (2.7% increased risk per 1 mm Hg decrease), lower GCS score (22.7% increased risk per one point decrease) and lower pH (60% increased risk per 0.1-unit decrease).

Table 3.

Univariable and Multivariable Analyses for the Risk of the Composite Outcome

The CIPHER model had poor predictive capability for our primary composite outcome (AUC 0.53, 95% CI 0.46–0.60) (Fig. 2), and consequently its calibration was also poor (Hosmer-Lemeshow test 16.7, P=.03) (Fig. 3). The performance was poor compared with the AUC in the initial model development study, in which the AUC was 0.84 (95% CI 0.83–0.85).[9]

Fig. 2.

Receiver operating characteristic curve for assessing the performance of the CIPHER (Collaborative Integrated Pregnancy High-Dependency Estimate of Risk) model for the external validation study. Area under curve: 0.53; P=.37; 95% CI 0.46–0.60. Cutoff: CIPHER model 11.6% or greater.

Silva. Prognostic Value of an Estimate-of-Risk Model. Obstet Gynecol 2021.

Fig. 3.

Calibration plot of the CIPHER (Collaborative Integrated Pregnancy High-Dependency Estimate of Risk) model developed using the predicted probabilities of outcome on the x-axis and the observed proportion in each decile with corresponding 95% CI on the y-axis from a Poisson regression model.

Silva. Prognostic Value of an Estimate-of-Risk Model. Obstet Gynecol 2021.

DISCUSSION

The CIPHER model had poor predictive capability in our population. These findings differ from the CIPHER model development study,[9] in which an AUC of 0.82 was found after adjustment for internal validation. We had only three (0.5%) patients with a CIPHER score of 50% or greater, which was the best cutoff to predict the primary composite in the development study. In the initial study conducted by Payne et al,[9] there were 59 (7.6%) patients with a CIPHER score of 50% or greater, and 39 (66%) of them had the composite outcome. Additionally, the number of deaths in the internal validation study was 59 (7.7%), whereas we had only 13 (2.2%), which suggests that our population had less severe outcomes.[9]

The observed difference in predictive capability between the initial study and our study may be due to the fact that the majority (57.6%) of the composite outcomes from the internal validation study came from one country, Pakistan.[9] It is possible that characteristics of the clinical setting in Pakistan differed from the present setting in Brazil, such as the finding of more septic obstetric patients requiring ICU admissions. In addition, there could be differences between settings for the two studies regarding the threshold for ICU admission, chronic underlying medical conditions and the availability of hospital resources.

Most of the women with the composite primary outcome were admitted for obstetric reasons. Hysterectomy was the most common lifesaving intervention, totaling 50 (8.5%) hysterectomies, whereas, in the model development study, only 16 (2%) hysterectomies were performed.[9] In contrast, in the model development study, women with the composite outcome were more frequently admitted for nonobstetric reasons (74.8%), and the need for respiratory support was the most common lifesaving intervention (73%). We speculate that our patients had more obstetric causes for ICU admission, which may explain the increased hysterectomy rate. This intervention, in turn, may have protected them from other organ dysfunctions. Emergency obstetric hysterectomy is recognized as a lifesaving intervention responsible for reducing maternal mortality related to both hemorrhage and infection.[21-23] Notably, our study was performed in two obstetric ICUs, whereas the original study was performed predominantly in general ICUs, perhaps prioritizing the admission of severely ill nonobstetric patients.

Only 5 of the 10 CIPHER model variables were significantly associated with the primary outcome in multivariable analyses: maternal age, surgical history, systolic blood pressure, GCS score, and arterial pH. Two of these variables (GCS score and arterial pH) were strongly associated with maternal death in the Maternal Severity Index developed in Brazil.[8] Blood pressure was associated with severe maternal morbidity with an AUC of 0.90 (95% CI 0.74–1.0) in another Brazilian study[24] that evaluated the performance of the SOFA score applied in cases of severe maternal morbidity. However, the Maternal Severity Index does not predict severe maternal outcomes, and the CIPHER model was developed to fill this gap.

Although the primary outcome occurred more frequently at IMIP than at CAISM, there was not a significant difference between the centers in the performance of the CIPHER model. Because both centers are public hospitals and have obstetric ICUs, their populations likely had similar characteristics. In addition, both of these centers are tertiary care referral centers, which might mean that patients were promptly treated and arrived at the ICU in less severe conditions than in nontertiary care settings, thereby reducing observed maternal death events.[25]

Vaginal birth was associated with the primary composite outcome. These births probably occurred in settings where assistance could be precarious owing to incomplete obstetric teams or absence of protocols in the setting of acute complications. Lower BMI was also associated with the outcome; this may be noteworthy because anemia, which is more common among underweight woman, exacerbates the effects of hemorrhage and may lead to higher rates of blood transfusion and hysterectomy.[26,27]

Limitations of this study include the lower rate of adverse outcomes observed compared with the expected rate based on the prevalence of adverse outcomes in the model development study. This was probably due to the differences between populations of both studies.

This was intended to be an external validation study of the CIPHER model to using data from two obstetric centers in Brazil. However, this cohort did not validate the CIPHER model, rather it did not perform well at all resulting in poor predictive capability. Although the CIPHER model has the potential to predict maternal death and severe maternal morbidity in ICU settings, it may not be relevant for populations in low- and middle-income countries. Evaluation of the predictive capability of the CIPHER model in other settings will help determine whether the model can be used as a clinical tool to optimize resources and reduce operational costs by more efficiently identifying critically ill pregnant and postpartum women at highest risk for adverse outcomes.