Assessment of the fullPIERS Risk Prediction Model in Women With Early-Onset Preeclampsia.

Early-onset preeclampsia is associated with severe maternal and perinatal complications. The fullPIERS model (Preeclampsia Integrated Estimate of Risk) showed both internal and external validities for predicting adverse maternal outcomes within 48 hours for women admitted with preeclampsia at any gestational age. This ability to recognize women at the highest risk of complications earlier could aid in preventing these adverse outcomes through improved management. Because the majority (≈70%) of the women in the model development had late-onset preeclampsia, we assessed the performance of the fullPIERS model in women with early-onset preeclampsia to determine whether it will be useful in this subgroup of women with preeclampsia. Three cohorts of women admitted with early-onset preeclampsia between 2012 and 2016, from tertiary hospitals in Canada, the Netherlands, and United Kingdom, were used. Using the published model equation, the probability of experiencing an adverse maternal outcome was calculated for each woman, and model performance was evaluated based on discrimination, calibration, and stratification. The total data set included 1388 women, with an adverse maternal outcome rate of 7.3% within 48 hours of admission. The model had good discrimination, with an area under the receiver operating characteristic curve of 0.80 (95% confidence interval, 0.75-0.86), and a calibration slope of 0.68. The estimated likelihood ratio at the predicted probability of ≥30% was 23.4 (95% confidence interval, 14.83-36.79), suggesting a strong evidence to rule in adverse maternal outcomes. The fullPIERS model will aid in identifying women admitted with early-onset preeclampsia in similar settings who are at the highest risk of adverse outcomes, thereby allowing timely and effective interventions.

See Editorial Commentary, pp

Preeclampsia affects up to 5% of pregnancies worldwide and contributes substantially to maternal and fetal morbidity and mortality.[1,2] Maternal complications that could arise from preeclampsia include placental abruption and acute renal failure while fetal complications include small-for-gestational age babies, respiratory distress, and stillbirth.[3,4] Preeclampsia can be classified as early-onset preeclampsia, that is, preeclampsia occurring before 34 weeks of gestation, or as late-onset preeclampsia occurring from 34 weeks onwards. Although the pathogenesis of preeclampsia is not fully understood, studies have suggested that the causes of these 2 types of preeclampsia may be different.[5,6] It has been proposed that early-onset preeclampsia is as a result of shallow invasion of the maternal spiral arteries by the trophoblasts resulting in impaired remodeling of the arteries (placental preeclampsia) while late-onset preeclampsia is associated with maternal predisposition to arterial disease resulting in a hyperinflammatory state during pregnancy (maternal preeclampsia).[5,7] Although late-onset preeclampsia is more common, early-onset preeclampsia is associated with more severe outcomes, such as fetal growth restriction.[3] The management of early-onset preeclampsia is complicated because delivery remains the only cure for preeclampsia and could result in early preterm birth with the concomitant severe consequences of prematurity.[3,8,9]Therefore, delaying delivery where possible would be preferable although the length of time for expectant management is unclear because the mother is also at increased risk of complications.[1,8] The ability to predict the risk of maternal complications for women admitted with early-onset preeclampsia would be highly beneficial to guide their management in care facilities.[1,10]

The fullPIERS model (Preeclampsia Integrated Estimate of Risk) was developed to predict severe maternal complications, including adverse central nervous system, cardiorespiratory and hematological outcomes (full list of outcomes in Table S1 in the online-only Data Supplement) from preeclampsia occurring within 48 hours of admission; this time frame was chosen to allow for clinical decisions, such as administration of corticosteroids, transfer to higher care units, and delivery. The model was developed using a prospective cohort of 2023 women admitted with preeclampsia in tertiary units in high-income countries and had a good excellent discriminatory ability with an area under the receiver operating curve (AUROC) of 0.88 (95% confidence interval [CI], 0.84–0.92).[11] The fullPIERS model was internally validated and also showed externally validity with AUROC of 0.82 (95% CI, 0.76–0.87). Although the majority of the cohort used for the model development was from women with late-onset preeclampsia, 31.4% of the women included in the study had early-onset preeclampsia.[11] A study assessing the model in a cohort of women with severe early-onset preeclampsia also showed an excellent discriminatory performance (AUROC, 0.97 [95% CI, 0.94–0.99]) although this study was underpowered to detect significant changes in model performance.[12,13] Therefore, our objective was to assess and confirm the validity of the fullPIERS model for early-onset preeclampsia, using a fully-powered, broad cohort of women admitted with early-onset preeclampsia in high-income countries, other than the one used in the development study.

Methods

Data, Analytic Methods (Code), and Research Materials Transparency

Because of the sensitive nature of the data collected for this study, requests to access the data set from qualified researchers trained in human subject confidentiality protocols may be sent to the ethics board of the University of British Columbia and the other organizations listed for the corresponding authors.

Ethics

Ethical approval for this validation study was obtained from the Research Ethics Board of the University of British Columbia on March 1, 2014 (CREB#: H07-02207).

Data Collection

Data used for this model assessment study were derived from 3 pre-existing cohorts of women admitted with early-onset preeclampsia in high-income countries. These were the (1) BCW hospital cohort (British Columbia Women), (2) the PETRA cohort (Preeclampsia Eclampsia Trial Amsterdam), and (3) the PREP cohort (Prediction of Complications in Early-Onset Preeclampsia).

The BCW cohort comprised data that were extracted from medical chart and electronic records of women admitted into the tertiary unit of the BCW hospital in Canada between January 2012 and May 2016. For this study, we restricted the BCW cohort to the women admitted with preeclampsia before 34 weeks of gestation. The PETRA cohort was made up of women recruited into the PETRA randomized trial study in the Netherlands between April 2000 and May 2003.[14] Data for the PETRA study were collected prospectively and only included women admitted with severe preeclampsia into tertiary centers between 24 and <34 weeks of gestation. The PREP cohort was made up of women recruited into the PREP study in the United Kingdom between December 2011 and April 2014.[15] Data for the PREP study were also collected prospectively and only included women admitted with preeclampsia into secondary and tertiary centers before 34 weeks of gestation. These cohorts were merged into a combined data set for our study with study marker retained to allow for separate study analysis.

Definition of Preeclampsia and Adverse Outcomes

Preeclampsia was defined as hypertension and either proteinuria or hyperuricemia, or HELLP syndrome (hemolysis, elevated liver enzyme levels, and low platelet levels), as in the fullPIERS development study.[11] However, the PETRA cohort included only women if they had severe preeclampsia (diastolic blood pressure >110 mm Hg) HELLP syndrome[14] or gestational hypertension (diastolic blood pressure ≥90 mm Hg with the absence of proteinuria) with fetal growth restriction (estimated fetal weight <10th centile).

The primary outcome used in our study was the same as in the model development study.[11] This was a composite outcome comprising of ≥1 of the severe maternal complications listed in the online-only Data Supplement occurring within 48 hours of admission for preeclampsia (Table S1).

Statistical Analyses

Using the worst measured predictor variables within 48 hours of admission measured before any outcome occurrence, the published fullPIERS model equation[11] (Equation 1) was applied to the combined data set to calculate the predicted probabilities of experiencing an adverse outcome for each woman.

The fullPIERS Logistic Regression Equation for the prediction of adverse maternal outcomes from preeclampsia:

−logit(pi)=2.68+(–5.41×10–2; gestational age at eligibility)+1.23(chest pain or dyspnoea)+(–2.71×10–2; creatinine)+(2.07×10–1; platelets)+(4.00×10–5; platelets2)+(1.01×10–2; aspartate trans aminase)+(–3.05×10–6; aspartate aminotransferase2)+(2.50×10–4; creatinine×platelet)+(–6.99×10–5; platelet×aspartate transaminase)+ (–2.56×10–3; platelet×Spo2)

The calculated probabilities were then used to assess the model performance for predicting adverse maternal outcomes within 48 hours of admission based on discrimination, calibration, and stratification and classification accuracy.[16,17]

Discriminative ability was interpreted as noninformative (area under the curve ≤0.5), poor discrimination (0.5< area under the curve <0.7), or good discrimination (area under the curve ≥0.7).[18] Before the merging of cohorts, the discriminative ability of the fullPIERS model was assessed in the individual cohorts.

Calibration was assessed by estimating the slope on a calibration plot of predicted versus observed outcome rates in each decile of predicted probability. Similar to the AUROC, calibration ability was interpreted as poor calibration (slope <0.7), good calibration (slope 0.7≤ slope <1.3).[17] The stratification capacity and classification accuracy of the model were assessed using a classification table with generated risk groups (based on categories established in the model development study). Stratification and classification ability were assessed based on the ability of the model to correctly classify the women into low- and high-risk categories.

Likelihood ratios were calculated for each group using the Deeks and Altman[19] method for a multicategory diagnostic test; the true- and false-positive rates, negative predictive values, and positive predictive values were also computed for each group.

Missing Data

Multiple imputations by chained equations were used to generate plausible values for any missing variable except for missing Spo2 values that were imputed with 97%, similarly done in the fullPIERS model development study and to ensure consistency.[11] We used 10 iterations of multiple imputation to generate 10 data sets. The predicted probabilities of experiencing an adverse outcome for each woman were calculated in each data set, and the final predicted risks were combined by averaging the predicted probabilities for each individual. The final average-predicted probabilities were used to evaluate the performance of the model for the imputation results.

Sensitivity Analyses

For secondary analyses, we evaluated the discriminatory performance of the model for predicting adverse outcomes within 7 days of admission. Because of known differences in the study design and definition of preeclampsia in the PETRA cohort[14] compared with the BCW and PREP cohorts, we conducted a sensitivity analysis evaluating the discriminatory performance of the model in the combined cohort excluding the PETRA cohort.

Recalibration of the model was also performed to account for differences between the development and validation cohort (early-onset preeclampsia).

All statistical analyses were performed using R version 3.1.3 (The R Project for Statistical Computing).

Sample Size

Our sample size was guided by simulation studies that recommend that validation studies should have at least 100 events (outcomes) to have 80% power at the 5% significance level.[13,17]

Results

Cohort Description

The BCW, PETRA, and PREP cohorts included 218, 216, and 954 women, respectively, making a total of 1388 women admitted with preeclampsia before 34 weeks of gestation in our analytic data set. The women in the BCW cohort appeared to be older and have a higher rate of chest pain or dyspnea and more interventions during pregnancy (higher administration of corticosteroids, antihypertensive medication, and magnesium sulfate; Table 1). The PETRA cohort had the highest reported rate of the HELLP syndrome and higher rates of stillbirth and neonatal death. The PREP cohort had higher multiparity and lower use of magnesium sulfate during pregnancy.

Table 1.

Maternal Characteristics for the Data Sets With Women GA <34 Years (BCW <34, Dutch PETRA, PREP)

Compared with the fullPIERS development cohort, the early-onset cohorts reported more chest pain or dyspnea, higher administration of corticosteroid, shorter admission-to-delivery interval, and lower birth weights.

The PETRA cohort also had the highest rate of adverse maternal outcomes within 48 hours of admission (14.8%) while the PREP cohort had the lowest rate (4.8%; Table 2). In total, the rate of adverse outcomes in the combined data set within 48 hours of admission was 7.3% (n=101), which was slightly higher than in the fullPIERS cohort with 5%.[11] The most commonly reported adverse outcomes within 48 hours of admission were low platelet count (n=26) and placental abruption (n=19); there was no reported case of maternal mortality (Table S2).

Table 2.

fullPIERS Prediction and Outcomes Rates During Admission for Preeclampsia in Data Sets

Data Completeness and Imputation Analysis

After substituting missing Spo2 values with 97% similar to the fullPIERS model development,[11] there were 43 (3.1%) cases of platelet, 46 (3.3%) cases of creatinine, and 77 (5.5%) cases of aspartate aminotransferase, missing within 48 hours of admission. There were no missing cases of gestational age at admission for preeclampsia, and none reported for chest pain or dyspnea.

Imputation of missing values did not seem to alter the model performance significantly; these results are presented in the model performance below.

Model Performance

The women in the PETRA cohort had a higher median of calculated fullPIERS probability (Table 2) and AUROC (AUROC of 0.97 [95% CI, 0.94–0.99]). The model, combined data, showed a good discrimination with an AUROC of 0.80 (95% CI, 0.75–0.86; Figure 1) although the calibration was poor with a slope of 0.68 (95% CI, 0.56–0.79; Figure 2). Imputation of the combined data did not result in any change in discrimination (AUROC of 0.80 [95% CI, 0.75–0.85]) and calibration (0.63 [95% CI, 0.52–0.74]).

Figure 1.

Receiver operating characteristic curve for performance of the fullPIERS model (Preeclampsia Integrated Estimate of Risk) in predicting adverse maternal outcome in the early-onset preeclampsia combined cohort within 48 h of admission. PV indicates predictive value.

Figure 2.

Calibration plot of the fullPIERS model (Preeclampsia Integrated Estimate of Risk) performance in the early-onset preeclampsia combined cohort. ROC indicates receiver operating characteristic curve.

The stratification capacity in the early-onset preeclampsia cohort was good as with the model development study.[11] The fullPIERS model stratified the majority of the women (64%) into the low-risk groups (predicted probability of <2.5%) and 4.4% into the highest risk group (predicted probability of ≥30%; Table 3). Conversely, only ≈3% of women in the low-risk group of <2.5% had an adverse outcome while ≈55% of the women in the highest risk group experienced an adverse outcome. At the highest predicted probability group of ≥30%, the model had a likelihood ratio of 23.4 (95% CI, 14.8–36.8), showing strong evidence to rule in an adverse outcome; the positive predictive values and negative predictive values were 96% and 65%, respectively. There was no predicted range showing strong evidence for ruling out adverse outcomes.

Table 3.

Risk Stratification Table to Assess the Performance of the fullPIERS Model for Predicting Maternal Outcome at Varying Predicted Probability Cutoff Values Within 48 Hours in the Early-Onset Preeclampsia Data Set

On secondary analyses, the fullPIERS model maintained a good discriminatory performance with AUROC of 0.74 (95% CI, 0.70–0.79) for predicting maternal adverse outcomes within 7 days of admission (Figure S1).

The performance of the model appeared to decrease after the exclusion of the PETRA cohort with AUROCs of 0.74 (95% CI, 0.67–0.81) and 0.70 (95% CI, 0.65–0.75) for predicting adverse maternal outcomes within 48 hours and 7 days of admission, respectively, although these were not significant as the CIs overlapped (Figure S2). Updating of the model intercept and slope resulted in improvement of the calibration performance (Figure S3) without affecting discriminatory performance. The updated model equation after model recalibration is shown in Equation 2.

Recalibrated fullPIERS Logistic Regression Equation for the prediction of adverse maternal outcomes from early-onset preeclampsia:

−logit(pi)=−0.29+(0.6777×original fullPIERS model)(2)

Discussion

Main Findings

We assessed the fullPIERS model in women admitted with early-onset preeclampsia. The model maintained a good discriminatory and stratification performance within 48 hours; the model also performed well for predicting adverse outcomes occurring within 7 days. There was a marginal decrease in AUROC compared with the model performance in development (AUROC of 0.80 [95% CI, 0.75–0.86] in early-onset preeclampsia versus 0.88 [95% CI, 0.84–0.92] on development). The calibration performance of the model reduced in our cohort from an ideal slope of 1 to 0.68. Simple updating methods, such as recalibration of the intercept and slope, may be used to improve the model calibration performance for this population to account for the differences in the population characteristics between the combined cohorts and the original fullPIERS population as shown in Figure S3.[17]

The case-mix differences between our cohort and the fullPIERS cohort may have attenuated the model’s performance, particularly the calibration performance.[17,20] The most obvious case-mix difference was the selective inclusion of women with early-onset preeclampsia compared with the fullPIERS cohort that had a higher proportion of women with late-onset preeclampsia. In addition, earlier onset of preeclampsia (gestational age of onset) were associated with more adverse outcomes as shown by the overall higher rate of outcomes in this data set compared with the fullPERS cohort. Therefore, it is possible that the predictor effect of gestational age in the fullPIERS model may have been different in our cohort compared with the fullPIERS cohort. Difference in predictor effect can affect a model’s performance, especially the calibration accuracy.[17,21] Other contributors to case-mix differences include the addition of women admitted with severe preeclampsia as in the PETRA cohort compared with all women with preeclampsia in the model development, as well as the addition of women admitted into both secondary and tertiary units in the PREP cohort compared with those admitted to tertiary units in the model development cohort. Another possible reason for the overall model performance reduction is the lack of spread or balance between low- and high-risk women in the combined data, that is, less heterogeneity among the women in the cohort.[17] Despite these known differences, our primary goal was to assess how well the model would perform in this subset of preeclampsia to determine whether it would be useful for this population.

The AUROCs in all the individual data sets were good (≥0.70) although the discriminatory performance appeared to be higher in the PETRA data set, even better than the original model performance. We suspect that the inclusion of a more severe-case mix of women may have resulted in the observed higher discrimination performance. In addition, this cohort had the highest proportion of both adverse maternal and fetal outcome, indicating a sicker group of women. However, our sensitivity analyses excluding the cases in the PETRA data did not result in a significant change in the AUROC of the model and still had good discriminatory performances for identifying women at the highest risk of maternal complications.

Strengths and Weaknesses

An important strength in our study is the combination of cohorts from different centers which added to the robustness and generalisability of our findings. In our study, we used a data set with adequate sample which enabled us to detect any true changes in the model performance. Because we were interested in assessing the model in a general population of early-onset preeclampsia, we think that the combination of these cohorts resulted in a broader cohort of cases that could be presented to a clinician in the hospital.

Although we had a few cases of missing data, there was no significant change in the model performance results after imputation; this suggests that the point estimates obtained were less likely to be biased.[17]

A possible limitation in our study is that we were not able to exclude the women with only gestational hypertension and fetal growth restriction from the PETRA data because of lack of availability of information to test the model performance in the women with only early-onset preeclampsia using the exact definition as in the model development study. This may have provided information to test the proposed reasons stated above for heterogeneity case-mix in the data.

Comparison to Existing Literature

A prediction model study (PREP model) on the prognosis of women with early-onset preeclampsia reported an AUROC of 0.84 (95% CI, 0.81–0.87) on development.[15] Preterm delivery was included as an adverse outcome in the study to possibly account for treatment paradox for delivery. The majority of the adverse outcomes predicted in this study by Thangaratinam et al[15] were preterm deliveries (61%), and no sensitivity analysis was reported for the performance of the model in predicting other adverse maternal outcomes excluding preterm delivery. In addition, observational studies have already shown that 50% of women with early-onset preeclampsia will deliver within 2 weeks and 25% within 4 weeks.[22-24] Therefore, it is possible that this model may not be useful for the prediction of maternal complications as iatrogenic delivery could be because of maternal or fetal indications or both. Another concern with the use of the PREP model is the inclusion of >14 variables, making it cumbersome compared with the fullPIERS model that requires only 6 variables. Model development studies have encouraged the use of a more parsimonious model because this reduces the chances of overfitting and enhances clinical utility.[17] Finally, the model in the study included treatment variables, such as antihypertensive and magnesium sulfate; the administration and timing of these treatments may vary based on the clinician’s training and experience. Therefore, we propose that the fullPIERS model might be better for identifying women with early-onset preeclampsia at highest risk of adverse maternal outcomes, regardless of treatment. It may, however, be worthwhile to test this latter hypothesis in a similar study.

Perspectives

The fullPIERS model was able to predict adverse maternal outcomes in women admitted with early-onset preeclampsia within 48 hours of admission and up to 7 days. Our findings could guide decision making especially the timing of delivery and planning of transfer to units for required care and administration of corticosteroid and magnesium sulfate. Thus, we think that the fullPIERS model could aid in averting severe maternal complications. We propose that women who fall in the highest risk category should be considered for delivery in settings where iatrogenic delivery can be instituted immediately and both the mother and newborn can be cared for, or at the least, close maternal and fetal surveillance. We recommend the use of the updated model (Equation 2) for management of women with early-onset preeclampsia to optimize performance. Future studies should consider dynamic modeling for risk reassessment.

Acknowledgments

We are grateful for the contribution made by the fullPIERS (Preeclampsia Integrated Estimate of Risk), PETRA (Preeclampsia Eclampsia Trial Amsterdam), and PREP (Prediction of Complications in Early-Onset Preeclampsia) data collectors and study sites investigators who retrieved the data for this study.

Sources of Funding

This study was supported by the Canadian Institutes of Health Research (operating grants). The sponsors of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Disclosures

None.