External validation and clinical usefulness of first-trimester prediction models for small- and large-for-gestational-age infants: a prospective cohort study.

OBJECTIVE: To assess the external validity of all published first-trimester prediction models based on routinely collected maternal predictors for the risk of small- and large-for-gestational-age (SGA and LGA) infants. Furthermore, the clinical potential of the best-performing models was evaluated.
DESIGN: Multicentre prospective cohort.
SETTING: Thirty-six midwifery practices and six hospitals (in the Netherlands). POPULATION: Pregnant women were recruited at <16 weeks of gestation between 1 July 2013 and 31 December 2015.
METHODS: Prediction models were systematically selected from the literature. Information on predictors was obtained by a web-based questionnaire. Birthweight centiles were corrected for gestational age, parity, fetal sex, and ethnicity. MAIN OUTCOME MEASURES: Predictive performance was assessed by means of discrimination (C-statistic) and calibration.
RESULTS: The validation cohort consisted of 2582 pregnant women. The outcomes of SGA <10th percentile and LGA >90th percentile occurred in 203 and 224 women, respectively. The C-statistics of the included models ranged from 0.52 to 0.64 for SGA (n = 6), and from 0.60 to 0.69 for LGA (n = 6). All models yielded higher C-statistics for more severe cases of SGA (<5th percentile) and LGA (>95th percentile). Initial calibration showed poor-to-moderate agreement between the predicted probabilities and the observed outcomes, but this improved substantially after recalibration.
CONCLUSION: The clinical relevance of the models is limited because of their moderate predictive performance, and because the definitions of SGA and LGA do not exclude constitutionally small or large infants. As most clinically relevant fetal growth deviations are related to 'vascular' or 'metabolic' factors, models predicting hypertensive disorders and gestational diabetes are likely to be more specific. TWEETABLE ABSTRACT: The clinical relevance of prediction models for the risk of small- and large-for-gestational-age is limited.

Introduction

Fetal growth deviations are associated with short‐ and long‐term health consequences for both mother and child. Delivering an infant that is large for gestational age (LGA) is associated with trauma to the birth canal, induction of labour, instrumental vaginal delivery, caesarean section, shoulder dystocia, and perinatal asphyxia.1, 2, 3, 4, 5 Infants born small for gestational age (SGA) are at increased risk of perinatal asphyxia, respiratory distress, intubation at term, sepsis, and mortality.4, 6, 7, 8, 9 Long‐term risks of infants born SGA or LGA are the development of obesity, hypertension, cardiovascular complications, and diabetes later in life.10, 11, 12, 13, 14, 15, 16, 17, 18

Fetal growth is determined by a complex interplay of genetic factors, uterine conditions, environmental factors, fetal syndromes, hormones, pregnancy complications, and maternal characteristics.17, 19, 20, 21 Risk factors for LGA are a high pregestational body mass index (BMI), pre‐existing diabetes mellitus, previous LGA, gestational diabetes mellitus (GDM), and a high BMI of the father.2, 22, 23, 24 Smoking, short maternal height, chronic hypertension, nulliparity, placental pathology, and intrauterine infections are associated with an increased risk of SGA.17, 25, 26 A number of these risk factors are modifiable, but others are not.

The early and correct identification of women at risk would enable personalized follow‐up management, which could help to avoid adverse perinatal outcomes. Prediction modelling combines risk factors into a single model that takes into account the risk‐dependent weight of each factor and possible interrelations.27, 28 Several prediction models based on maternal characteristics, biomarkers, and biophysical tests have been developed for the risk of SGA or LGA, showing promising discriminative performance in separating fetal growth deviations from normal growth. Biomarkers and biophysical tests may improve the accuracy of the model beyond using maternal characteristics alone. Published studies show only a limited contribution of these factors to improved discriminative performance, however.29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46 Moreover, most of these more complex predictors are relatively expensive, not readily available in general antenatal settings, and are possibly inconvenient for pregnant women.28 To our knowledge, no external validation studies of prediction models for SGA or LGA have been published so far. External validation is a crucial step before implementing a model in clinical practice, by evaluating the performance using data that were not used to develop the model.47

In this study, an overview of all published prediction models for the risk of SGA or LGA based on maternal characteristics and standard antenatal measurements (i.e. blood pressure) is provided. We validated the selected models in an independent Dutch prospective cohort consisting of 2582 pregnant women. Furthermore, we evaluated the clinical potential of the best‐performing models.

Methods

Selection of prediction models

We systematically searched PubMed to select all published early prediction models, based on routinely collected parameters and applicable in the first trimester of pregnancy, for the risk of SGA or LGA. The searches were performed in April 2013, before the development of the study questionnaires, and were updated until 22 June 2017. The search strategies and selection criteria have been published elsewhere.48

Validation cohort

We performed a multicentre prospective cohort study in the south‐eastern part of the Netherlands (Expect Study I). The primary objective of this study was to validate published first‐trimester prediction models for several adverse pregnancy outcomes. Six hospitals and 36 midwifery practices recruited pregnant women aged ≥18 years old at <16 weeks of gestation between 1 July 2013 and 1 January 2015, with follow‐up continuing until 31 December 2015. Eligible pregnant women were invited to complete two web‐based questionnaires (paper‐based questionnaires were available, upon request): one before 16 weeks of gestation (pregnancy questionnaire) and one 6 weeks after the due date (postpartum questionnaire). Medical records and discharge letters were requested from health care providers. Pregnancies ending in miscarriage (<16 weeks of gestation), terminations of pregnancy before 24 weeks of gestation, and women lost‐to‐follow‐up were excluded. For this study, we also excluded multiple pregnancies and women who delivered between 16+0 and 25+0 weeks of gestation, as the customised birthweight curves are only available from 25 weeks of gestation onwards.49 A detailed description of Expect Study I has been published in full elsewhere.48 Patients were involved in the development of the recruitment process and the study questionnaires. The design, results, and conclusions of this pilot study are described in the published study protocol.48 The study was funded by The Netherlands Organization for Health Research and Development (ZonMw grant 209020007).

Assessment of predictors and outcomes

The predictors in the included prediction models were assessed by means of the pregnancy questionnaire. Blood pressure was measured according to routine antenatal care and self‐reported in the pregnancy questionnaire. We used the same definitions as published in the original articles (Appendix S1; Tables 1 and 2).

Table 1

Baseline characteristics of the validation cohort (Expect Study I)

Characteristics	Missing values n (%)	Observed validation cohort (Expect Study I)a
		Overall (n = 2582)	SGA <10th percentile (n = 203)	No SGA (n = 2379)	LGA >90th percentile (n = 224)	No LGA (n = 2358)
Age, years	0 (0.0)	30.2 (3.9)	30.0 (4.4)	30.2 (3.9)	30.1 (3.8)	30.2 (3.9)
Ethnicity	0 (0.0)
White		2503 (96.9)	197 (97.0)	2306 (96.9)	218 (97.3)	2285 (96.9)
Afro‐Caribbean		2 (0.1)	1 (0.5)	1 (0.0)	0 (0.0)	2 (0.1)
South Asian		4 (0.2)	0 (0.0)	4 (0.2)	0 (0.0)	4 (0.2)
East Asian		16 (0.6)	1 (0.5)	15 (0.6)	0 (0.0)	16 (0.7)
Other Asian		0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)
Hispanic		11 (0.4)	0 (0.0)	11 (0.5)	1 (0.4)	10 (0.4)
Mixed		46 (1.8)	4 (2.0)	42 (1.8)	5 (2.2)	41 (1.7)
Tertiary education	3 (0.1)	1403 (54.3)	90 (44.3)	1313 (55.2)	134 (59.8)	1269 (53.8)
Height, cm	3 (0.1)	168.8 (6.4)	166.2 (6.2)	169.0 (6.4)	171.7 (6.4)	168.5 (6.3)
Weight, kg	4 (0.2)	68.9 (13.0)	65.2 (11.7)	69.2 (13.1)	75.4 (13.7)	68.3 (12.8)
Body mass index, kg/m 2	4 (0.2)	24.2 (4.3)	23.5 (3.7)	24.2 (4.3)	25.6 (4.7)	24.0 (4.2)
<18.5		87 (3.4)	8 (3.9)	79 (3.3)	2 (0.9)	85 (3.6)
18.5–24.9		1645 (63.7)	136 (67.0)	1509 (63.4)	120 (53.6)	1525 (64.7)
25.0–29.9		576 (22.3)	46 (22.7)	530 (22.3)	61 (27.2)	515 (21.8)
≥30.0		270 (10.4)	13 (6.4)	257 (10.8)	41 (18.3)	229 (9.7)
Smoking	1 (0.0)
Ever <16 weeks of gestation		312 (12.1)	54 (26.6)	258 (10.8)	25 (11.2)	287 (12.2)
Current (at completion questionnaire)		152 (5.9)	35 (17.2)	117 (4.9)	7 (3.1)	145 (6.1)
Diabetes mellitus	0 (0.0)	11 (0.4)	0 (0.0)	11 (0.5)	4 (1.8)	7 (0.3)
Type 1		9 (0.3)		9 (0.4)	3 (1.3)	6 (0.3)
Type 2		1 (0.0)		1 (0.0)	1 (0.4)	0 (0.0)
Other	0 (0.0)	1 (0.0)		1 (0.0)	0 (0.0)	1 (0.0)
History of chronic hypertension	0 (0.0)	27 (1.0)	0 (0.0)	27 (1.1)	4 (1.8)	23 (1.0)
Folate use at completion questionnaire	3 (0.1)	2198 (85.1)	165 (81.3)	2033 (85.5)	189 (84.4)	2009 (85.2)
Parity	0 (0.0)
Nulliparous		1311 (50.8)	103 (50.7)	1208 (50.8)	121 (54.0)	1190 (50.5)
Primiparous		1015 (39.3)	74 (36.5)	941 (39.6)	74 (33.0)	941 (39.9)
Multiparous		256 (9.9)	26 (12.8)	230 (9.6)	29 (12.9)	227 (9.6)
Conception	0 (0.0)
Spontaneous		2412 (93.4)	187 (92.1)	2225 (93.5)	207 (92.4)	2205 (93.5)
Ovulation induction		92 (3.6)	11 (5.4)	81 (3.4)	10 (4.5)	82 (3.5)
IVF/ICSI		78 (3.0)	5 (2.5)	73 (3.1)	7 (3.1)	71 (3.0)
Interpregnancy interval, months	11 (0.4)	29.0 (24.2)	32.0 (29.5)	28.7 (23.7)	25.8 (17.9)	29.3 (24.7)
History of SGA
<5th percentile	51 (2.0)	42 (1.6)	8 (3.9)	34 (1.4)	1 (0.4)	41 (1.7)
<10th percentile	51 (2.0)	106 (4.1)	18 (8.9)	88 (3.7)	1 (0.4)	105 (4.5)
History of LGA
>90th percentile	51 (2.0)	167 (6.5)	1 (0.5)	166 (7.0)	44 (19.6)	123 (5.2)
>95th percentile	51 (2.0)	89 (3.4)	0 (0.0)	89 (3.7)	30 (13.4)	59 (2.5)
Birthweight z‐score of previous pregnancy	49 (1.9)	0.15 (1.0)	−0.55 (0.7)	0.21 (1.0)	1.13 (1.0)	0.07 (0.9)
History of pregnancy induced hypertension	18 (0.7)	114 (4.4)	10 (4.9)	104 (4.4)	9 (4.0)	105 (4.5)
History of pre‐eclampsia	18 (0.7)	71 (2.7)	5 (2.5)	66 (2.8)	3 (1.3)	68 (2.9)
History of gestational diabetes mellitus	19 (0.7)	14 (0.5)	1 (0.5)	13 (0.5)	5 (2.2)	9 (0.4)
Systolic blood pressure, mmHg	257 (10.0)	114.4 (12.4)	115.0 (13.0)	114.3 (12.4)	116.3 (11.7)	114.2 (12.5)
Diastolic blood pressure, mmHg	267 (10.3)	67.6 (8.5)	67.6 (8.2)	67.6 (8.6)	68.6 (8.3)	67.5 (8.5)
Mean arterial pressure, mmHg	267 (10.3)	83.2 (8.8)	83.4 (8.5)	83.2 (8.8)	84.5 (8.4)	83.1 (8.8)

Abbreviations: ICSI, intracytoplasmic sperm injection; IVF, in vitro fertilization; LGA, large‐for‐gestational‐age; SGA, small for gestational age.

Original data (not imputed) presented as mean (SD) or absolute number (%).

Table 2

Discriminative performance of included prediction models for small‐for‐gestational‐age infants

Study (author, year)	AUROC (95% CI) Original publication	AUROC (95% CI) Validation cohort SGA <5th percentile (n = 104)	AUROC (95% CI) Validation cohort SGA <10th percentile (n = 203)	AUROC (95% CI) Validation cohort, nulliparous (n = 1311) SGA <10th percentile (n = 103)	AUROC (95% CI) Validation cohort, multiparous (n = 1271) SGA <10th percentile (n = 100)
González González (2017)33	0.615 (0.571–0.658) Internal validation: 0.594 (NR)	0.60 (0.55–0.65)	0.57 (0.53–0.61)	0.56 (0.52–0.59)	0.57 (0.53–0.61)
MacDonald‐Wallis (2015)57	0.70 (0.68–0.71)	0.67 (0.61–0.72)	0.63 (0.59–0.67)	0.66 (0.60–0.71)	0.67 (0.61–0.72)
Boucoiran (2013)29	0.62 (0.58–0.66)	0.63 (0.56–0.59)	0.64 (0.60–0.68)	0.64 (0.59–0.70)	0.64 (0.58–0.70)
Syngelaki (2011)58	NR	0.59 (0.54–0.65)	0.58 (0.54–0.62)	0.56 (0.50–0.61)	0.61 (0.55–0.67)
Poon (2011)42	0.719 (0.706–0.732)	0.52 (0.46–0.57)	0.52 (0.48–0.56)	0.52 (0.47–0.58)	0.52 (0.46–0.58)
Seed (2011)59	0.65 (NR) Internal validation: 0.57 (NR)	0.55 (0.50–0.61)	0.54 (0.50–0.58)	0.50 (0.45–0.56)	0.57 (0.51‐0.63)

AUROC, area under the receiving operating characteristic curve; CI, confidence interval; NR, not reported; SGA, small for gestational age.

The outcomes of SGA and LGA were defined as infants with a birthweight below the tenth percentile or above the 90th percentile, respectively, corrected for gestational age, ethnicity, gender, and parity.49 We also evaluated the performance of the models for SGA and LGA using the cut‐off values of the fifth percentile and the 95th percentile, respectively. Birthweight was obtained from the medical records. Data from the postpartum questionnaire were used in the case of a missing birthweight (n = 1) or absence of the medical record (n = 16).

Statistical analysis

There is no generally accepted rule for the required sample size for external validation of prediction models. We followed Vergouwe et al.,50 who recommend a minimum of 100 events and 100 non‐events.

The baseline characteristics of the validation cohort were described as mean ± standard deviation (SD) for continuous variables and as an absolute value with percentage for categorical variables. Stochastic regression imputation was used to enter missing predictor variables, with predictive mean matching used as the imputation model.51 We also evaluated the similarity of the validation cohort to the derivation cohorts.

We computed the individual probabilities for the risk of SGA or LGA using the original prediction algorithms (Appendix S1; Tables 3 and 4). The predictive performance of each model was assessed by means of discrimination and calibration. Discriminative performance, i.e. the ability of the model to distinguish between women who will have the outcome and those who will not, was quantified as the area under the receiver operating characteristic curve (AUROC) with 95% confidence interval (95% CI).47 A subgroup analysis was performed for nulliparous women, as a history of SGA or LGA is a strong predictor for recurrent SGA or LGA, respectively. Calibration is a measure of the agreement between the predicted probabilities of the model and the actual outcomes.47 We assessed calibration graphically by calibration plots, in which women were divided into groups of equal size (up to ten) with similar predicted risks, and computed calibration‐in‐the‐large and the calibration slope. Calibration‐in‐the‐large (intercept) indicates whether predictions are systematically too low (intercept >0 | slope = 1) or too high (intercept <0 | slope = 1) by comparing the mean predicted risk with the observed proportion of cases.47 The slope refers to the average strength of the predictor effects (overfitting, <1; underfitting, >1). Calibration plots that indicate perfect agreement have an intercept of 0 and a slope of 1 (45° line).47 We recalibrated the models by adjusting the intercept and slope using the linear predictor as the only covariate. This recalibration method has no influence on the discriminative performance.52

Table 3

Discriminative performance of included prediction models for large‐for‐gestational‐age infants

Study (author, year)	AUROC (95% CI) Original publication	AUROC (95% CI) Validation cohort LGA >95th percentile (n = 105)	AUROC (95% CI) Validation cohort LGA >90th percentile (n = 224)	AUROC (95% CI) Validation cohort, nulliparous (n = 1311) LGA >90th percentile (n = 121)	AUROC (95% CI) Validation cohort, multiparous (n = 1271) LGA >90th percentile (n = 103)
Frick (2016)31	NR	0.74 (0.70–0.79)	0.69 (0.66–0.72)	0.66 (0.61–0.71)	0.80 (0.76–0.84)
González González (2013)32	0.680 (0.659–0.700)	0.67 (0.62–0.72)	0.64 (0.60–0.68)	0.67 (0.62–0.72)	0.70 (0.64–0.74)
Plasencia (2012)40	0.705 (0.684–0.725)	0.64 (0.59–0.70)	0.62 (0.58–0.65)	0.66 (0.61–0.72)	0.70 (0.65–0.75)
Syngelaki (2011)58	NR	0.64 (0.58–0.70)	0.60 (0.56–0.64)	0.58 (0.53–0.63)	0.71 (0.66–0.77)
Nanda (2011)37	0.722 (0.710–0.735)	0.73 (0.68–0.78)	0.68 (0.64–0.71)	0.64 (0.59–0.69)	0.73 (0.69–0.78)
Poon (2011)42	0.715 (0.710–0.719)	0.73 (0.68–0.78)	0.68 (0.64–0.71)	0.64 (0.59–0.69)	0.74 (0.70–0.79)

AUROC, area under the receiving operating characteristic curve; CI, confidence interval; LGA, large for gestational age; NR, not reported.

Table 4

Performance measures at different risk thresholds for the recalibrated model from Boucoiran (2013),29 predicting the risk of small‐for‐gestational‐age infants

Risk thresholda, %	High risk, % (n/n)	Sensitivity, % (n/n)	Specificity, % (n/n)	PPV, % (n/n)	NPV, % (n/n)
2	99.6 (2571/2582)	100 (203/203)	0.46 (11/2379)	7.9 (203/2571)	100 (11/11)
4	92.1 (2377/2582)	97.0 (197/203)	8.4 (199/2379)	8.3 (197/2377)	97.1 (199/205)
6	70.1 (1811/2582)	82.3 (167/203)	30.9 (735/2379)	9.2 (167/1811)	95.3 (735/771)
8	32.1 (829/2582)	52.7 (107/203)	69.7 (1657/2379)	12.9 (107/829)	94.5 (1657/1753)
10	9.7 (250/2582)	25.1 (51/203)	91.6 (2180/2379)	20.4 (51/250)	93.5 (2180/2332)
12	5.9 (153/2582)	17.2 (35/203)	95.0 (2261/2379)	22.9 (35/153)	93.1 (2261/2429)
14	5.5 (143/2582)	15.8 (32/203)	95.3 (2268/2379)	22.4 (32/143)	93.0 (2268/2439)

NPV, negative predictive value; PPV, positive predictive value.

Predicted risk at or above this level was considered as high risk.

Lastly, we evaluated the clinical potential of the best‐performing models by means of decision curve analysis. Decision curve analysis provides insight into the net benefit of the prediction model over a range of risk thresholds compared with the scenarios that all (‘treat all’) or no (‘treat none’) women are at high risk of the outcome.53 The net benefit of a model can be clinically interpreted as the net increase in the proportion of appropriately treated patients or as the net decrease in the proportion of patients treated unnecessarily (proportion of true positives and false positives).54 Next, we calculated sensitivity, specificity, and positive and negative predictive values at certain risk thresholds for the models with the highest overall net benefit.

Statistical analyses were performed with r 3.4.1 (using the packages mice, rms, proc, and decisioncurve).

Results

The search strategies identified 1522 and 334 articles for SGA and LGA, respectively. Fifteen articles fulfilled the eligibility criteria for the outcome SGA. The cross‐checking of citation lists yielded three additional articles. We excluded ten articles for the following reasons: algorithm not available (n = 8),35, 38, 39, 40, 43, 45, 46, 55 predictors not applicable in a high‐income country (n = 1),56 and model already published in another included article (n = 1).34 The eight eligible articles described nine prediction models aimed at predicting any SGA (n = 6),29, 33, 44, 57, 58, 59 preterm SGA (n = 2),60, 61 and late SGA (n = 1).60 For the outcome LGA, we selected nine eligible articles all describing a prediction model for any LGA.31, 32, 37, 39, 40, 41, 42, 58, 62 No additional articles were found by cross‐checking references. Three articles were excluded as the algorithm was not available.39, 40, 62

We only validated models predicting the risk of any SGA or any LGA, as the number of preterm SGA infants was too low in our validation cohort (n = 6 at <37 weeks of gestation). None of the models were used in antenatal care during the study period.

The characteristics of the included models for SGA (n = 6),29, 33, 44, 57, 58, 59 or for LGA (n = 6),31, 32, 37, 41, 42, 58 are summarized in Tables S1 and S2, respectively. The models for SGA were published by five different research groups from the UK, Canada, and Spain between 2011 and 2017. Four models originally defined SGA as an infant with a birthweight below the tenth percentile, and two models used a cut‐off value of below the fifth percentile. Two studies used the same study data to determine the birthweight centiles. The other studies used national charts (n = 1), their own study data (n = 1), or a customised birthweight calculator (n = 1). Three different research groups from the UK and Spain published models predicting the risk of LGA between 2011 and 2016. Four models defined LGA as a birthweight above the 95th percentile, based on the same study data, and two models used a birthweight above the 90th percentile, based on national charts.

We included 2582 women in the validation cohort (Figure S1). The outcome of SGA with a birthweight below the tenth percentile occurred in 203 women (7.9%). Six SGA infants were born prematurely (<37 weeks of gestation) (3.0%), and 14 SGA infants were born to mothers whose pregnancy was complicated by a hypertensive pregnancy disorder (6.9%). Of the 224 infants who were LGA, with birthweights above the 90th percentile (8.7%), 20 were born to mothers with GDM (8.9%). Table 1 shows the characteristics of the overall cohort, and for SGA, non‐SGA, LGA, and non‐LGA groups in the observed data. The characteristics of the imputed validation cohort were generally comparable with those of the observed data (Table S3). We also compared the characteristics of the validation cohort with the derivation cohorts (Appendix S2; Tables 1 and 2). In contrast to most derivation cohorts, our validation cohort had a low prevalence of non‐white ethnicity and smoking during pregnancy. The average height and weight of the women was higher compared with all other development cohorts, but the mean BMI was similar. The occurrence of the outcome SGA was considerably higher in the derivation cohorts of Seed et al. (high‐risk women), 59 and of González González et al.33 The prevalence of LGA was comparable between the development cohorts and our validation cohort. Compared with all other derivation cohorts, nulliparous women in our cohort delivered LGA infants more often than not. Syngelaki et al.58 neither reported predictor characteristics nor reported the numbers of SGA and LGA infants.

Predictive performances

Table 2 shows the AUROCs for the prediction of SGA. The discriminative performance decreased considerably for all models compared with the development cohorts, but decreased most for the models with the highest AUROCs. The model of Boucoiran et al. retained the highest AUROC (0.64, 95% CI 0.60–0.68).29 All models demonstrated a higher ability to predict the risk of SGA below the fifth percentile compared with SGA below the tenth percentile, with AUROCs of up to 0.67; however, the 95% CIs were wider because of the lower sample size, with lower limits close to 0.50. Subgroup analysis showed no difference in the discriminative ability of the models between nulliparous and multiparous women. The ROC curves are displayed in Figure S2. The three models for which the full algorithm was provided showed poor calibration (Figure 1). The recalibration of all models led to better agreement between the predicted and the observed risks of most models (Figure S3). The model of MacDonald‐Wallis et al.57 showed the closest fit to the perfect calibration line. The predicted risks of all models were closely clustered around the overall risk.

Figure 1

Calibration plots of externally validated first‐trimester prediction models for small‐for‐gestational‐age infants with birthweights below the tenth percentile. The grey line is the reference line with intercept = 0 and slope = 1 (perfect calibration). Triangles correspond to grouped predicted risks with 95% confidence intervals (vertical lines).

The discriminative performances of the prediction models for LGA are presented in Table 3. Although the AUROCs also decreased for all models after external validation, three models showed moderate discriminative ability (AUROCs 0.68–0.69). The model of Frick et al. showed the highest discriminative performance with an AUROC of 0.69 (95% CI 0.66–0.72) and 0.74 (95% CI 0.70–0.79) for LGA >90th percentile and >95th percentile, respectively.31 All models showed a higher discriminative ability for LGA >95th percentile compared with LGA >90th percentile. In contrast to the outcome SGA, most models for LGA were also originally developed to predict the 5% of most extreme birthweight deviations (>95th percentile). Figure S4 presents the ROC curves. Subgroup analysis showed better discriminative performance among multiparous women, with the highest AUROC (0.80) for the model of Frick et al.31 Performance among nulliparous women was slightly lower than in the total group (AUROC up to 0.67). The three fully available algorithms for LGA showed better calibration as compared with models for SGA (Figure 2). All models overestimated the probabilities on average (intercept < 0) and showed an overfitting of the predictor effects (slope < 1, low predictions too low, and high predictions too high). Recalibration of all models considerably improved the agreement between the predicted probabilities and the observed outcomes for almost all models (Figure S5).

Figure 2

Calibration plots of externally validated first‐trimester prediction models for large‐for‐gestational‐age infants with birthweights above the 90th percentile. The grey line is the reference line with intercept = 0 and slope = 1 (perfect calibration). Triangles correspond to grouped predicted risks with 95% confidence intervals (vertical lines).

Clinical usefulness

Decision curve analysis of the two best‐performing models for SGA revealed a positive net benefit compared with classifying all or no women as being at high risk of SGA for a risk threshold of 4–22% (Figure 3); however, the overall net benefit remained low throughout this range. This low clinical usefulness is also demonstrated in Table 4. Choosing a low cut‐off is associated with a high sensitivity, so the risk of missing women with the outcome is minimized (low rate of false negatives). High sensitivity leads to a large proportion of women unnecessarily indicated as being at high risk, however. A higher risk threshold ensures a high specificity (low rate of false positives), but only a small proportion of women with the outcome will be detected.

Figure 3

Decision curve analysis of the two best‐performing models for the risk of small‐for‐gestational‐age infants with birthweights below the tenth percentile. The solid grey line is the net benefit when considering all women as being at high risk, and the horizontal black line is the net benefit when considering no women being at high risk.

Figure 4 shows the net benefit of the three best discriminative models for the risk of LGA. The models were beneficial compared with classifying all or no women as being at high risk over a threshold range of 1–40%. The three curves differed only slightly. Table 5 presents the performance measures at different risk thresholds.

Figure 4

Decision curve analysis of the three best‐performing models for the risk of large‐for‐gestational‐age infants with birthweights above the 90th percentile. The solid grey line is the net benefit when considering all women as being at high risk, and the horizontal black line is the net benefit when considering no women being at high risk.

Table 5

Performance measures at different risk thresholds for recalibrated model from Frick et al.,31 predicting the risk of large‐for‐gestational‐age

Risk thresholda, %	High risk, % (n/n)	Sensitivity, % (n/n)	Specificity, % (n/n)	PPV, % (n/n)	NPV, % (n/n)
1	98.5 (2541/2582)	100 (224/224)	1.7 (41/2358)	8.8 (224/2541)	100 (41/41)
2	89.7 (2424/2582)	99.6 (223/224)	11.3 (266/2358)	9.6 (223/2424)	99.6 (266/267)
4	72.7 (1877/2582)	92.9 (208/224)	29.2 (689/2358)	11.1 (208/1877)	97.7 (689/705)
8	50.7 (1309/2582)	73.2 (164/224)	51.4 (1213/2358)	12.5 (154/1309)	95.3 (1213/1273)
10	37.8 (976/2582)	59.4 (133/224)	64.2 (1515/2358)	13.6 (133/976)	94.3 (1515/1606)
14	17.0 (438/2582)	34.8 (78/224)	84.7 (1998/2358)	17.8 (78/438)	93.2 (1998/2144)
18	5.7 (148/2582)	17.4 (39/224)	95.4 (2249/2358)	26.4 (39/148)	92.4 (2249/2434)
20	2.7 (69/2582)	10.7 (24/224)	98.1 (2313/2358)	34.8 (24/69)	92.0 (2313/2513)

NPV, negative predictive value; PPV, positive predictive value.

Predicted risk at or above this level was considered as high risk.

Discussion

Main findings

Six early non‐invasive prediction models for the risk of SGA as well as six models for the outcome LGA were selected from the literature, some of which showed promising original discriminative performance (AUROC up to 0.72). We validated these models in an independent prospective cohort of 2582 women. The discriminative performance decreased for all models, especially those predicting SGA. All models showed better discriminative ability for predicting the more severe cases of SGA and LGA, which are also associated with a higher risk of adverse outcomes.31, 63 Calibration was poor for the prediction models for SGA. The models predicting the risk of LGA all overestimated the risk in our population. Recalibration provided better agreement between the predicted risks and the actual risks for most models. The predictive performance is usually lower in other populations, even when a similar population as the one in which the model was developed is being used.47 The studies used different sources to define the birthweight centile (i.e. customised and population‐based charts) that may have contributed to the different performance in our population. Evaluation of the predictive performance of all available models in one independent cohort allowed for a fair comparison.

Interpretation

The validation of promising prediction models is essential in order to gain insight into their robustness across other populations. Only two of the selected models were internally validated and their performance stayed fairly stable after external validation.33, 59 MacDonald‐Wallis et al. validated their developed model in another cohort from the same country.57 To our knowledge, no independent or other external validation studies have been published on prediction models for SGA or LGA. Validating prediction models in an independent population provides insight into the generalisability of the model, an essential element before clinical application can be considered.28 Further research should focus on validation and the updating of existing models instead of investing energy in developing yet another prediction model similar to those already available.64

Predictive performance measures of a prediction model do not coincide with the usefulness of the model in clinical practice. Decision curve analysis and performance parameters at different risk thresholds give a first impression of the clinical utility. A prediction tool should ideally separate individuals who will develop the disease from those who will not. But in fact, there is a trade‐off between sensitivity and specificity. Evaluation of the clinical utility is therefore dependent on several other factors, such as the severity of the health consequences related to fetal growth deviations and the availability of effective follow‐up management, to prevent either the development of fetal growth deviations (primary) or the related adverse effects (secondary).

The heterogeneous aetiology of fetal growth deviations makes prediction difficult.24, 65 Infants who are constitutionally SGA or LGA are less likely to develop adverse outcomes and also less likely to benefit from interventions.65, 66 A subset of possibly clinically relevant SGA and LGA frequently has a ‘metabolic’ (i.e. high BMI or GDM) or ‘vascular’ (i.e., hypertensive disorder) origin. The predictors in the included models for SGA and LGA overlap considerably with those of models predicting hypertensive pregnancy disorder and GDM, respectively.67, 68 Although most SGA and LGA infants are born to mothers without a hypertensive pregnancy disorder or GDM, respectively, the conditions share common pathophysiological aspects.24, 65

Regarding primary prevention strategies, recent meta‐analyses have demonstrated that aspirin modestly reduces the risk of delivering an SGA infant in women at high risk, with most benefit derived by starting treatment before 16 weeks of gestation and using a dose of ≥100 mg (risk ratio 0.56–0.76).69, 70 Patient selection of those at increased risk was primarily based on an increased risk of developing a hypertensive pregnancy disorder rather than delivering an SGA infant.71 Currently, there are no effective interventions for the primary prevention of LGA available, except for the treatment of women with GDM that indirectly lowers the risk of LGA, such as diet.24 In summary, the application of the currently available prediction models for the risk of SGA or LGA, in settings in which models for the identification of ‘vascular’ (pre‐eclampsia)‐ and ‘metabolic’ (GDM)‐related complications are already applied, are not likely to result in additional benefit regarding the overlap of predictors and preventive interventions.

The identification of women at risk may also allow for the secondary prevention of adverse effects related to SGA and LGA. Antenatal detection of infants born SGA and delivery at the appropriate time may reduce the risk of severe morbidity and mortality.72, 73, 74, 75 Induction of labour at or near term for pregnancies suspected to deliver a LGA infant results in a lower mean birthweight, and fewer birth fractures and shoulder dystocia.76, 77 In most clinical settings, ultrasound fetal biometry is the current method for the prediction of birthweight. Based on the decision curve analysis, the use of prediction models to select women for ultrasound fetal biometry will probably not be any more beneficial compared with providing this for all women. Moreover, should ultrasound fetal biometry be restricted to high‐risk women, it is again clinically relevant that the model selects the pathological fetal growth deviations. Another important aspect is that even infants who do not meet the criteria for SGA or LGA can have a pathological growth pattern, such as asymmetrical growth or a declining or accelerated growth pattern.66 These pathological growth patterns are also likely to be related to ‘vascular’ and ‘metabolic’ complicated pregnancies, and serial ultrasound fetal biometry is needed for detection. In conclusion, models that would predict pathological fetal growth deviations are more likely to improve clinical outcomes than models predicting SGA or LGA.

Strengths and limitations

We externally validated all published non‐invasive prediction models in an independent population. The multicentre prospective study design, with no strict inclusion criteria, ensured the sample was as heterogeneous as possible. Our data contained a low level of missing data (<1% for most predictors) and out‐of‐range values, as we incorporated validation checks in the web‐based questionnaires. Missing data were handled by the use of imputation in order to prevent biased results. Nevertheless, a substantial number of blood pressure measurements were missing (10%), most likely because of the self‐reporting of measurements in the web‐based questionnaire. Only two models contained a predictor based on blood pressure measurement. Another limitation to be mentioned is that we had to exclude women who delivered between 16+0 and 24+6 weeks of gestation (n = 8), as the Dutch population‐based reference curves for birthweight centiles are available from 25 weeks of gestation onwards.49 Lastly, we had to exclude two prediction models in the selection process, as we did not dispose of routine blood parameters (random glucose, rhesus group) and ultrasound measurements (crown–rump length).36, 78

Conclusion

The clinical relevance of prediction models for SGA and LGA can be questioned, both for the moderate predictive performance and the heterogeneous aetiology of fetal growth deviations. It is important to distinguish between constitutional and pathological fetal growth deviations to improve clinical outcomes. Not much additional clinical benefit is expected of the current prediction models for SGA and LGA over models that predict pre‐eclampsia and GDM, due to overlap of predictors and available treatment strategies.

Disclosure of interests

None declared. Completed disclosure of interests form available to view online as supporting information.

Contribution to authorship

The Expect Study I was designed by LS and MS. LM elaborated and carried out Expect Study I under the supervision of LS and HS. LM conducted the analyses, interpreted the data, and drafted the manuscript. LS and HS contributed to the interpretation of the outcomes and critically reviewed draft versions. SvK contributed to the imputation of the data and critically reviewed draft versions. RA, IvD, JL, IZ, and MS collaborated in data collection and critically reviewed draft versions. All authors gave approval of the final version of the manuscript.

Details of ethics approval

The Medical Ethics Committee (MEC) of the Maastricht University Medical Centre evaluated the study protocol and declared that the study did not fall within the scope of the Dutch Medical Research Involving Human Subjects Act (WMO) (MEC 13‐4‐053). All participating women gave informed consent through the Internet. The study was registered at The Netherlands Trial Registry on 21 August 2013 (NTR4143, www.trialregister.nl).

Funding

The Expect Study I was funded by The Netherlands Organization for Health Research and Development, Pregnancy and Childbirth program (ZonMw grant 209020007). The funding organization had no role in the design and conduct of the study, analysis or interpretation of data, decision to publish, or preparation of the manuscript.

Figure S1. Flowchart of validation cohort fetal growth deviations.

Click here for additional data file.

Figure S2. Receiver operating characteristic (ROC) curves of externally validated first‐trimester prediction models for small‐for‐gestational‐age infants with birthweights below the fifth and the tenth percentiles.

Click here for additional data file.

Figure S3. Calibration plots of recalibrated first‐trimester prediction models for small‐for‐gestational‐age infants with birthweights below the tenth percentile.

Click here for additional data file.

Figure S4. Receiver operating characteristic (ROC) curves of externally validated first‐trimester prediction models for large‐for‐gestational‐age infants with birthweights above the 90th and 95th percentiles.

Click here for additional data file.

Figure S5. Calibration plots of recalibrated first‐trimester prediction models for large‐for‐gestational‐age >90th percentile.

Click here for additional data file.

Table S1. Characteristics included prediction models for small‐for‐gestational‐age.

Click here for additional data file.

Table S2. Characteristics included prediction models for large‐for‐gestational‐age.

Click here for additional data file.

Table S3. Characteristics of observed and imputed validation cohort.

Click here for additional data file.

Appendix S1. Predictor assessment and model algorithms.

Click here for additional data file.

Appendix S2. Characteristics original cohorts and validation cohort.■

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.