Risk assessment of preterm birth through identification and stratification of pregnancies using a real-time scoring algorithm.

INTRODUCTION: Preterm birth poses a significant challenge. This study evaluated a real-time scoring algorithm to identify and stratify pregnancies to indicate preterm birth.
METHODS: All claims data of pregnant women were reviewed between 1 January 2014 and 31 October 2018 in Kentucky.
RESULTS: A total of 29,166 unique women who were matched to a live newborn were documented, with the pregnancy identified during the first trimester in 54.1% of women. Negative predictive values, sensitivity, and positive likelihood ratios increased from the first to third trimesters as pregnant women who were matched to a live newborn had more visits with their physicians. The area under the receiving-operating characteristics curve on test data classifying preterm birth was 0.59 for pregnancies identified during the first trimester, 0.62 for pregnancies identified in the second trimester, and 0.73 for pregnancies identified in the third trimester.
CONCLUSIONS: This study presents a real-time scoring algorithm of indicating preterm birth in the first trimester of gestation which permits stratification of pregnancies to provide more efficient early care management.

Introduction

Preterm birth refers to a delivery that occurs before 37 weeks of gestation and is the leading cause of neonatal death in the United States.[1] More than 380,000 infants are born preterm in the United States annually. Infants born before 32 weeks of gestation have a higher risk of mortality and morbidity, including intraventricular hemorrhage, necrotizing enterocolitis, respiratory distress syndrome, feeding difficulties, neurological deficits, developmental delay, and vision and hearing problems.[1,2] Preterm birth is often associated with a neonate’s admission to the intensive care unit and re-admission to the hospital in the first year of life.[3] In 2013, employer-sponsored health plans in the United States paid at least US $6 billion in addition for infants born preterm, with a substantial portion attributed to infants with major birth defects.[4]

Factors associated with a high risk of preterm birth include women with a previous preterm birth, a history of cervical surgery, and a short cervix (<25 mm) on routine ultrasound.[2,5-7] While testing of the vaginal posterior fornix for fetal fibronectin has been reported as an independent predictor of preterm birth,[5,7] it only yields meaningful positive tests after 22 weeks gestation.[2,6,8] Several studies have investigated the use of risk scores to indicate preterm birth; however, these data are generally poor quality, inaccurate, and lack clinically significant reference standards.[9] Furthermore, risk scores are often assigned to pregnant women after 22 weeks of gestation when it is usually too late for meaningful interventions.[5,10]

In this study, we present a unique algorithm of scoring pregnant women beginning in early gestation and continuing through pregnancy using risk factors of preterm birth. Our objective was to develop an algorithm that effectively targeted high-risk pregnant women, successfully managed resources, and better assisted care managers. We also discuss the benefits of our real-time scoring system compared to previous methods.

Methods

Data source and preparation

Under an Institutional Review Board-approved protocol, we retrospectively evaluated all claims data of pregnant women over a 4-year period (1 January 2014–31 October 2018) in a metropolitan community in Kentucky. The claims data were acquired from Passport Health Plan, Kentucky’s only non-profit, community-based healthcare provider that administers Kentucky Medicaid benefits. These data are rich with information and have been standardized. Furthermore, it tracks women prior to, during, and between pregnancies. The raw data were reformatted for pregnancy identification, risk ratio calculation, scoring algorithm determination, risk stratification, and quality assurance. Receiver operating characteristic (ROC) curves for each trimester of pregnancy assessed the efficacy of our scoring algorithm.

As our study focused on observational data, our sample size comprised the entire available population who qualified for the inclusion criteria set forth in our study. Since the acquired claims data set included the entire history of encounters and not just encounters related to pregnancy, our inclusion criteria initially began with the process of identification. Identification is how we grouped relevant encounters together to form a single, unique pregnancy. During this process, we determined which women were pregnant, the approximate week of gestation, whether they were pregnant on more than one occasion during the study period, and the outcome of the pregnancy. This is a proprietary, rules-based approach to extract (i.e. identify) pregnancies from a claims data set. Our study consisted of women within the data set of total identified pregnancies who could be matched to the delivery of a live infant. The exclusion criteria comprised women within the data set of total identified pregnancies whose pregnancy was lost or who could not be matched to an infant at birth. An inability to match a pregnancy with a newborn delivery may have been due to different health plans for the woman and infant.

Development of preterm birth risk assessment algorithm

Based on previous literature, we determined 71 risk factors of preterm birth that constituted six groups: anatomical, behavioral, demographic, disease, historical, and environmental (Table 1).[5,10-14] The claims data also included laboratory findings, the electronic medical record (EMR), and pharmacy visits. A certified professional coder mapped out the International Classification of Diseases, ninth revision and tenth revision (ICD-9 and ICD-10) codes with the 71 risk factors. A scoring algorithm was employed for risk ratio calculation that weighed the preterm birth risk of a particular risk factor from data claims outcomes. The risk factors were derived from the pregnant woman’s data, and the outcomes were obtained from the infant’s data.

Table 1.

71 risk factors of preterm birth.

Alcohol useCerclage procedureCervical incompetenceCervical shorteningDepressionDomestic violenceDrug use high riskDrug useFetal fibronectin procedureGenitourinary infectionsGestational hypertensionGingivitis/periodontal diseaseHomelessnessInfections (non-STDs)Insufficient prenatal careSystemic lupus erythematosusNutritional deficiencyMyasthenia gravisNon-gynecologic cancerPhysical/emotional health issuesPre-eclampsiaPre-existing diabetes mellitusPre-existing hypertensionHistory of prior post-term delivery

History of prior preterm deliverySleep disorderSTDTeen pregnancyThrombophiliaThyroid diseaseTobacco/nicotine usePre-eclampsia high riskSTD high riskGenitourinary infections high riskCardiac diseaseChronic renal diseaseInadequate birth spacingEating disorderGestational diabetes mellitusGynecologic cancerLow pre-pregnancy BMILow weight gain during pregnancySexual abuseFetal abnormalityEthnicityHemorrhageIUGRLow socioeconomic status

Multi-fetal pregnancyObesityPlacenta previaUterine cervix anomaliesPesticide exposureAnemiaAnxietyGroup B streptococcusIrritable bowel syndromeInaccessibility to healthcareUnwanted pregnancySpottingPreterm labor without deliveryHistory of miscarriageZika virusMental disorderElderly pregnancyHistory of preterm laborHistory of premature rupture of membranesEducation/literacy problemsProblems related to employment/unemploymentCondition complicating pregnancyHigh-risk pregnancy

STD: sexually transmitted disease; BMI: body mass index; IUGR: intrauterine growth restriction.

The risk ratios were calculated by evaluating the frequencies of preterm birth for each risk factor in Passport Health Plan’s historical claims data for women who were pregnant and delivered. The risk ratio calculations evaluated pregnant women who did or did not trigger a risk factor and whether the birth outcome was preterm or not. Deriving risk ratios from within Passport Health Plan’s population of pregnant women rather than using regional averages was more representative of the population’s general socioeconomic profile.

A pregnant woman’s aggregate risk of preterm birth across the pregnancy was calculated, with the relative risk weight converted into 0–100 score by a standard link function. As our scoring algorithm was a clinical tool, we determined how a pregnant woman scored at a particular time during gestation irrespective of future information. The relative risk score was recalculated in real time as a pregnant woman’s daily data feeds were incorporated. This daily update permitted refinement of risk assessment accuracy as the pregnant woman visited her physician more frequently as gestation progressed. Certain risk factors in our scoring algorithm, such as cervical shortening and gestational diabetes, were unable to be determined until later than the first trimester. Pregnant women who experienced these factors in a previous pregnancy were at risk for these factors in subsequent pregnancies and were included in their score. Each pregnant woman’s score with all of the risk factors that were utilized to calculate the score were given to the nurse case managers at Passport. Based on the particular risk factors, the nurse case managers scheduled referrals to mental health providers, encouraged smoking cessation, educated pregnant women about the risk of diabetes mellitus to themselves and their fetuses, and encouraged adherence to prenatal medications.

Our scoring algorithm had different lookback periods for each of the 71 risk factors and required a claims history to permit a thorough exploration into risk factors of preterm birth. The lookback period refers to the investigation into a pregnant woman’s medical history starting at the Estimated Conception Date to determine clinically relevant information that may impact her risk of preterm birth. Each pregnant woman was assigned a risk score in real time through the scoring algorithm based on her cumulative risk factors.

As case management resources for pregnant women were limited, optimization was attained by placing pregnant women who were most at risk of preterm birth at the top of the care management clinical queue. The risk scoring process stratified pregnant women according to their level of risk of preterm birth.

Validation of our real-time scoring algorithm and statistical analysis

A randomized and stratified split of the data was performed on the 29,166 women who were matched to live newborns for validating. The training set contained 80% of the original data, and the test set consisted of 20% of the remaining data, reflecting 23,333 and 5833 pregnancies, respectively. The stratification was performed along two dimensions: preterm birth rate and trimester identification. The preterm birth rate was the same for both the training and test data, and the trimester identification was almost identical for both the training and test data.

The algorithm was trained on the training data set and applied to the test data set.

The statistical analysis to determine the sensitivity, specificity, positive and negative predictive values, likelihood ratios, and thresholds for high- and low-risks for preterm birth was calculated using the Youden J statistic (or Youden index).[15] This analysis was performed utilizing Stata software (STATACORP LLC; College Station, TX). To define the optimal threshold, Youden’s index calculated the sensitivity and specificity for each point on the ROC curve. All integer values between 0 and 100 determined the risk score. From the Youden index, the J statistic (sensitivity + specificity − 1) was calculated. The optimal cut-off was the risk score where the J statistic was maximum. After the optimal cut-off was determined on the training data set, it was used for the optimal cut-off on the test data set. The high- and low-risk thresholds were set for optimal balance between the true and false positive rates on the training data and then applied to the test data.

Results

Demographics and identification of pregnant women who were matched to their infants

A total of 29,166 unique pregnancies (pregnant women who were matched to their infants) were documented over the 4-year period of this study (Table 2). A total of 15,767 (54.1%) pregnancies were identified during the first trimester, 7148 (24.5%) in the second trimester, and 6251 (21.4%) in the third trimester. There was a total of 25,867 unique women, thus, 3299 women had multiple pregnancies during the study period. The mean age of pregnant women at delivery was 26.2 years (range: 13–48 years).

Table 2.

Demographics of pregnant women who were matched to a live newborn in our study (1 January 2014–31 October 2018).

Features	Category	Number of pregnant women
Age at delivery		Mean age: 26.2 years (range: 13–48 years)
Unique pregnancies (pregnant women matched to their infants)	Total	29,166
	First trimester	15,767
	Second trimester	7148
	Third trimester	6251
Unique women		25,867 (3299 women had multiple pregnancies during the study period)

High-risk scoring efficacy of preterm birth

Since our data were split in a stratified manner, the preterm birth rate was 9.9% for both the training and test data sets. The scoring effectiveness of preterm birth in pregnant women throughout gestation on the training and test sets is presented in Tables 3 and 4, respectively. Strong negative predictive values across all three trimesters were consistently observed, suggesting that our clinical tool was effective in identifying preterm birth. The negative predictive values increased (89.6%–97.3% on the training set and 90.0%–96.4% on the test set) with the duration of pregnancy as pregnant women had more visits with their physicians, thereby, providing additional data for the scoring algorithm to process.

Table 3.

High-risk scoring efficacy of preterm birth on the training data set using our scoring algorithm.

Trimester 1
	Preterm birth	Term birth
High risk (⩾81)	681	3114	PPV: 17.9%	LR+: 1.51(95% CI: 1.51–1.52)
Low-risk (<81)	918	7952	NPV: 89.6%	LR−: 0.80(95% CI: 0.80–0.80)
	Sensitivity: 42.6%	Specificity: 71.9%
Trimester 2
	Preterm birth	Term birth
High risk (⩾76)	212	1329	PPV: 13.8%	LR+: 1.79(95% CI: 1.79–1.80)
Low risk (<76)	257	3841	NPV: 93.9%	LR−: 0.73(95% CI: 0.73–0.73)
	Sensitivity: 45.2%	Specificity: 74.7%
Trimester 3
	Preterm birth	Term birth
High risk (⩾78)	138	829	PPV: 14.2%	LR+: 3.18(95% CI: 3.17–3.2)
Low risk (<78)	107	3857	NPV: 97.3%	LR−: 0.53(95% CI: 0.53–0.53)
	Sensitivity: 56.3%	Specificity: 82.3%

PPV: positive predictive value; NPV: negative predictive value; LR: likelihood ratio; CI: confidence interval.

Table 4.

High-risk scoring efficacy of preterm birth on the test data set using our scoring algorithm.

Trimester 1
	Preterm birth	Term birth
High risk (⩾81)	177	764	PPV: 18.8%	LR+: 1.60(95% CI: 1.59–1.60)
Low risk (<81)	216	1945	NPV: 90.0%	LR−: 0.77(95% CI: 0.76–0.77)
	Sensitivity: 45.0%	Specificity: 71.8%
Trimester 2
	Preterm birth	Term birth
High risk (⩾76)	52	323	PPV: 13.9%	LR+: 1.67(95% CI: 1.66–1.68)
Low risk (<76)	72	963	NPV: 93.0%	LR−: 0.78(95% CI: 0.77-0.78)
	Sensitivity: 42.0%	Specificity: 74.9%
Trimester 3
	Preterm birth	Term birth
High risk (⩾78)	43	218	PPV: 16.8%	LR+: 3.02(95% CI: 3.00–3.04)
Low risk (<78)	38	1022	NPV: 96.4%	LR−: 0.57(95% CI: 0.57–0.57)
	Sensitivity: 53.1%	Specificity: 82.4%

PPV: positive predictive value; NPV: negative predictive value; LR: likelihood ratio.

For the training data, the sensitivity and specificity were lowest in the first trimester and increased throughout gestation (42.6%–56.3% and 71.9%–82.3%, respectively) as the scoring algorithm gathered more information about the pregnant women. The same trend was observed on the test data (45.0%–53.1% and 71.8%–82.4%, respectively). The positive likelihood ratios on the training and test sets increased as gestation progressed (1.51–3.18 and 1.60–3.02, respectively), while the negative likelihood ratios decreased (0.80–0.53 and 0.77–0.57, respectively). The area under the curve (AUC) was 0.59 (95% confidence interval (CI): 0.58–0.61) in the training set and 0.59 (95% CI: 0.56–0.63) in the test set for pregnancies identified during the first trimester, 0.62 (95% CI: 0.60–0.65) in the training set and 0.62 (95% CI: 0.57–0.67) in the test set for pregnancies identified in the second trimester, and 0.73 (95% CI: 0.69–0.76) in the training set and 0.73 (95% CI: 0.67–0.79) in the test set for pregnancies identified in the third trimester.

High-risk scoring efficacy on critical care

While preterm birth was the focus of our algorithm, evaluating our algorithm on other measures, such as entry into critical care, were indicators of the algorithm’s potential to model other outcomes. The stratification of pregnant women in our test data set who delivered infants needing critical care (admitted to neonatal intensive care unit (NICU), intensive care unit (ICU), or pediatric intensive care unit (PICU)) within the first year of life is presented in Figure 1. There were 503, 235, and 154 pregnant women in each of the three trimesters, respectively. Of note, the majority of women whose infants required critical care within the first year of life scored above 70 during pregnancy using our scoring algorithm, reflecting the effectiveness of our tool in assessing risk of other adverse medical complications associated with preterm birth.

Figure 1.

Stratification of pregnant women in our test data set who delivered infants needing critical care (admitted to NICU, ICU, or PICU) within the first year of life.

Discussion

In this study, we present a unique algorithm of indicating preterm birth during the first trimester of gestation by scoring a pregnant woman which is continually updated throughout gestation. The pregnancy was identified during the first trimester in the majority of women utilizing our clinical tool. Some of the patients who were identified later in their pregnancy may have been due to their joining the health plan in the third trimester. Negative predictive values, sensitivity, and positive likelihood ratios increased from the first to the third trimesters of gestation as pregnant women had more visits with their physicians, thereby, providing additional data for the scoring algorithm to process.

Passport’s conventional Health Plan differs significantly from our scoring algorithm based on two primary factors. Passport relies entirely on physician referrals when a pregnancy is identified. In this respect, these referrals are often not received by Passport’s care manager until the second or third trimester of a woman’s pregnancy. Using our clinical tool, we identified a greater number of pregnant women both in volume and at earlier stages in pregnancy through claims data and determined their risk of preterm birth. The pregnancy was identified during the first trimester in 15,766 (54.1%) of women utilizing our clinical tool compared with only 3817 (23.0%) of women using Passport’s conventional Health Plan over the study time period. Passport’s conventional Health Plan also looks at risk factors in isolation and does not provide a solution when more than one pregnant woman has the same factor. Furthermore, care managers are not trained to know which ICD codes correspond to specific medical risk factors and may miss relevant factors which are in a pregnant woman’s medical history. Using our scoring algorithm, we give the care manager a pregnant woman’s risk score and list of risk factors to permit more efficient management. A low incidence risk factor, such as education/literacy problems, does not occur frequently and, therefore, has a low contribution to our model. However, this risk factor is still clinically relevant for care managers to consider in their clinical decision making process. We provide these valuable details to care managers in a timely and efficient manner.

Care management plays an important role in mitigating preterm birth. Women with a history of spontaneous preterm delivery are 1.5–2 times more likely to have a subsequent preterm delivery.[16] Antenatal progesterone is associated with a significant decrease in subsequent preterm delivery in pregnant women.[16-19] Vaginal progesterone is recommended for women with a shortened cervix and no history of preterm delivery, while progesterone supplementation is suggested for all women with a history of spontaneous preterm delivery.[16,17] Cervical cerclage is beneficial for cervical weakening in women with a shortened cervix, and antenatal corticosteroids have been shown to improve post-delivery neonatal outcomes.

The identification of pregnancies in the earliest months of gestation allows interventions directed at averting preterm birth. The goal of recognizing pregnant women early in their pregnancy who are at an increased risk of preterm birth permits antenatal care aimed at delaying preterm birth and arranging for delivery at a hospital equipped with a neonatal intensive care.[3] Attempts at a clinical protocol for general preterm birth prediction have been reported,[20] although none has been broadly accepted or achieved clinical success. While the discovery and validation of biomarkers to classify pregnant women at the highest risk of preterm birth are currently being investigated,[21] no specific clinical biomarker of preterm birth presently exists. Furthermore, fetal fibronectin is only clinically beneficial after 22 weeks of gestation which is later in pregnancy than is optimal for intervention.[2,6,8]

Several scoring systems have been developed to classify the risk of preterm birth; however, their utility and validity have not been proven.[3,9,22,23] In 1969, Papiernik-Berkhauer proposed an empirical method for estimating the risk of premature delivery.[22,24] Maternal characteristics were grouped into four variables (social status, obstetric history, work conditions, and pregnancy characteristics), placed in a two-dimensional table, and assigned point values which determined the risk of preterm birth.[23,24] Creasy et al.[25] subsequently modified this risk table and proposed a system for scoring the risk of preterm delivery. Data mining has also been utilized to determine demographic predictors of preterm birth.[26] Using logistic regression analysis, Tekesin et al.[14] developed the CLEOPATRA score based on potential predictors of spontaneous preterm delivery. Pregnant women with preterm labor between 24 and 34 weeks of gestation were assigned a score. The AUC for CLEOPATRA I and CLEOPATRA II (the latter which included fetal fibronectin) was 0.69 (95% CI: 0.56–0.82) and 0.81 (95% CI: 0.69–0.93), respectively.[14] Fetal fibronectin and previous preterm delivery were associated with a higher risk of preterm delivery, with odds ratios of 17.9 and 4.56, respectively.

While our study is similar to that of Tekesin et al., their work only included 170 pregnant women in Germany. Our study featured 29,166 unique pregnancies in a Kentucky Medicaid population. Due to their small sample size, the CIs in Tekesin et al.’s work are wide compared to the much smaller CI in our study due to the larger data set. In addition, the access and quality of healthcare may differ greatly between our two populations. Furthermore, their analysis included data over two trimesters of gestation, while our work presented predictive scores for each trimester which were continually updated as additional medical information for pregnant women was obtained.

Dabi et al.[10] investigated six parameters (singleton or twin pregnancy, duration of pregnancy, cervical length, vaginal bleeding, preterm premature rupture of membranes, and uterine contraction requiring tocolysis) and included pregnant women with threatened preterm delivery between 24 and 34 weeks of gestation. These authors determined that their nomogram was efficient and clinically relevant. They proposed that an optimal threshold of 15% would minimize the risk of preterm deliveries in singleton pregnancies. Other studies in the literature highlighting predictive models of preterm birth report an AUC ranging between 0.60 and 0.67.[27-30]

Strengths and limitations of the present study

The focus of this study was to determine a risk score for pregnant women as a clinical tool and to understand the conditions which may modify risk. Our scoring algorithm may be used prospectively for intervention that may prevent a preterm birth or provide a clinician with information to alter management to limit complications of preterm birth. The major strength of our study includes both a large patient population and our real-time scoring algorithm which may be applied to other insurance carriers besides Passport’s Health Plan. Our clinical tool is used by the care management team to efficiently manage the high volumes of pregnant women and prioritize their patients who may have several chronic care conditions.

In addition, our scoring algorithm generalizes well to unseen data since the performance on the test data set was only slightly less (with wider CIs) than the training data set. The majority of previous studies highlighting scoring algorithms that indicate preterm birth are performed in the late second trimester and assign a score to a pregnant woman on one occasion. Not only is our algorithm capable of identifying pregnant women during the first trimester at which time a score is assigned, but it continually updates this score throughout gestation as additional information is acquired.

Figure 2 depicts the gestational timeline of a pregnant woman using our clinical tool. Our algorithm assigned her a score of 92 at identification which was 6 weeks post-conception. This score increased to 93 by the third trimester due to additional insurance claims of depression and anxiety. Care management intervened at 8 weeks post-conception and enrolled the patient in a detox program and subsequently arranged for mental health treatment. The neonate was delivered at 39 weeks, went to a general nursery, and was treated as an inpatient for 4 days in the first year for a total cost of US $6536. This case reflects several important aspects of our clinical tool, specifically, identification of the pregnant woman early in gestation, a real-time alteration of her score based on triggers indicative of preterm birth, and care management intervention. Thus, the neonate was born full-term and required minimal inpatient care in the first year of life.

Figure 2.

Gestational timeline of a pregnant woman using our scoring algorithm.

The retrospective nature and use of claims data in our study represent limitations of our work. Physicians may not include all ICD codes billed on a claim and may not be able to gather social determinant data (low income, homelessness, and inability to access healthcare) in claims. Utilizing both claims and EMR data may shed more light on a pregnant woman’s medical and social history. While the AUC for the second trimester of our algorithm has predictive ability (0.62 in test data set) and is similar to that of previously published works,[27-30] it is not modest and not ideal. As the AUC is used for care management, we anticipate evaluating the success of this approach in the future publications. Our scoring algorithm currently incorporates 71 risk factors of preterm birth; however, further work on our algorithm will apply logistic lasso regression and k-fold cross-validation to perform better variable selection and achieve higher predictive performance. Our false positives may have been overly high since our scoring algorithm incorporated conditions which may not have been strong risk factors of preterm birth. We anticipate updating our scoring algorithm with revisions to the computational and coding structure, including a focus on predicting preterm birth at different weeks of gestation (<28 weeks or <34 weeks). In addition, we also intend to apply our scoring algorithm of risk identification and stratification to the population at large who may have commercial insurance. These latter patients would have a more robust medical history, greater number of visits with their obstetricians, and more access to mental health professionals skilled to treat anxiety and postpartum depression.

Conclusion

Preterm birth represents a significant public health concern with both financial and societal repercussions. Our real-time scoring algorithm of preterm birth effectively identifies and stratifies pregnant women early in gestation. Further analysis will ascertain the risk factors that are most indicative of preterm birth which will be incorporated into our scoring algorithm. Future studies will also elucidate the impact of care management, permit obstetricians to view a pregnant woman’s risk score to provide more effective treatment to those who are at high-risk, and interventions such as drug detoxification, mental health services, administration of betamethasone or magnesium, and treatment at a facility with a NICU that may prove beneficial on preterm birth.