A Model to Predict Treatment Failure of Single‑Dose Methotrexate in Patients with Tubal Pregnancy.

BACKGROUND In China, approximately 15% of tubal pregnancy patients treated with MTX eventually required surgery because the ectopic mass was ruptured; therefore, it is essential to develop a model to predict the risk of failure with methotrexate treatment in tubal pregnancy. MATERIAL AND METHODS In this research, 168 patients met the eligibility criteria, and 29 candidate risk factors for treatment failure were collected. Multivariable logistic regression analysis was used to analyze the factors, and a full model was developed. We used a multiple fractional polynomial model and a stepwise model to increase the reliability. Bootstrap resampling for 500 times was used to internally test the prediction model. The integral performance of the model depends on the evaluation of the nomogram, the discriminative performance by receiver operating characteristic (ROC) curve analysis, and calibration. RESULTS The model showed excellent discrimination and calibration. The area under the ROC curve for the prediction model, mfp model, and stepwise model were 0.879 (95% CI: 0.812-0.942), 0.872 (95% CI: 0.805-0.931), and 0.880 (95% CI: 0.817-0.949), respectively. At a cutoff value of ≥0.40, sensitivity was 60%, specificity was 91%, positive predictive value (PPV) was 81%, and negative predictive value (NPV) was 77%. The model provides a net benefit when clinical decision thresholds are between 0% and 40% of predicted risk. CONCLUSIONS This model indicated good accuracy in predicting methotrexate treatment failure for tubal pregnancy patients.

Background

Medical management with methotrexate (MTX) is widely used in hemodynamically stable patients who have an unruptured ectopic mass [1]. Many studies found that MTX provides economical and practical management [2,3]. For patients with early tubal pregnancy, treatment with MTX can effectively avoid the hidden risks of surgery. However, some research indicated that approximately 15% of tubal pregnancy patients treated with MTX eventually required surgery because the ectopic mass was ruptured [4], but the reason for this failure is unknown. Ectopic pregnancy is a major cause of morbidity and mortality, including infertility and subsequent ectopic pregnancy [5,6]. It results in about 9% of all pregnancy-related deaths and is a life-threatening condition [7]. More than 90% of EPs occur in the fallopian tube [8]. Recently, several studies have focused on the prognostic factors of successful management of MTX. For example, initial HCG levels >1300 IU/L and/or use of combined oral contraception before pregnancy were regarded as risk factors of MTX failure in France [9]. In Korea, Jae Hoon Lee performed a risk prediction model and indicated the presence of gestational sac, the size of ectopic mass, and follow-up HCG levels above the threshold on days 4 and 7 were risk predictors [10]. However, these studies did not include data on Chinese patients. Because of China’s huge population and the initiation of the second child policy, it is necessary to develop a prediction model focusing on Chinese patients.

This research included the risk factors, symptom, signs, and laboratory test results of tubal pregnancy patients treated with MTX, and we used them to develop a prediction model. This study aimed to enhance the success rates of treatment using MTX among patients with tubal pregnancy.

Material and Methods

Diagnostic criteria and patient selection

This retrospective cohort study was performed at the First Affiliated Hospital of Guangzhou University of Traditional Chinese Medicine from June 2015 to November 2018 and was approved by the Ethics Committee. Because it was a retrospective study, there was no requirement of prospective ethics approval. We initially included 989 patients diagnosed with ectopic pregnancy, and extracted 508 patients with surgical treatment and 247 patients with expectant management. Non-tubal pregnancy, including cervical pregnancy, cesarean section, and scar pregnancy, were excluded. Patients who received double-dose methotrexate were also excluded. Finally, 168 patients were included in this research (Figure 1).

Figure 1

Flow chart of study process.

Patients recruited into this study needed to meet the following ACOG clinical criteria [11]: positive result of pregnancy test and a transvaginal ultrasound evaluation were the minimal criteria. When the women met this criteria, serial assessments such as serum HCG, serum progesterone, or/and transvaginal ultrasound were used to confirm the diagnosis.

Tubal pregnancy patients treated with single-dose methotrexate in this study also met the following conditions [11]: an unruptured mass and stable hemodynamics. Patients who received methotrexate treatment were informed about the importance of follow-up surveillance. Exclusion criteria were: hemodynamically unstable, patients preferred to have surgical management and expectant management, patients who were diagnosed with non-tubal pregnancy, patients treated by two-dose methotrexate, and patients who had absolute contraindications to methotrexate treatment.

Treatment protocol

Before the final decision of treatment was made, both methotrexate management and surgery management were provided to the patients. The clinicians explained the related risks to the patients in details. After obtaining the consent of patients, selected patients received methotrexate treatment in a single-dose protocol.

The participating patients received a single intramuscular injection of 50 mg/m2 of body surface area (BSA). The day when tubal pregnancy patients accepted MTX treatment was regarded as day 1. To evaluate the condition, serum β-HCG level was measured on day 4 and day 7, and the ultrasound was reviewed on day 7 [12,13].

Measurement of variables

Outcome variables

Failure of MTX treatment was defined as one of the following conditions: serum β-HCG levels decreased to under 15% or even increased at day 7 after treatment, or acute abdominal pain, blood pressure drop, and shock due to rupture tubal pregnancy for emergency surgery, as defined in other related clinical studies [9,23].

Predictors involved in model development

According to the recommendation in Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guide (TRIPOD guide) [14], predictors are demographic characteristics (e.g., age and BMI), previous medical history, physical examination results from blood measurements, and imaging in other biological measurement platforms associated with the disease. Therefore, our research used the predictors that simply summarized demographic characteristic, previous medical history, symptom, signs, blood measurement, and imaging measurements. Given the applicability and popularity of this prediction model, our research included predictors that are easily ascertained in clinical practice. We also referred to the predictors used in other similar research [4]. Changes in serum β-HCG and 2 predictors – differences in β-HCG and ratio of β-HCG – were evaluated by the following algorithm: difference of β-HCG=β-HCG on day 4 – β-HCG on day 1 and ratio of β-HCG=β-HCG on day 4 – β-HCG on day 1.

The predictors in our research were age, BMI, gravida, number of births, number of cesarean sections, number of ectopic pregnancies, previous medical history (e.g., history of pelvic inflammatory disease and history of infertility), clinical symptoms (e.g., menopause, vaginal bleeding, and abdominal pain), signs (e.g., abdominal tension, abdominal tenderness, abdominal rebound), blood test results (white blood cell count, neutrophil percentage, hemoglobin, platelets), β-HCG on day 1, progesterone on day 1, β-HCG on day 4 (Day 4 HCG), difference in β-HCG, and vaginal ultrasound (mass size, endometrial thickness, pelvic effusion, the presence of yolk sac, the presence of embryo, the presence of fetal heart beat). These predictors were investigated by doctors when patients were hospitalized. The result of the laboratory tests and ultrasound were recorded in the medical record system at the First Affiliated Hospital of Guangzhou University of Traditional Chinese Medicine. The researchers extracted the variables and recorded them in the Empower electronic data capture for subsequent analysis.

Statistical analysis

Continuous variables are expressed as standard deviations (SDs) as determined by Kruskal-Wallis rank sum test. Categorical variables are presented as proportions and absolute numbers as determined by Fisher’s exact test.

To construct the prediction model, several steps were performed. The prediction model was built up by selecting predictors and combining them into a multivariable model using multivariate logistic regression, and the full model was developed with all selected predictors. Due to the multicollinearity of the predictors, the multiple fractional polynomial (MFP) model was performed. According to the Akaike’s Information Criterion (AIC) [15], the stepwise model was fitted using stepwise backward selection.

Internal validation is an important step of model development to evaluate the degree of optimism, and we used bootstrap resampling by 500 times internally. Moreover, calibration was performed to assess the agreement between predicted and observed risks.

The nomogram, according to the model based on internal validation, aimed to help calculating the predicted probabilities for each patient. Clinicians can use the nomogram to calculate a specific sum score for patients according to the presence or level of parameters to assess the prognosis of each patient.

In this study, the area under the ROC curve showed the association between specificity and sensitivity at specific values of nomogram scores. The positive predictive value (PPV) indicates the success of MTX treatment if the nomogram score surpasses or is equal to the threshold. The negative predictive value (NPV) indicates the probability of failure of MTX treatment if the score of the nomogram is lower than the threshold.

All statistical analyses were performed using Empower. P<0.05 was considered statistically significant.

Results

Patients characteristic

Finally, 168 patients were included in this study (Table 1). Among all the participants, medical treatment failed in 42 patients and succeeded in 126 patients. Significant difference between these 2 groups were found in the previous history of pelvic inflammatory disease, β-HCG on day 1, progesterone on day 1, follow-up β-HCG on day 4, endometrial thickness, and the existence of a yolk sac.

Table 1

The baseline characteristics of the MTX treatment in the 2 groups.

Treatment result	Success group	Failure group	P value
N	126	42
Age (yr)	30.47 (5.51)	30.45 (5.74)	0.84
BMI	20.46 (4.47)	20.42 (3.75)	0.99
Gravida	3.20 (1.58)	3.07 (1.62)	0.52
Number of births	1.33 (0.60)	1.50 (0.66)	0.22
Number of cesarean sections	1.20 (0.82)	1.17 (0.39)	0.47
Number of ectopic pregnancies	1.08 (0.28)	1.28 (0.57)	0.18
History of infertility			0.86
No	115 (91.27%)	38 (90.48%)
Yes	11 (8.73%)	4 (9.52%)
History of pelvic inflammatory			<0.01
No	114 (90.48%)	26 (61.90%)
Yes	12 (9.52%)	16 (38.10%)
Menopause	49.52 (15.52)	46.86 (12.03)	0.19
Vaginal bleeding	10.00 (0.00–40.00)	8.00 (0.00–49.00)	0.16
Abdominal pain	1.00 (0.00–30.00)	0.10 (0.00–49.00)	0.47
Abdominal tenderness			0.12
No	106 (84.13%)	39 (92.86%)
Yes	20 (15.87%)	3 (7.14%)
Abdominal rebound			0.98
No	119 (94.44%)	39 (92.86%)
Yes	7 (5.56%)	3 (7.14%)
WBC	7.00 (1.71–13.26)	7.60 (3.90–128.00)	0.083
NEU%	65.39 (10.13)	67.66 (9.68)	0.209
HGB	123.43 (11.08)	122.09 (10.06)	0.368
PLT	245.74 (55.09)	250.30 (56.56)	0.715
β-HCG at day 1	804.66 (28.11–5689.00)	1555.50 (4.64–19995.00)	<0.001
Progesterone at day 1	14.16 (0.00–190.80)	20.43 (2.26–355.20)	0.004
β-HCG at day 4	641.70 (46.63–6159.00)	1565.00 (29.20–12446.00)	<0.001
Difference of β-HCG	−23.70 (−2447.00–3429.40)	10.27 (−12257.00–3326.00)	0.661
Ratio of β-HCG	0.94 (0.07–9.27)	1.01 (0.06–152.33)	0.175
Mass size	28.50 (11.00–81.00)	25.00 (11.00–100.00)	0.189
Endometrial thickness	7.46 (2.67)	11.06 (3.02)	<0.001
Pelvic effusion	14.00 (0.00–92.00)	0.00 (0.00–76.00)	0.205
The presence of yolk sac			<0.001
No	116 (92.80%)	32 (76.19%)
Yes	9 (7.20%)	10 (23.81%)
The presence of embryo			0.369
No	125 (100.00%)	41 (97.62%)
Yes	0 (0.00%)	1 (2.38%)
The presence of fetal heart beat			0.369
No	125 (100.00%)	41 (97.62%)
Yes	0 (0.00%)	1 (2.38%)

If the data was normally distributed, it was shown in Mean±SD/N(%). If the data was skew distributed, it was shown in Mean(SD) Median (Q1−Q3)/N(%)

P-value: Continuous variables were reported as standard deviations (SDs) by Kruskal Wallis rank sum test. If the theoretical number of count variable is less than 10, Fisher rank will be performed.

Multivariate analyses and internal validation

The regression coefficients in the prediction model were adjusted and are shown in Table 2.

Table 2

Multivariable regression analyses of the association between predictors and MTX treatment failure in tubal pregnancy.

	Statistics	OR(95% CI), P value	P value
History of pelvic inflammatory
No	132 (78.85%)	1.0	1.0
Yes	36 (21.15%)	6.33 (2.98, 13.45)	<0.01
WBC	8.10±8.71	1.15 (1.01, 1.31)	0.04
Baseline β-HCG	1703.41±2338.74	1.00 (1.00, 1.00)	<0.01
Baseline progesterone	28.92±40.45	1.01 (1.00, 1.02)	0.04
Follow up β-HCG at day 4	1640.49±2010.47	1.00 (1.00, 1.00)	<0.01
Emdometrial thickness	8.89±3.37	1.54 (1.35, 1.76)	<0.01
The presence of yolk sac
No	144 (85.78%)	1.0	1.0
Yes	24 (14.22%)	4.22 (1.78, 9.99)	<0.01

The full prediction model contained the following variables: previous history of pelvic inflammatory disease, WBC, baseline β-HCG, baseline progesterone, follow-up β-HCG at day 4, presence of a yolk sac, and endometrial thickness, and the areas under the curve (AUC) of the corresponding ROC was 0.879 (95% CI, 0.812–0.942). Due to the multicollinearity, the MFP model was used, and the AUC of the ROC was 0.872 (95% CI, 0.805–0.931). The stepwise model was the final model and the AUC was 0.880 (95% CI, 0.817–0.949), which was similar to the full prediction model and MFP model, so the prediction model was stable. The results were shown in Figure 2. After the prediction model was constructed, internal validation was performed by bootstrapping (n=500). The regression coefficients in the prediction model and calibration plot of the internally validated model was presented in Figure 3.

Figure 2

The ROC of prediction model. ROC curves relevant to the full prediction model (OR 0.879 95% CI: 0.812–0.942), mfp model (OR=0.872 95% CI: 0.805–0.931) and stepwise model (OR 0.880 95% CI: 0.817–0.949). ROC – receiver operating characteristic.

Figure 3

Calibration charts. Calibration charts of the nomogram for the possibility of treatment failure by bootstrap resampling by 500 times.

Nomogram predicts treatment failure

The nomogram based on the multivariable logistic model is shown in Figure 4. Clinicians can use the nomogram to define and sum the points according to the patients’ characteristics. For example, a previous history of pelvic inflammatory disease receives 32 points. The points were added up from all characteristics and are shown in the “total points” proportion. The predicted possibility of the MTX treatment failure was assessed by the total points in “treatment result”, which is in the bottom of the figure.

Figure 4

Nomogram for the prediction model. Sum up the points on the scale for independent covariate in the nomogram. The total points cast to the scale on the bottom demonstrate the possibility of treatment failure.

Optimal cutoff value and performance in individual patients

Specificity and sensitivity for the prediction of MTX treatment failure at different thresholds is shown in Table 3. When the cutoff value is higher than or equal to 0.40, sensitivity is 60% and specificity is 91%. Higher cutoff values increase specificity, but decrease sensitivity. PPV and NPV were related to the sensitivity and specificity at different cutoff values, which are shown in Table 3. For example, at the cutoff value of ≥0.40, PPV is 81% and NPV is 77%.

Table 3

Specificity, sensitivity, negative predictive value, and positive predictive value of the Nomogram scores at different thresholds.

Predicted probability	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
≥0.30	62	87	81	79
≥0.40	60	91	81	77
≥0.50	57	91	80	76
≥0.60	57	92	80	76
≥0.70	55	93	84	76
≥0.80	53	94	86	75
≥0.90	52	94	86	75

The predicted probability/nomogram score is a numeric value representing the prediction model score of the individual patient. The nomogram score can be used as a test parameter, and a positive test result can be defined as a score that is equal to or exceeds a specific cutoff value. A negative test result is defined as a score less than the cutoff value. NPV – negative predictive value; PPV – positive predictive value.

Discussion

The aim of this study was to develop a model to predict the treatment failure of patients with tubal pregnancy who were treated by single-dose MTX. The final model included previous history of pelvic inflammatory disease, β-HCG on day 1, progesterone on day 1, β-HCG on day 4, the presence of a yolk sac, and the endometrial thickness as the predictors. In this study, the model was generated by several sensitivity analyses, including full model, MFP model, and stepwise model. In the final model, the AUC of stepwise model was 0.880 (95% CI: 0.817–0.949), which is similar to that of the full model and MFP model. By bootstrapping 500 times, the calibration showed a stable accuracy in the final prediction model. The model is shown as a nomogram in Figure 4. At the cutoff value of ≥0.40 in the nomogram, the PPV of the prediction model was 81% and the NPV was 77% for the occurrence of the MTX treatment failure. The sensitivity at this cutoff value is 60% and specificity is 91%, so the model is believed to predict approximately three-quarters of MTX treatment failures of tubal pregnancy.

We have successfully built and temporally validated a model based on Chinese patients; this model accurately predicts the risk of MTX treatment failure in a patient. The model offers several advantages. First, predictors included in this model accurately defined the clinical factors routinely used in clinical practice and provides an easy-to-use calculator, which increases the model’s potential ease of use. Secondly, this model focused on the possibility of multicollinearity, as prediction models generated by full model may create bias in the estimated regression coefficients and mask multicollinearity [24]. It is also important to develop a model without variables that may add little or no useful information. Omitting necessary predictive factors leads to an inaccurate prediction due to biased estimation of the regression coefficients [25]. To avoid these problems, the MFP model was used to evaluate the sensitivity analysis of possible linear relationships, and the stepwise model was used to filter the unnecessary factors and develop a more accurate model. The final model was based on the stepwise model and its regression coefficients were similar to those of the full prediction model and MFP model. Hence, the prediction in our study is more robust and stable. Thirdly, some research only considered the relationship between a single factor and the treatment outcome [17-22], and the others focused on the association between laboratory tests or ultrasound and the treatment outcome [10]. Risk factors can affect the occurrence of tubal pregnancy and also can play a significant role in treatment outcome, so the prediction model of this study included the risk factors compared with previous studies.

This study also has important secondary findings, which is different from previous studies. Firstly, we found that β-HCG on day 4 was a potential predictor of MTX treatment failure, which was comparable to a previous study [10]. The use of β-HCG on day 4 in model development is a disadvantage of this study; however, if D4 β-HCG decreases by less than 15% or does not decrease, it is a warning sign for failure of MTX treatment. This reminds clinicians to change to surgical treatment, avoiding treatment plans leading to ruptured ectopic pregnancy. Further research with larger sample sizes may give the model better predictive ability. Secondly, previous research showed that progesterone is a predictor of a non-viable pregnancy, with an excellent specificity of 98.4% (95% CI, 90.9–99.7%) [16]. It was found to be a predictor of failure treatment of tubal pregnancy and it is possible that non-viable ectopic pregnancy is more easily cured. Thirdly, while most previous studies focused on the endometrial thickness as a diagnostic indicator to distinguish between intrauterine pregnancy and ectopic pregnancy [19-21], we found that endometrial thickness was a predictor of failure of single-dose MTX treatment of tubal pregnancy. Finally, research found that EP is more common in women with pelvic inflammation, and more than 50% of these women are unaware their pelvic inflammation [22]. In our study, pelvic inflammation and WBC were found to be the predictors of MTX treatment failure, which explains why inflammation is strongly associated with MTX treatment failure, but the mechanism underlying this association is unclear. Moreover, we demonstrated the association between pelvic inflammatory disease and the failure of single-dose MTX treatment of tubal pregnancy, which shows that clinicians should focus on the related medical history before deciding on a treatment plan.

Several limitations should be considered when interpreting our results. This study only included patients who were treated with single-dose MTX. Further research should expand the sample size by including the patients who were treated with double-dose MTX. Although the sample size in our study was small, this study provides strong clinical evidence for subsequent related research.

Conclusions

In conclusion, our prediction model shows that a previous history of pelvic inflammatory disease, WBC, β-HCG on day 1, progesterone on day 1, β-HCG on day 4, the existence of a yolk sac, and endometrial thickness were associated with the failure of single-dose MTX treatment of tubal pregnancy, which conflicts with previous research, and we also built a graphic nomogram for predicting MTX treatment failure for clinician use. Further research with larger sample sizes should include tubal pregnancy patients treated with double-dose MTX treatment in order to make a prediction model with better prediction ability.