Literature DB >> 30509329

Risk prediction for estrogen receptor-specific breast cancers in two large prospective cohorts.

Kuanrong Li1, Garnet Anderson2, Vivian Viallon1, Patrick Arveux3,4,5,6, Marina Kvaskoff5,6, Agnès Fournier5,6, Vittorio Krogh7, Rosario Tumino8, Maria-Jose Sánchez9,10, Eva Ardanaz10,11,12, María-Dolores Chirlaque10,13,14, Antonio Agudo15, David C Muller16, Todd Smith16, Ioanna Tzoulaki16, Timothy J Key17, Bas Bueno-de-Mesquita16,18,19,20, Antonia Trichopoulou21,22, Christina Bamia21,22, Philippos Orfanos21,22, Rudolf Kaaks23, Anika Hüsing23, Renée T Fortner23, Anne Zeleniuch-Jacquotte24,25,26,27, Malin Sund28, Christina C Dahm29, Kim Overvad29,30, Dagfinn Aune16,31, Elisabete Weiderpass32,33,34,35, Isabelle Romieu36, Elio Riboli16, Marc J Gunter36, Laure Dossus37, Ross Prentice2, Pietro Ferrari38.   

Abstract

BACKGROUND: Few published breast cancer (BC) risk prediction models consider the heterogeneity of predictor variables between estrogen-receptor positive (ER+) and negative (ER-) tumors. Using data from two large cohorts, we examined whether modeling this heterogeneity could improve prediction.
METHODS: We built two models, for ER+ (ModelER+) and ER- tumors (ModelER-), respectively, in 281,330 women (51% postmenopausal at recruitment) from the European Prospective Investigation into Cancer and Nutrition cohort. Discrimination (C-statistic) and calibration (the agreement between predicted and observed tumor risks) were assessed both internally and externally in 82,319 postmenopausal women from the Women's Health Initiative study. We performed decision curve analysis to compare ModelER+ and the Gail model (ModelGail) regarding their applicability in risk assessment for chemoprevention.
RESULTS: Parity, number of full-term pregnancies, age at first full-term pregnancy and body height were only associated with ER+ tumors. Menopausal status, age at menarche and at menopause, hormone replacement therapy, postmenopausal body mass index, and alcohol intake were homogeneously associated with ER+ and ER- tumors. Internal validation yielded a C-statistic of 0.64 for ModelER+ and 0.59 for ModelER-. External validation reduced the C-statistic of ModelER+ (0.59) and ModelGail (0.57). In external evaluation of calibration, ModelER+ outperformed the ModelGail: the former led to a 9% overestimation of the risk of ER+ tumors, while the latter yielded a 22% underestimation of the overall BC risk. Compared with the treat-all strategy, ModelER+ produced equal or higher net benefits irrespective of the benefit-to-harm ratio of chemoprevention, while ModelGail did not produce higher net benefits unless the benefit-to-harm ratio was below 50. The clinical applicability, i.e. the area defined by the net benefit curve and the treat-all and treat-none strategies, was 12.7 × 10- 6 for ModelER+ and 3.0 × 10- 6 for ModelGail.
CONCLUSIONS: Modeling heterogeneous epidemiological risk factors might yield little improvement in BC risk prediction. Nevertheless, a model specifically predictive of ER+ tumor risk could be more applicable than an omnibus model in risk assessment for chemoprevention.

Entities:  

Keywords:  Breast cancer; EPIC; Estrogen receptor; Prospective cohort; Risk prediction; WHI

Mesh:

Substances:

Year:  2018        PMID: 30509329      PMCID: PMC6276150          DOI: 10.1186/s13058-018-1073-0

Source DB:  PubMed          Journal:  Breast Cancer Res        ISSN: 1465-5411            Impact factor:   6.466


Background

Breast cancer (BC) screening and chemoprevention strategies should prioritize women who are expected to benefit from the interventions. Risk prediction models could be useful assessment tools to facilitate this strategy, as long as the models themselves possess sufficient predictive power. So far, more than 20 risk prediction models have been developed for BC since the first model developed by Gail in 1989 [1, 2]. Initially, the Gail model (hereinafter referred to as ModelGail) was based on age, age at menarche and at first live birth, previous breast biopsy, and family history of BC, yielding moderate discriminatory power (C-statistic) of 0.58 in external validations [3, 4]. New predictors, such as breast density, hormone replacement therapy (HRT), anthropometric measures, and lifestyle factors (e.g. alcohol intake), were continuously introduced into the succeeding models, resulting in marginal improvements in prediction [5]. BC comprises etiologically distinct subtypes defined by molecular factors. Hormonal and reproductive factors, such as elevated circulating sex hormone levels, early menarche, delayed childbirth, and nulliparity, are only or are more strongly related to increased risks of subtypes expressing estrogen receptor (ER+) and progesterone receptor (PR+) [6]. Further, ER+ breast tumors respond more favorably to hormone therapy than ER-/PR- tumors [6-8]. It has been hypothesized that combining etiologically distinct subtypes as one single outcome undermines BC prediction [9]. However, most of the published BC risk prediction models are omnibus models and only one model differentiates risk associations by hormone receptor status [10]. In the current analysis, using data from the European Prospective Investigation into Cancer and Nutrition (EPIC) and the Women’s Health Initiative (WHI) study in the USA, we examined whether modeling heterogeneous risk associations by ER status, which entails building ER-specific risk prediction models, could yield better prediction of BC risk.

Methods

Study population for model derivation and internal validation

The study population for model derivation consisted of women recruited into the EPIC cohort from 1992 to 2000 in 10 European countries (Norway, Sweden, Denmark, the UK, the Netherlands, Germany, France, Spain, Italy, and Greece) [11, 12]. Women with one or more of the following characteristics were excluded: (1) < 40 or > 70 years of age at recruitment (n = 49,410); (2) diagnosed with cancer before recruitment (n = 39,760); and (3) no information on censoring date and/or disease status (n = 142). All women recruited in the study center of Malmö, Sweden were also excluded due to lack of information on ER status for all BC diagnoses (n = 14,396). After these exclusions, 281,330 women (51% postmenopausal at recruitment) were included in the analysis.

Study population for external validation

The WHI study was launched in 1993 and recruited 161,808 postmenopausal women aged 50–79 years into either an observational study or one of the three clinical trials that tested the health effects of HRT, a low-fat diet, and calcium-vitamin D supplementation, respectively [13]. For the purpose of the present study, we excluded non-Caucasian women (n = 28,267), women in the HRT trial (n = 27,347), women who had mastectomy or a history of cancer at recruitment (n = 16,501), and women with incomplete information on the risk factors considered in our models (n = 29,431), resulting in a validation population of 82,319 women. All women in the EPIC and WHI studies provided written informed consent. In the WHI study, Human Subjects Committee approval at each participating institution was provided. The present study was approved by the Ethical Review Board of the International Agency for Research on Cancer (Lyon, France).

Risk factors and disease outcomes

Among the most frequently included predictors in current BC risk prediction models [5], the following variables were available in EPIC and WHI, and were therefore included in this study: menopausal status, age at menopause, age at menarche, duration of HRT, duration of breastfeeding, full-term pregnancy (FTP), number of FTPs, age at first FTP, body height, body mass index (BMI), interaction between BMI and menopausal status, alcohol intake, and country. Table 5 in Appendix provides the coding of these predictor variables. We retained all the women for analysis and handled the missing values by five-time multiple imputations with chained equations [14]. Three predictor variablesin the Gail model were not included in our models, i.e. family history of BC in first-degree relatives, previous breast biopsy, and history of atypical hyperplasia. In the EPIC study, family history of BC was only available for 49% of women, while information on previous breast biopsy and history of atypical hyperplasia were not collected. Sensitivity analyses that included effect modification of parity by menopausal status in the EPIC study showed no evidence of statistically significant interactions. Similarly, no effect modifications were observed for HRT by BMI and breastfeeding by parity. These interactions were hence not retained further. Incident BC diagnoses among the EPIC women were ascertained through national cancer registries or a combination of health insurance records, pathology registries, and regular questionnaire surveys. The definition of positive hormone receptor status was standardized using the following criteria: ≥ 10% cells stained, any “plus-system description”, ≥ 20 fmol/mg, an Allred score of ≥ 3, immunoreactive score (IRS) ≥ 2, or an H-score ≥ 10. Among the WHI women, centrally trained, locally based physician adjudicators verified BC diagnoses by medical record and pathology report review, and positive hormone receptor status was defined as ≥ 10% cells stained [15].

Absolute risk modeling

Using the EPIC data, we fitted cause-specific piecewise-constant hazards models [16] for ER+ and ER- tumors separately (hereinafter referred to as ModelER+ and ModelER-). The cutoffs were placed at 45, 50, 55, 60, 65, 70, and 75 years of age. Whether a risk association is heterogeneous by ER status was examined using the likelihood ratio test [17]. Tumors with unknown ER status, primary cancers at other sites, and deaths from non-cancer causes were modelled as competing events to ER+ tumors and ER- tumors. A Gompertz model with age as the time scale was fitted for all these competing events combined. In addition, ER+ and ER- tumors were considered mutually competing. To evaluate the improvement in risk prediction by modeling the heterogeneous risk associations, an omnibus model was also fitted following the same methodology described above, treating ER+ and ER- tumors as one single disease outcome.

Model validation

First, we validated our ER-specific models internally by fivefold cross-validation [18] and then externally using WHI data. For external validation using the WHI data, we combined the model coefficients derived from the EPIC women and the ER-specific baseline hazards of the WHI women to project 5-year ER-specific absolute risks. We calculated C-statistics to assess discriminatory accuracy and the ratio of expected-to-observed number of tumors occurring in the first 5 years (E/O) to assess overall calibration. In the WHI women, the 5-year absolute risk of developing BC was projected using ModelGail, enabling us to compare the performance of our model with that of ModelGail. We performed decision curve analysis in the WHI women to compare the clinical applicability of ModelER+ and ModelGail for identification of women for chemoprevention. Let B denote the benefit of receiving chemoprevention for an individual who would develop BC, H the harm of receiving chemoprevention for an individual who would never develop BC, and pi indicates an individual risk. The rationale of decision curve analysis is that positive net benefits is guaranteed at the population level if chemoprevention only covers individuals with risk projections pi above the risk threshold pt, where: pt × B = (1 − pt) × H [19, 20]. Given the fact that quantities of B and H of chemoprevention remain unknown, net benefits are calculated through all the possible risk thresholds between two extremes, i.e. zero and the maximal risk estimate, representing a treat-all strategy and a treat-none strategy, respectively. The clinical applicability of a risk prediction model is indicated by how much the model’s net benefit curve is above the treat-all and treat-none strategies, i.e. the area formed by the model’s net benefit curve and the two extreme strategies.

Results

Cohort description

Country-specific distributions of the risk factors among the EPIC women are shown in Table 6 in Appendix. Distributions of the same risk factors among the WHI women are shown in Table 7 in Appendix. During an average follow-up period of 14.7 years, 12,067 BC cases (7210 ER+ tumors, 1598 ER- tumors, and 3259 tumors with unknown ER status), 16,929 primary cancers at other sites, and 6548 deaths from non-cancer causes were ascertained among the EPIC women, as reported in Table 1.
Table 1

Distribution of incident breast cancer (BC) by country, estrogen receptor (ER) status, and baseline menopausal status among the women from the European Prospective Investigation into Cancer and Nutrition (EPIC) and Women’s Health Initiative (WHI) studies

NumberAge at recruitment (years)Years of follow-upIncident BCCrude incidence rate (/105 person-years)PremenopausalPostmenopausal
ER+ER-IndefiniteBCER+ER-IndefiniteBC
EPIC study
 France68,70751.514.733863821232323221177611502542061610
 Italy27,85152.015.0113528740188925814027082554
 Spain20,29850.216.755617116443922991483871257
 UK35,34952.316.116023003287626767147565391931
 Netherlands22,60154.815.09753052063514338433355203591
 Greece11,33755.611.7201158196446935394132
 Germany22,08552.611.674332219852492993087363444
 Sweden914250.516.43702471433838219872737151
 Denmark29,30956.316.41887428257611004189212073411469
 Norway34,65148.014.112122632455448077915830245433
 Total281,33052.114.712,067312319377615265495401782217336572
WHI study82,31963.28.2295145722764212542951
Distribution of incident breast cancer (BC) by country, estrogen receptor (ER) status, and baseline menopausal status among the women from the European Prospective Investigation into Cancer and Nutrition (EPIC) and Women’s Health Initiative (WHI) studies

The ER-specific absolute risk models

Among the risk factors with identical associations by ER status (Table 2), being postmenopausal compared with premenopausal at recruitment was associated with a reduced tumor risk after controlling for age (hazard ratio (HR) = 0.66, 95% confidence interval (CI) = 0.60 to 0.74). For postmenopausal women, a statistically significant and monotonically increasing tumor risk was observed with older age at menopause compared with reaching menopause before the age of 45 years (ptrend < 0.001). No statistically significant association was observed for breastfeeding and breast cancer risk among parous women. Later age at menarche (≥ 15 vs ≤ 11 years of age) was statistically significantly associated with decreased tumor risk (HR = 0.85, 95% CI = 0.79 to 0.92). Duration of HRT was statistically significantly associated with increased breast cancer risk (ptrend < 0.001). BMI was associated with breast cancer and exhibited a statistically significant interaction with menopausal status: for postmenopausal women, HRs (95% CIs) for the BMI categories in ascending order were 1.11 (1.04 to 1.18), 1.21 (1.10 to 1.34), and 1.30 (1.11 to 1.53), respectively. For alcohol intake, exceeding one drink per day, compared with nondrinking, was statistically significantly associated with an increased breast cancer risk.
Table 2

Risk associations for ER+ and ER- tumors, the EPIC study and the WHI studya

Risk factorsEPIC studyWHI study
ER+, n = 7210HR (95% CI)ER-, n = 1598HR (95% CI)ER+, n = 2276HR (95% CI)ER-, n = 421HR (95% CI)
Menopausal status:
 postmenopausalb vs premenopausal0.66 (0.60–0.74)
Age at menopause, years:
 45.1–50.0 vs ≤ 45.01.16 (1.06–1.28)1.16 (1.05–1.29)
 50.1–55.0 vs ≤ 45.01.25 (1.13–1.38)1.41 (1.27–1.56)
 > 55.0 vs ≤ 45.01.41 (1.21–1.63)e1.40 (1.20–1.62)
Breastfeeding, months:
 0.1–6 vs 01.01 (0.95–1.08)1.04 (0.94–1.14)
 6.1–12 vs 00.96 (0.88–1.04)1.04 (0.91–1.18)
 > 12 vs 01.01 (0.93–1.11)1.07 (0.95–1.20)
Age at menarche, years:
 12 vs ≤ 111.06 (0.98–1.14)0.89 (0.80–0.99)
 13 vs ≤ 111.00 (0.93–1.07)0.82 (0.74–0.91)
 14 vs ≤ 110.97 (0.91–1.05)0.86 (0.75–0.98)
 ≥ 15 vs ≤ 110.85 (0.79–0.92)0.78 (0.67–0.91)
HRT use, years:
 0.1–1.0 vs 01.17 (1.09–1.26)1.01 (0.86–1.19)
 1.1–2.0 vs 01.27 (1.15–1.40)1.17 (0.97–1.40)
 2.1–3.0 vs 01.39 (1.24–1.56)1.37 (1.13–1.65)
 > 3.0 vs 01.55 (1.44–1.66)e1.53 (1.39–1.67)e
BMI, kg/m2:
 25.0–29.9 vs < 25.00.99 (0.92–1.07)1.02 (0.93–1.12)
 30.0–34.9 vs < 25.00.97 (0.85–1.10)1.14 (1.02–1.28)
 ≥ 35.0 vs < 25.01.12 (0.92–1.36)1.23 (1.07–1.41)
BMI* menopausec:
 1 vs 01.11 (1.01–1.23)
 2 vs 01.26 (1.07–1.47)
 3 vs 01.17 (0.91–1.50)
Alcohol intake, drinks per day:
 < 1.0 vs 01.00 (0.94–1.07)1.08 (0.91–1.29)
 1.0–1.9 vs 01.14 (1.05–1.24)1.20 (0.98–1.47)
 ≥ 2.0 vs 01.22 (1.12–1.33)1.26 (1.01–1.59)
FTPd:
 Yes vs no0.81 (0.71–0.91)0.97 (0.76–1.24)0.85 (0.67–1.08)0.65 (0.37–1.14)
Number of FTP:
 2 vs 10.99 (0.92–1.06)1.05 (0.90–1.22)1.14 (0.97–1.34)1.15 (0.78–1.70)
 ≥ 3 vs 10.87 (0.80–0.95)0.95 (0.81–1.13)0.99 (0.84–1.17)0.96 (0.65–1.41)
Age at 1st FTP, years:
 20.1–25.0 vs ≤ 20.01.05 (0.97–1.14)1.04 (0.89–1.23)1.03 (0.89–1.20)1.37 (0.97–1.95)
 25.1–30.0 vs ≤ 20.01.20 (1.10–1.31)0.93 (0.78–1.12)1.14 (0.97–1.33)1.34 (0.92–1.96)
 30.1–35.0 vs ≤ 20.01.32 (1.18–1.48)0.96 (0.75–1.23)1.59 (1.31–1.94)1.01 (0.58–1.75)
 > 3.05 vs ≤ 20.01.46 (1.24–1.73)e0.91 (0.59–1.38)1.56 (1.16–2.09)1.10 (0.48–2.53)
Height, per 10-cm increment1.19 (1.15–1.24)1.06 (0.98–1.16)1.14 (1.06–1.22)1.04 (0.89–1.22)

BMI body mass index, CI confidence interval, EPIC European Prospective Investigation into Cancer and Nutrition, ER estrogen receptor, FTP full-term pregnancy, HR hazard ratio, HRT hormone replacement therapy, WHI Women’s Health Initiative

aHeterogeneous risk associations among the EPIC women were examined using the likelihood ratio test. The resulting heterogeneous risk factor profiles were applied to the WHI women

bAge at menopause ≤ 45 years

c0: premenopausal or postmenopausal and BMI < 25 kg/m2; 1: postmenopausal and BMI 25.0–29.9 kg/m2; 2: postmenopausal and BMI 30.0–34.9 kg/m2; 3: postmenopausal and BMI ≥ 35 kg/m2. Among postmenopausal women, the HRs (95% CIs) for BMI from low to high categories were 1.11 (1.04–1.18), 1.21 (1.10–1.34), and 1.30 (1.11–1.53)

dThe number of FTP = 1 and age at first FTP ≤ 20 years.

eptrend < 0.001

Risk associations for ER+ and ER- tumors, the EPIC study and the WHI studya BMI body mass index, CI confidence interval, EPIC European Prospective Investigation into Cancer and Nutrition, ER estrogen receptor, FTP full-term pregnancy, HR hazard ratio, HRT hormone replacement therapy, WHI Women’s Health Initiative aHeterogeneous risk associations among the EPIC women were examined using the likelihood ratio test. The resulting heterogeneous risk factor profiles were applied to the WHI women bAge at menopause ≤ 45 years c0: premenopausal or postmenopausal and BMI < 25 kg/m2; 1: postmenopausal and BMI 25.0–29.9 kg/m2; 2: postmenopausal and BMI 30.0–34.9 kg/m2; 3: postmenopausal and BMI ≥ 35 kg/m2. Among postmenopausal women, the HRs (95% CIs) for BMI from low to high categories were 1.11 (1.04–1.18), 1.21 (1.10–1.34), and 1.30 (1.11–1.53) dThe number of FTP = 1 and age at first FTP ≤ 20 years. eptrend < 0.001 Tests for heterogeneity showed differential risk associations for FTP, number of FTPs, age at first FTP, body height, and country by ER status (Table 2 and Table 8 in Appendix). Parity (one single FTP, age at FTP ≤ 20 years) compared with nulliparity was associated with a statistically significant reduction in ER+ tumor risk (HR = 0.81, 95% CI = 0.71 to 0.91). Among parous women, having three or more FTPs was associated with a further risk reduction for ER+ tumors compared with one single FTP (HR = 0.87, 95% CI = 0.80 to 0.95), and delayed age at first FTP was associated with increased ER+ tumor risk (ptrend < 0.001). In addition, every 10-cm increment in body height was associated with a 19% increase in ER+ tumor risk (95% CI = 1.15 to 1.24). None of these factors, however, was statistically significantly associated with ER- tumor risk. Table 8 in Appendix shows the coefficients for different countries by ER status. Based on the same heterogeneous risk factor profiles, we also estimated the risk associations using the WHI data (Table 2), which were largely comparable to those from the EPIC study, with the exception of age at menarche, and especially for ER- tumors, FTP, number of FTP, and age at first FTP. Table 3 shows the predictive performance of the ER-specific models (C-statistic and E/O) corrected by the fivefold cross-validation. ModelER+, ModelER- and the omnibus model shared a C-statistic of 0.68. Elimination of the country effect reduced the C-statistic notably to 0.64 for ModelER+, 0.59 for ModelER-, and 0.63 for the omnibus model. A minor difference in C-statistic was observed between premenopausal and postmenopausal women. The omnibus model exhibited a higher C-statistic for ER+ than for ER- tumors (0.64 vs 0.59). ModelER+ significantly overestimated the 5-year tumor risk by 10% (E/O = 1.10, 95% CI = 1.05 to 1.14), particularly among premenopausal women (13%). ModelER- non-significantly underestimated the risk (E/O = 0.96, 95% CI = 0.88 to 1.05) overall and by menopausal status.
Table 3

Internal validation of the estrogen receptor (ER)-specific risk prediction models (ModelFR+ and ModelFR-) by fivefold cross-validation, overall and by age, in the women from the European Prospective Investigation into Cancer and Nutrition (EPIC) study

ModelER+ModelER-Omnibus model
C-statistic (95% CI)
 Before eliminating country effect0.68 (0.65–0.70)0.68 (0.64–0.72)0.68 (0.66–0.70)
 After eliminating country effect
  Overall0.64 (0.61–0.67)0.59 (0.54–0.64)0.63 (0.60–0.65)
  By menopausal status
   Premenopausal0.64 (0.59–0.68)0.58 (0.51–0.66)0.62 (0.59–0.66)
   Postmenopausal0.62 (0.59–0.66)0.60 (0.52–0.67)0.62 (0.59–0.65)
  By ER status
   ER+0.64 (0.62–0.67)
   ER−0.59 (0.53–0.64)
Ratio of observed–expected (95% CI)
 Overall1.10 (1.05–1.14)0.96 (0.88–1.05)1.07 (1.03–1.11)
 By menopausal status
  Premenopausal1.13 (1.06–1.20)0.97 (0.85–1.10)1.09 (1.02–1.15)
  Postmenopausal1.07 (1.02–1.13)0.96 (0.84–1.08)1.06 (1.00–1.11)
Internal validation of the estrogen receptor (ER)-specific risk prediction models (ModelFR+ and ModelFR-) by fivefold cross-validation, overall and by age, in the women from the European Prospective Investigation into Cancer and Nutrition (EPIC) study External validation with the WHI data resulted in a C-statistic of 0.59 (95% CI = 0.58 to 0.60) for ModelER+ and 0.53 (95% CI = 0.50 to 0.57) for ModelER- (Table 4). ModelGail yielded an overall C-statistic of 0.57 (95% CI = 0.56 to 0.59) with a markedly lower C-statistic of 0.53 (95% CI = 0.50 to 0.57) for ER- tumors. Regarding calibration, an overestimation was observed for ER+ tumors (E/O = 1.09, 95% CI = 1.03 to 1.14) whereas a statistically non-significant underestimation was observed for ER- tumors (E/O = 0.94, 95% CI = 0.82 to 1.06). ModelGail underestimated the overall BC risk by 22% (E/O = 0.78, 95% CI = 0.73 to 0.82). Among the EPIC women, the overestimation of ER+ tumor risk occurred largely in low-risk individuals (Fig. 1a); for ER- tumor risk, overestimation was observed mainly among low-risk individuals whereas underestimation was observed mainly among high-risk individuals (Fig. 1b). Among WHI women, the overestimation by ModelER+ and the underestimation by ModelGail were largely systematic (Fig. 1c and e). The statistically non-significant underestimation by ModelER- in the WHI women showed no clear pattern (Fig. 1d).
Table 4

External validation of the estrogen receptor (ER)-specific risk prediction models and the Gail model in women from the Women’s Health Initiative (WHI) study

ER-specific risk prediction modelsModelGail
ModelER+ModelER−
C-statistic (95% CI)
 Overall0.59 (0.58–0.60)0.53 (0.50–0.57)0.57 (0.56–0.59)
 By ER status
  ER+0.58 (0.57–0.60)
  ER−0.53 (0.50–0.57)
Ratio of observed–expected (95% CI)1.09 (1.03–1.14)0.94 (0.82–1.06)0.78 (0.73–0.82)
Fig. 1

Calibration of the risk prediction model of ER-positive tumors (ModelER+), risk prediction model of ER-negative tumors (ModelER-), and Gail risk prediction model (ModelGail) by risk deciles. a ModelER+ in women from the European Prospective Investigation into Cancer and Nutrition (EPIC); b ModelER- in the EPIC women; c ModelER+ in the women from the Women’s Health Initiative (WHI); d ModelER- in the WHI women; e ModelGail in the WHI women

External validation of the estrogen receptor (ER)-specific risk prediction models and the Gail model in women from the Women’s Health Initiative (WHI) study Calibration of the risk prediction model of ER-positive tumors (ModelER+), risk prediction model of ER-negative tumors (ModelER-), and Gail risk prediction model (ModelGail) by risk deciles. a ModelER+ in women from the European Prospective Investigation into Cancer and Nutrition (EPIC); b ModelER- in the EPIC women; c ModelER+ in the women from the Women’s Health Initiative (WHI); d ModelER- in the WHI women; e ModelGail in the WHI women Figure 2 shows the net benefit curves of ModelER+ and ModelGail. The net benefit curves of the two models started to diverge from the treat-all strategies at the risk threshold of 0.55%, which was roughly the minimal risk projected by both models. ModelER+ would yield higher net benefits than both the treat-all strategy and the treat-none strategy (denoted by the x-axis at y = 0) if the risk threshold lay between 0.55% and 2.5%, corresponding to an assumption that the benefit of chemoprevention was 180 to 40 times the harm. In contrast, ModelGail would yield lower net benefits than the treat-all strategy if a risk threshold below 2% were selected, including 1.67%, the currently adopted risk threshold for chemoprevention in the USA, and would yield negative net benefits if a risk threshold above 4% (i.e. benefit ≈ 25 × harm) were selected. The clinical applicability of ModelGaili, as indicated by the sum of Area A and Area B shown in Fig. 2, was 3.0 × 10− 6. The clinical applicability of ModelER+ was 12.7 × 10− 6 (Area C).
Fig. 2

Net benefit curves for the risk prediction model of ER-positive tumors (ModelER+) (black solid line) and the Gail risk prediction model (ModelGail) (black broken line) applied to women from the Women’s Health Initiative study. Corresponding curves for the treat-all strategy are represented in gray (solid line for all breast cancer cases, broken line for ER+ tumors only). Area A = − 7.84 × 10-6; Area B = 1.08 × 10-5; Area C = 1.27 × 10-5

Net benefit curves for the risk prediction model of ER-positive tumors (ModelER+) (black solid line) and the Gail risk prediction model (ModelGail) (black broken line) applied to women from the Women’s Health Initiative study. Corresponding curves for the treat-all strategy are represented in gray (solid line for all breast cancer cases, broken line for ER+ tumors only). Area A = − 7.84 × 10-6; Area B = 1.08 × 10-5; Area C = 1.27 × 10-5

Discussion

The heterogeneous risk associations in our ER-specific risk prediction models are consistent with the established knowledge that FTP, number of FTPs, and delayed childbirth are associated with ER+ tumors but not with ER- tumors [6-8]. Our study also confirms a largescale meta-analysis of epidemiological data showing that BC risk increases with prolonged duration of HRT use [21]. Data from the WHI randomized trial showed a statistically significant increase in the incidence and mortality of invasive BC in the estrogen-plus-progestin arm compared with the placebo arm [22, 23], whereas estrogen alone decreased BC incidence and mortality among postmenopausal women with prior hysterectomy [24, 25]. Stronger positive associations for estrogen plus progestin than for estrogen alone were reported for BC [26, 27]. In the present study, we could not separate estrogen alone and estrogen plus progestin due to unknown HRT compounds among former users in EPIC. Among current HRT users at baseline, use of estrogen plus progestin was more common in EPIC than in the WHI cohort (76% vs 44%, respectively). However, similar associations between the duration of lifetime HRT use and BC risk were observed in both the EPIC and the WHI study. In ER-specific risk models, statistically significant and homogeneous risk associations were fitted for age at menopause and age at menarche, in line with a pooled analysis of previous investigations where nearly identical effects were observed for ER+ tumors and ER- tumors [28]. The present study demonstrated a null association between breastfeeding and BC risk, inconsistent with previous investigations where inverse associations were reported [6, 8, 29]. We note that most previous studies were case-control studies, which were subject to recall bias. In fact, the inverse association disappeared in some cohort studies [30, 31]. In a more recent pooled analysis, breastfeeding was not associated with ER+ and/or PR+ tumors but was inversely associated with ER-/PR- tumors [32]. In a pooled analysis of prospective cohort data, every 10-cm increment in body height was statistically significantly associated with ER+ tumor risk (HR = 1.18) but had null association with ER- tumor risk [33], supporting the way we modeled body height in the present study. Prediction of ER+ tumor risk might be practically more useful than prediction of overall BC risk [3]. The reason for this is twofold. First, projecting subtype-specific risks allows for accurate estimation of the risk associations of factors that are etiologically heterogeneous and as a result might increase the discriminatory power. Second, since currently used chemoprevention only reduces the risk of ER+ tumors [34], there is a need for a model that can specifically predict the risk of developing ER+ tumors. The discriminatory accuracy of ModelER+ in internal validation performed no better than most of the current omnibus models using questionnaire-derived data, suggesting limited improvement in discrimination after accounting for etiological heterogeneity. This was not surprising given that ER+ tumors are the dominant subtype and the omnibus model shared nearly equivalent parameters (data not shown) with ModelER+ in the present study. According to the only study so far that has modeled ER-specific risks, the discriminatory power of the ER+/PR+ model was moderately higher than that of the ER-/PR- model (0.64 vs 0.61) [10]. In that study, risk factors with heterogeneous associations included age, menopausal status, BMI, age at first birth, and past use of postmenopausal HRT, and its subtype-specific models were based on a relatively small number of tumors (1281 ER+/PR+ tumors, 417 ER-/PR- tumors). Notably, in that study there was no correction for potential overfitting by either internal or external approaches. When externally validated in the WHI cohort, ModelER+ exhibited moderate discriminatory accuracy comparable to that of ModelGail. Women in the USA with 5-year BC risk of 1.67% or higher, projected by ModelGail, are considered potentially eligible for chemoprevention [35]. This risk threshold would lead to coverage of 36,265 (44.0%) women in our WHI validation population, of whom 1239 were subsequently diagnosed with ER+ tumors and 194 with ER- tumors. According to ModelER+, a risk threshold of 1.97% would cover the same number of women with 16 more prospective ER+ tumors and 2 fewer prospective ER- tumors. The decision curve analysis provided some interesting insight into the clinical applicability of ModelER+ and ModelGail. As indicated by the net benefit curves, ModelER+ would demonstrate no advantage over the treat-all strategy if the benefit-to-harm ratio of chemoprevention were higher than 180, equivalent to any risk threshold below the minimal risk projection (≈ 0.55%), while such a boundary benefit-to-harm ratio was 50 for ModelGail. Interestingly, the treat-all strategy would even outperform ModelGail when the risk threshold was situated at 1.67%. In contrast to ModelGail, ModelER+ had a wider threshold range where higher net benefits could be obtained by a model-based decision-making than by either the treat-all or the treat-none strategy. Considering the unknown benefit and harm associated with chemoprevention, ModelER+ thus has broader applicability than ModelGail, as indicated by the areas formed by the two models’ net benefit curves and the two extreme strategies. As shown in Fig. 2, the lowest benefit-to-harm ratio for chemoprevention against BC to produce a positive net benefit is 25, whereas such a benefit-to-harm ratio for chemoprevention against ER+ tumors is 40, suggesting that chemoprevention against ER+ tumors might be 1.6 times (40/25) more efficient than chemoprevention against all types of BC. Among both the EPIC women and the WHI women, ModelER+ overestimated the 5-year risk by about 10%, possibly due to potential misspecifications of our models, such as imperfect fit of the baseline hazard functions (the baseline hazard estimates are given in Table 9 in the Appendix). More importantly, this overestimation was systematic rather than in an overfitting pattern, i.e. underestimation occurs in low-risk individuals and overestimation occurs in high-risk individuals [36]. We derived ER-specific models from a large prospective cohort and validated them in another large independent cohort for external validation. This is a strong approach to robust parameterization and assessment of model performance. However, some limitations characterize the present study. Our models did not include some established risk factors such as family history of BC (FHBC) and previous breast biopsy, as these variables were not available in the EPIC study. A complete-case analysis of EPIC women with known FHBC (n = 138,257, 49% of the sample) showed positive homogenous associations between FHBC and tumor subtypes (HRER+ = 1.64, 95% CI = 1.49 to 1.81; HRER- = 1.50, 95% CI = 1.23 to 1.91; pheterogeneity = 0.57), suggesting that inclusion of this factor would increase the predictive power of the model, though not differentially across the hormonal receptor status of the tumors. Another limitation of our study was the underestimation of baseline hazards due to EPIC tumors with unknown ER status, which accounted for about 25% of BC diagnoses. Under the assumption of ER-status data missing at random, parameter estimates are expected to be unbiased, a necessary requisite to carry out proper external validation, whereas the underestimated baseline hazard would be replaced with the actual baseline hazard function of the test population.

Conclusions

In summary, we found that modeling heterogeneous risk associations of epidemiological factors yields little improvement in BC risk prediction. Nevertheless, compared with the current omnibus models, a model specifically predictive of ER+ tumor risk could be more applicable in risk assessment for chemoprevention.
Table 5

Coding for predictor variables

Predictor variableCoding
Menopausal status0: premenopausal1: postmenopausal
Age at menopause0: premenopausal or ≤ 45 years1: 45.1–50.0 years2: 50.1–55.0 years3: > 55.0 years
Breastfeeding0: parous women who never breastfed andnulliparous women1: 0.1–6 months2: 6.1–12 months3: > 12 months
Age at menarche0: ≤ 11 years1: 12 years2: 13 years3: 14 years4: ≥ 15 years
Hormone replacement therapy (HRT) use0: never used HRT1: 0.1–1 year2: 1.1–2 years3: 2.1–3 years4: > 3 years
Full-term pregnancy (FTP)0: nulliparous1: parous
Number of FTPs0: nulliparous or 1 FTP1: 2 FTPs2: ≥ 3 FTPs
Age a first FTP0: nulliparous or ≤ 20 years1: 20.1–25.0 years2: 25.1–30.0 years3: 30.1–35.0 years4: > 35.0 years
Body mass index0: < 25.0 kg/m21: 25–29.9 kg/m22: 30–34.9 kg/m23: > 35.0 kg/m2
Body height per 10-cm incrementBody height in centimeters divided by 10, continuous variable
Alcohol intake0: non-drinker1: < 1 drink/day2: 1.0–1.9 drinks/day3: ≥ 2 drinks/day
Table 6

Distributions of the risk factors among the EPIC women by country

FranceItalySpainUKNetherlandsGreeceGermanySwedenDenmarkNorwayTotal
n = 68,707n = 27,851n = 20,298n = 35,349n =22,601n = 11337n = 22,085n = 9142n = 29,309n = 3,4651n = 281,330
Menopausal status:
 Premenopausal54.049.454.046.439.135.048.661.722.669.548.5
 Postmenopausal46.050.646.053.660.965.051.438.377.430.551.5
Age at menopause, years:
 ≤ 45.09.719.126.119.522.630.718.310.616.810.417.2
 45.1–50.020.443.939.129.235.636.033.632.733.522.530.9
 50.1–55.021.728.328.022.229.226.226.218.227.59.924.1
 > 55.03.52.52.83.23.22.53.03.13.90.03.0
 Missinga44.76.24.0269.44.62035.418.357.224.8
FTP:
 0 (no)9.012.19.815.513.28.613.54.511.68.510.9
 1 (yes)83.886.689.480.885.590.886.257.488.091.585.2
 Missing7.21.30.83.71.30.60.338.10.40.03.9
Number of FTP:
 118.424.99.115.66.911.329.613.817.713.316.8
 249.449.937.449.931.355.249.247.451.449.247.5
 ≥ 332.225.253.534.333.333.421.237.930.937.533.3
 Missingb0.00.00.00.228.50.10.00.90.00.02.4
Age at 1st FTP, years:
 ≤ 209.88.710.411.810.023.320.217.123.621.514.6
 20.1–25.053.143.652.242.245.141.547.345.546.445.947.3
 25.1–30.027.834.630.331.834.825.023.125.922.523.628.1
 31.1–35.07.010.25.510.17.96.87.16.25.06.87.4
 > 35.02.02.71.53.62.22.32.21.61.32.22.2
 Missingb0.30.20.10.50.01.10.13.71.20.00.4
Breastfeeding, months:
 027.716.811.215.813.611.116.906.95.715.5
 0.1–6.047.239.633.040.134.631.259.6043.826.539.7
 6.1–12.014.023.421.116.213.116.213.8026.124.018.0
 > 12.05.420.034.422.69.840.39.2021.042.219.7
 Missingb5.70.20.35.328.91.20.51002.21.67.1
Age at menarche, years:
 ≤ 1117.023.218.119.210.79.710.86.18.98.514.6
 1224.625.019.618.119.922.521.912.013.619.820.7
 1326.024.323.823.923.925.925.517.922.428.725.0
 1420.617.622.419.021.820.123.015.125.224.421.1
 ≥ 1511.28.516.114.321.620.718.611.426.217.115.6
 Missing0.61.40.05.52.11.10.237.53.71.53.0
HRT:
 1 (yes)31.017.211.331.722.55.638.514.043.93128.0
 0 (no)68.381.185.764.974.593.825.745.454.26966.4
 Missing0.71.73.03.43.00.635.840.61.905.6
Duration of HRT, years:
 ≤ 1.026.455.262.030.038.062.021.619.227.918.329.2
 1.1–2.021.011.717.114.113.813.511.312.310.213.314.8
 2.1–3.014.08.36.610.89.35.910.37.68.210.010.6
 > 3.038.617.112.037.931.816.850.720.552.730.138.0
 Missingc0.07.72.37.27.11.86.140.41.028.37.4
Height, cm, median161158156162164156163164164167162
BMI, kg/m2:
 < 25.077.246.921.158.250.620.947.457.351.164.256.0
 2.05–29.918.537.543.730.336.438.434.631.534.627.730.4
 30.0–34.93.511.825.38.510.127.813.18.310.66.410.1
 ≥ 35.00.83.89.93.02.912.94.92.93.71.73.5
Alcohol intake, drinks/day:
 014.022.453.37.417.135.34.616.42.721.017.0
 < 1.054.446.634.269.556.457.270.882.958.277.559.8
 1.0–1.918.016.68.510.914.65.414.90.720.81.513.0
 ≥ 2.013.613.14.06.411.41.69.6018.209.3
 Missing01.305.80.50.50.100.100.9

Values are percentages unless otherwise indicated

BMI body mass index, EPIC European Prospective Investigation into Cancer and Nutrition, FTP full-term pregnancy, HRT hormone replacement therapy

aAmong postmenopausal women

b‡Among parous women (FTP = 1)

cAmong HRT users

Table 7

Distributions of the risk factors among the WHI women by ER status

Tumor-freeER+ER-Indefinite
n = 79,368n = 2276n = 421n = 254
Age at menopause, years:
 ≤ 4526.822.825.625.6
 45.1–50.034.431.938.035.8
 50.1–55.030.334.930.028.7
 > 55.08.510.46.49.9
Breastfeeding, months:
 047.945.648.750.8
 0.1–6.025.826.526.126.0
 6.1–12.011.412.58.811.0
 > 12.014.915.416.412.2
Age at menarche, years:
 ≤ 1121.524.023.022.4
 1226.226.228.323.6
 1330.228.528.333.1
 1413.113.311.913.0
 ≥ 159.08.08.57.9
HRT use, years:
 033.327.030.431.5
 0.1–1.08.56.68.38.7
 1.1–2.05.55.34.56.7
 2.1–3.04.44.84.34.3
 > 3.048.356.352.548.8
BMI, kg/m2:
 < 25.038.837.842.329.5
 25.0–29.935.235.230.133.9
 30–34.916.717.118.821.6
 ≥ 35.09.39.98.815.0
Alcohol intake, drinks/day:
 05.65.14.35.9
 < 1.077.775.48.076.0
 1.0–1.911.212.712.413.4
 ≥ 2.05.46.80.34.7
FTP:
 0 (no)10.310.011.27.1
 1 (yes)89.790.088.892.9
Number of FTP:
 19.59.78.812.3
 230.233.234.527.5
 ≥ 360.357.156.760.2
Age at 1st FTP, years:
 ≤ 20.013.011.09.916.5
 20.1–25.049.445.652.444.9
 25.1–30.028.530.030.228.8
 30.1–35.07.010.35.68.5
 > 35.02.13.11.81.3
Height, cm, mean162.2162.6162.4162.9

Values are percentages unless otherwise indicated

BMI body mass index, ER estrogen receptor, FTP full-term pregnancy, HRT hormone replacement therapy, WHI Women’s Health Initiative

Table 8

Coefficients for different countries by ER status in the women from the EPIC study

ModelER+ModelER−
Coefficient P Coefficient P
France00
Italy−0.186< 0.001−0.397< 0.001
Spain−0.791< 0.001−0.806< 0.001
UK−0.582< 0.001−0.842< 0.001
Netherlands−0.513< 0.001−0.762< 0.001
Greece−1.610< 0.001−2.023< 0.001
Germany−0.214< 0.001−0.2110.035
Sweden−0.473< 0.001−0.2990.024
Denmark−0.179< 0.001−0.0820.302
Norway−1.006< 0.001−1.244< 0.001

Model risk prediction model of ER-positive tumors, ModelER- risk prediction model of ER-negative tumors, EPIC European Prospective Investigation into Cancer and Nutrition, ER estrogen receptor

Table 9

Baseline piecewise constant hazard estimates for postmenopausal ER+ tumors and ER- tumors in the women from the EPIC and the WHI studiesa

Age groups, years
40.0–44.945.0–49.950.0–54.955.0–59.960.0–64.965.0–69.970.0–74.9≥ 75.0
EPIC study
 ER+3088152189227239233186
 ER−1935444342414031
WHI study
 ER+126173185237243270
 ER−485043503950

Hazards are presented as number of tumors per 105 person-years

EPIC European Prospective Investigation into Cancer and Nutrition, WHI Women’s Health Initiative, ER estrogen receptor

aRisk factor profiles for baseline hazard: country = France (only for EPIC), menopausal status = 1 (postmenopausal), age at menopause ≤ 45 years, duration of breastfeeding = 0, age at menarche ≤ 11 years, duration of hormone replacement therapy = 0, full-term pregnancy = 0 (nulliparous), body mass index < 25 kg/m2, height_d10 (body height in cm divided by 10) = 16, alcohol intake (drinks/day) = 0

  35 in total

1.  Outcomes ascertainment and adjudication methods in the Women's Health Initiative.

Authors:  J David Curb; Anne McTiernan; Susan R Heckbert; Charles Kooperberg; Janet Stanford; Michael Nevitt; Karen C Johnson; Lori Proulx-Burns; Lisa Pastore; Michael Criqui; Sandra Daugherty
Journal:  Ann Epidemiol       Date:  2003-10       Impact factor: 3.797

2.  Multiple imputation by chained equations: what is it and how does it work?

Authors:  Melissa J Azur; Elizabeth A Stuart; Constantine Frangakis; Philip J Leaf
Journal:  Int J Methods Psychiatr Res       Date:  2011-03       Impact factor: 4.035

3.  Validation of the Gail et al. model of breast cancer risk prediction and implications for chemoprevention.

Authors:  B Rockhill; D Spiegelman; C Byrne; D J Hunter; G A Colditz
Journal:  J Natl Cancer Inst       Date:  2001-03-07       Impact factor: 13.506

4.  Reproductive history and oral contraceptive use in relation to risk of triple-negative breast cancer.

Authors:  Amanda I Phipps; Rowan T Chlebowski; Ross Prentice; Anne McTiernan; Jean Wactawski-Wende; Lewis H Kuller; Lucile L Adams-Campbell; Dorothy Lane; Marcia L Stefanick; Mara Vitolins; Geoffrey C Kabat; Thomas E Rohan; Christopher I Li
Journal:  J Natl Cancer Inst       Date:  2011-02-23       Impact factor: 13.506

5.  Effects of conjugated equine estrogens on breast cancer and mammography screening in postmenopausal women with hysterectomy.

Authors:  Marcia L Stefanick; Garnet L Anderson; Karen L Margolis; Susan L Hendrix; Rebecca J Rodabough; Electra D Paskett; Dorothy S Lane; F Allan Hubbell; Annlouise R Assaf; Gloria E Sarto; Robert S Schenken; Shagufta Yasmeen; Lawrence Lessin; Rowan T Chlebowski
Journal:  JAMA       Date:  2006-04-12       Impact factor: 56.272

6.  Decision curve analysis: a novel method for evaluating prediction models.

Authors:  Andrew J Vickers; Elena B Elkin
Journal:  Med Decis Making       Date:  2006 Nov-Dec       Impact factor: 2.583

7.  Assessing the performance of prediction models: a framework for traditional and novel measures.

Authors:  Ewout W Steyerberg; Andrew J Vickers; Nancy R Cook; Thomas Gerds; Mithat Gonen; Nancy Obuchowski; Michael J Pencina; Michael W Kattan
Journal:  Epidemiology       Date:  2010-01       Impact factor: 4.822

Review 8.  Systematic review: comparative effectiveness of medications to reduce risk for primary breast cancer.

Authors:  Heidi D Nelson; Rongwei Fu; Jessica C Griffin; Peggy Nygren; M E Beth Smith; Linda Humphrey
Journal:  Ann Intern Med       Date:  2009-11-17       Impact factor: 25.391

9.  European Prospective Investigation into Cancer and Nutrition (EPIC): study populations and data collection.

Authors:  E Riboli; K J Hunt; N Slimani; P Ferrari; T Norat; M Fahey; U R Charrondière; B Hémon; C Casagrande; J Vignat; K Overvad; A Tjønneland; F Clavel-Chapelon; A Thiébaut; J Wahrendorf; H Boeing; D Trichopoulos; A Trichopoulou; P Vineis; D Palli; H B Bueno-De-Mesquita; P H M Peeters; E Lund; D Engeset; C A González; A Barricarte; G Berglund; G Hallmans; N E Day; T J Key; R Kaaks; R Saracci
Journal:  Public Health Nutr       Date:  2002-12       Impact factor: 4.022

Review 10.  Breastfeeding and breast cancer risk by receptor status--a systematic review and meta-analysis.

Authors:  F Islami; Y Liu; A Jemal; J Zhou; E Weiderpass; G Colditz; P Boffetta; M Weiss
Journal:  Ann Oncol       Date:  2015-10-26       Impact factor: 32.976

View more
  6 in total

1.  Assessment of breast cancer risk: which tools to use?

Authors:  Montserrat Garcia-Closas; Nilanjan Chatterjee
Journal:  Lancet Oncol       Date:  2019-02-21       Impact factor: 41.316

2.  Prospective evaluation of a breast-cancer risk model integrating classical risk factors and polygenic risk in 15 cohorts from six countries.

Authors:  Amber N Hurson; Parichoy Pal Choudhury; Chi Gao; Anika Hüsing; Mikael Eriksson; Min Shi; Michael E Jones; D Gareth R Evans; Roger L Milne; Mia M Gaudet; Celine M Vachon; Daniel I Chasman; Douglas F Easton; Marjanka K Schmidt; Peter Kraft; Montserrat Garcia-Closas; Nilanjan Chatterjee
Journal:  Int J Epidemiol       Date:  2021-03-23       Impact factor: 9.685

3.  Validation of two US breast cancer risk prediction models in German women.

Authors:  Anika Hüsing; Anne S Quante; Jenny Chang-Claude; Krasimira Aleksandrova; Rudolf Kaaks; Ruth M Pfeiffer
Journal:  Cancer Causes Control       Date:  2020-04-06       Impact factor: 2.506

4.  Risk Factors Associated with Breast Cancer among Women in Addis Ababa, Ethiopia: Unmatched Case-Control Study.

Authors:  Lidia Tolessa; Endalew Gemechu Sendo; Negalign Getahun Dinegde; Assefa Desalew
Journal:  Int J Womens Health       Date:  2021-01-18

5.  Lipid exposure activates gene expression changes associated with estrogen receptor negative breast cancer.

Authors:  Shivangi Yadav; Ranya Virk; Carolina H Chung; Mariana Bustamante Eduardo; David VanDerway; Duojiao Chen; Kirsten Burdett; Hongyu Gao; Zexian Zeng; Manish Ranjan; Gannon Cottone; Xiaoling Xuei; Sriram Chandrasekaran; Vadim Backman; Robert Chatterton; Seema Ahsan Khan; Susan E Clare
Journal:  NPJ Breast Cancer       Date:  2022-05-04

6.  Risk assessment and prediction model of renal damage in childhood immunoglobulin A vasculitis.

Authors:  Ruqian Fu; Manqiong Yang; Zhihui Li; Zhijuan Kang; Mai Xun; Ying Wang; Manzhi Wang; Xiangyun Wang
Journal:  Front Pediatr       Date:  2022-08-17       Impact factor: 3.569

  6 in total

北京卡尤迪生物科技股份有限公司 © 2022-2023.