Derivation and validation of a prediction model for neonate unplanned rehospitalization in a tertiary center in China.

BACKGROUND: To construct and externally validate a prediction model for neonate unplanned rehospitalization within 31 days of discharge.
METHODS: A retrospective study was performed in the Department of Neonatology of the Children's Hospital of Fudan University. A binominal regression method was applied to construct and validate the prediction model. Analysis was performed on a total of 11,116 neonates with an index admission between 11/1/2016 and 12/31/2018. Neonates admitted from 11/1/2016 to 1/31/2018 were used for the selection of prognostic variables and construction of the model. Model validation was then performed with neonates admitted from 2/1/2018 to 12/31/2018.
RESULTS: The rehospitalization rate for neonates was 3.27% (373/11,116). A total of 512 neonates were enrolled for the construction of the prediction model. Gestational age (GA), NICU length of stay (LOS), nonmedical order discharge and younger maternal age were strongly correlated with rehospitalization. By incorporating these 4 strong risk factors, we constructed a model to predict neonate unplanned rehospitalization within 31 days of discharge. The formula was turned into a nomogram for use in clinical practice. The nomogram has a total score of 180, with a predicted risk from 0 to 100%. Neonates are at high risk for rehospitalization if they have a total score greater than 39 points, according to the cutoff point established by the Youden index. The model was shown to have good discriminatory ability, with area under the receiver operating characteristic curves of 0.68 and 0.65 in the model construction and validation datasets, respectively. A total of 39 points is the cutoff for follow-up.
CONCLUSIONS: The model is able to predict neonate unplanned rehospitalization well. A total score greater than 39 indicates that follow-up is necessary. 2021 Translational Pediatrics. All rights reserved.

Introduction

Recently, rehospitalization rates have been used as an indicator of the quality of care that patients received during hospital admission and after discharge (1-3). The definition of rehospitalization varies across studies. The most commonly used definition of rehospitalization is rehospitalization between 28 and 31 days after discharge for unplanned reasons (4-8). Rehospitalization rates for children aged 0–18 years range from 3.4–28.6% (5,9-12). There is a paucity of studies addressing neonate rehospitalizations. Gay et al. found that 11.4% of the rehospitalizations in the Monroe Carell Jr Children’s Hospital in Vanderbilt involved neonates, second to patients aged >10 years (13).

Unplanned rehospitalization within a short period of time is potentially preventable, and studies have indicated that the proportion of preventable rehospitalizations among all rehospitalizations ranges from 9% to 59% (5,9-12,14). Preventable rehospitalizations result in unnecessary hospitalization, an increased risk of nosocomial infections, additional costs, the waste of medical resources, increased stress in both neonates and mothers, delayed breast feeding, etc. (11,12,15-18). By analyzing possible factors affecting rehospitalization, additional support and medication could be given to neonates at a higher risk of rehospitalization, reducing the likelihood that those neonates would be rehospitalized.

Several studies have investigated the epidemiology of and risk factors for neonate rehospitalization, including sex, race, ethnicity, gestational age (GA), birth weight (BW), duration of index hospitalization, whether they are covered by Medicaid or private insurance, and maternal conditions (18-22). However, few researchers have studied the risk factors for neonate unplanned rehospitalization within a tertiary center, so these factors have not been included in clinical prediction models. To reduce the rehospitalization of neonates, we sought to build a neonate rehospitalization model. The findings will provide a basis for the assessment, development and improvement of neonatal clinical management, hopefully reducing the rate of neonate rehospitalization.

We present the following article in accordance with the TRIPOD reporting checklist (available at http://dx.doi.org/10.21037tp-20-184).

Methods

Study design, setting, and population

We developed a binominal regression model with unplanned rehospitalization as an outcome. Data were collected from the inpatient medical records from the Children’s Hospital of Fudan University (CHOF). Neonates who were admitted to the Department of Neonatology of CHOF between 11/1/2016 and 12/31/2018 were included. The dataset was divided into the model construction dataset and the validation dataset according to admission date. The subjects in the model construction dataset were neonates hospitalized from November 2016 to January 2018, while the validation dataset subjects were neonates hospitalized from February 2018 to December 2018. Both the model construction dataset and the validation dataset were divided into the rehospitalization group and the control group.

The inclusion criterion for the rehospitalization group was unplanned rehospitalization. Unplanned rehospitalization was defined as a second admission of a neonate to the Department of Neonatology of CHOF within 31 days after the discharge date of an index admission for unplanned reasons. The exclusion criteria included planned rehospitalization; hospitalization in a department other than the Department of Neonatology; transfer to a ward other than the Department of Neonatology; death; rehospitalization more than 31 days after the discharge date; and more than two hospitalizations.

Neonates who were admitted to the Department of Neonatology of CHOF between 11/1/2016 and 12/31/2018 were included as the control group, excluding neonates who were rehospitalized in either the neonatal inpatient ward or other pediatric inpatient wards within 31 days after discharge, neonates with more than one inpatient event, and those who died before discharge. Random coding was applied for the selection of the control group. The number of neonates chosen as the control group for the model construction dataset was 1.5 times the number of neonates with unplanned rehospitalizations, and 150 neonates were randomly selected for the validation dataset. A flow chart of the subjects is represented in .

Figure 1

Flow chart of the study subjects for neonate rehospitalization. CHOF, Children’s Hospital of Fudan University.

Main outcome measures

Unplanned rehospitalizations for any reason were measured because patients could be readmitted for related conditions even if the index admission and rehospitalization diagnoses were different.

Factors inclusion

The literature on neonatal risk factors associated with rehospitalization was examined. A total of 37 risk factors were identified. Based on their availability and clinical adaptability, 21 risk factors were selected for the initial stage of model development. The 21 risk factors were as follows: sex, GA, BW, blood culture positivity, the maximum total serum bilirubin (TSB) level during hospitalization, the procalcitonin (PCT) level, the use of antibiotics during hospitalization, antibiotic use duration, respiratory support duration, respiratory support during hospitalization, blood transfusion during hospitalization, invasive operation during hospitalization, neonatal intensive care unit (NICU) length of stay (LOS), nonmedical order discharge, hospital LOS, 5-minute Apgar score, maternal age, socioeconomic status and hospitalization fees. Risk factors with a more than 20% missing data were excluded. Data for included risk factors were collected from the inpatient medical records from the CHOF.

Sample size

The adequacy of the sample size (n) for the development of the predictive model was ensured by employing the required sample size equation (Equation 1). Overfitting was minimized with the combination of the predictor parameters (P), the coefficient of determination and the Van Houwelingen shrinkage factor (SVH). The coefficient of determination was set as 0.1, while the SVH was set as 0.9.

Statistical analyses

Continuous variables are expressed as the means (SDs) and were compared using an unpaired, 2-tailed t-test or the Mann-Whitney test. Categorical variables were compared using the χ2 test. P<0.05 was considered statistically significant for the analyses. The significance of each variable in the model construction dataset was assessed with a binominal regression model. Univariate regression analysis was used to evaluate the predictive accuracy of each variable in the prediction model.

A nomogram was formulated for use in clinical practice based on the results of binominal regression with R version 3.5.2. Each regression coefficient in binominal logistic regression was converted proportionally to a 0- to 100-point scale. The effect of the variable with the highest β coefficient (absolute value) is assigned 100 points. The points are added across independent variables to derive the total points, which are converted to the predicted probability. The predictive performance of the nomogram was measured by the concordance index (C-index) and calibration with bootstrap samples was performed to decrease the overfit bias. Receiver operating characteristic (ROC) curve analysis was used to calculate the optimal cutoff values that maximized the Youden index.

Validation

To validate the model derived from the model construction dataset, we applied the scoring method to the validation dataset (N=311). C-statistics were used for validation.

Ethical considerations

The study involving human participants were reviewed and approved by the Ethical Committee of Children’s Hospital, Fudan University [No. (2018)213]. Informed consents for participation were obtained from parents or legal guardians of the patients in this study. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

Results

Study population and sample

There were 11,116 hospitalization events for neonates in the CHOF between 11/1/2016 and 12/31/2018. The model construction dataset included 6,526 hospitalization events, with 206 rehospitalization events. The validation dataset included 4,590 hospitalization events, with 167 rehospitalization events. Four rehospitalization events from the model construction dataset and 6 rehospitalization events from the validation dataset were excluded according to the exclusion criteria. The unplanned rehospitalization rates in the model construction dataset and the validation dataset were 3.10% vs. 3.51%, respectively, and the overall unplanned rehospitalization rate was 3.27%.

In the model construction dataset, the rehospitalization group and the control group had a median GA of 35.8 vs. 37.49 weeks (P<0.001) and a median BW of 2,578.72 vs. 2,923.65 grams (P<0.001). The proportions of males in the rehospitalization group (n=202) and the control group (n=310) were 61.65% and 56.13% (P=0.271), respectively. The median LOS for the two groups was 8.5 days (P=0.558). The median NICU LOS for the rehospitalization group was 13 days (IQR: 4–44.75), while it was 6 days (IQR: 3–15.5) for the control group (P<0.001). The rehospitalization group and the control group median maternal ages of 29.59 and 29.83 years (P=0.349) and nonmedical discharge rates of 44% vs. 39% (P=0.018), respectively.

In contrast with the model construction dataset, the validation dataset had a median GA of 36 weeks (P=0.557) and a median BW of 2,780 grams (P=0.866). The median LOS was 7.5 days (IQR: 4.5–15) in the neonatal inpatient ward (P=0.082) and 9 days (IQR: 3–29) in the NICU (P=0.675). The median maternal age for the validation set was 29.87 years (P=0.483), with a nonmedical discharge rate of 66% (P=0.032) ().

Table 1

Characteristics of rehospitalized neonates

Characteristics	P	AUC
Sex	0.27	0.46
Male
Female
GA (weeks)	<0.001	0.6
BW (g)	<0.001	0.58
5-minute Apgar score
NICU hospitalization (count)	<0.001	0.58
NICU LOS (days)	0.00	0.57
Antibiotic use (count)	0.34	0.51
Antibiotic duration (days)	0.50	0.56
Antibiotic types (count)	0.01	0.55
Respiratory support (count)	0.00	0.56
Respiratory support duration (days)	0.98	0.54
Invasive operation (count)	0.001	0.59
Blood transfusion (count)	<0.001	0.56
Max value of TSB level (mg/dL)	0.99	0.52
PCT (ng/mL)
Culture positivity (count)	0.01	0.57
Blood culture positivity (count)	0.10	0.52
Hospitalization LOS (days)	0.56	0.5
Hospitalization fees (10 million RMB)	1.69	0.53
Nonmedical order discharge (count)	0.02	0.55
Economic concerns (count)	0.60	0.51
Maternal age (years)	0.35	0.53

AUC, area under the curve; NICU, neonatal intensive care unit; GA, gestational age; BW, birth weight; TSB, total serum bilirubin. PCT, procalcitonin; LOS, length of stay.

The nomogram and its predictive performance

Univariate logistic analysis of risk factors for infant rehospitalization identified 17 variables with AUCs >0.52 (). PCT and the 5-minute Apgar score were excluded due to missing data. Four risk factors were selected in the variable selection process, and risk factors associated with neonate unplanned rehospitalization according to the binominal regression model are shown in . Strong associations were observed for GA (OR: 0.99; 95% CI: 0.98–1.00), NICU LOS (OR: 1.02; 95% CI: 1.01–1.04), nonmedical order discharge (OR: 2.22; 95% CI: 1.34–3.68) and maternal age (OR: 0.96; 95% CI: 0.92–1.00). These strongly associated risk factors were used to generate the prediction model for neonate unplanned rehospitalization.

Table 2

Model construction and validation dataset characteristics

Characteristics	Construction dataset (n=512)a	Validation dataset (n=311)a	Z/t/χ2	P
Sex
Male	298 (58.20)	177 (56.91)	0.13	0.39
Female	214 (41.80)	134 (43.09)
GA (weeks)	36.82±3.6	36.71±3.51	0.59	0.56
BW (g)	2,790.86±844.91	2,780.52±851.65	0.17	0.87
5-minute Apgar	9.24±1.24	9.39±1.03
NICU hospitalization (count)	152 (29.69)	85 (27.33)	0.52	0.26
NICU LOS (days)	8 [3, 30]	9 [3, 29]	0.42	0.68
Antibiotic use (count)	415 (81.05)	233 (74.92)	4.35	0.02
Antibiotic duration (days)	7 [5, 11.5]	7 [5, 10]	−2.40	0.02
Antibiotic types (count)	1 [1, 2]	2 [1, 2]	2.35	0.02
Respiratory support (count)	126 (24.61)	84 (27.01)	0.59	0.25
Respiratory support duration (days)	4.55 (2, 15.94)	4.88 (1, 14.31)	−0.70	0.49
Invasive operation (count)	259 (50.59)	181 (58.2)	4.98	0.02
Blood transfusion (count)	76 (14.84)	56 (18.01)	1.44	0.14
Max value of TSB level (mg/dL)	153.8 (81.3, 247.7)	171 (95.8, 287)	2.26	0.02
PCT (ng/mL)	0.19 (0.11, 0.49)	0.22 (0.12, 0.72)
Culture positivity (count)	210 (41.02)	105 (33.76)	4.25	0.02
Blood culture positivity (count)	42 (8.2)	12 (3.86)	5.38	0.01
Hospitalization LOS (days)	8.5 (5.5, 14.5)	7.5 (4.5, 15)	−1.74	0.08
Hospitalization fees (10 million RMB)	1.31 (0.85, 2.59)	1.25 (0.76, 2.69)	−0.60	0.55
Nonmedical order discharge (count)	83 (16.21)	66 (21.22)	3.85	0.03
Economic concerns (count)	138 (26.96)	61 (19.61)	10.51	0.00
Maternal age (years)	29.64±4.66	29.87±4.76	−0.70	0.48

a, all values are presented as the median (interquartile range) X±S/n (%), n (%). n, number; NICU, neonatal intensive care unit; GA, gestational age; BW, birth weight; TSB, total serum bilirubin; PCT, procalcitonin; LOS, length of stay; RMB, Ren Min Bi.

Table 3

Binominal logistic regression analysis of neonate rehospitalization based on the model construction dataset

Predictor	OR (95% CI)	AUC
Gestational age	0.99 (0.98–1.00)	0.6
NICU length of stay	1.02 (1.01–1.04)	0.57
Nonmedical order discharge	2.2 (1.34–3.68)	0.55
Maternal age	0.96 (0.92–1.00)	0.53

OR, odd ratio; CI, confidence interval; AUC, area under the curve; NICU, neonatal intensive care unit.

The regression coefficients (β) from the binominal logistic model were used to construct the prediction model for neonate unplanned rehospitalization. The scoring system was as follows: Log [P/1-P] = 3.7535- 0.0132* (GA) + 0.0225* (NICU LOS) + 0.7976* (nonmedical order discharge) - 0.0132* (maternal age). The performance of the nomogram was measured by ROC curve analysis, and the AUC was 0.68 in the model construction dataset. A nomogram was generated to summarize the results of the scoring system for clinical use ().

Figure 2

Nomogram to estimate the risk of neonate rehospitalization. NICU, neonatal intensive care unit.

The model developed using the model construction dataset was tested for validity in the validation dataset. The predictive accuracy of the nomogram was measured by the bootstrap method. The C-index for the validation dataset was 0.65, and the risk estimation for the calibration curve was good ().

Figure 3

Predictive performance validity of the nomogram.

The actual result and the result predicted by the neonate unplanned rehospitalization model were analyzed by the Youden index. The model had the best performance when the Youden index was maximized at 1.22. The cutoff score that maximized the Youden index was 0.55, with a sensitivity of 63% and a specificity of 59%. The probability of the cutoff point was 37%, corresponding to a total of 39 points on the nomogram.

Discussion

In this study, a prognostic model for neonate rehospitalization was created by combining neonatal clinical data and demographic data based on 6,526 hospital admission hospital events. External validation was performed based on 4,590 admission events. The overall unplanned rehospitalization rate for neonates was 3.27%. Analysis of 21 risk factors, including 15 clinical factors and 6 demographic factors, produced a simple model that included 4 risk factors. Based on the results, lower GA, a longer NICU LOS, nonmedical order discharge, and a lower maternal age were confirmed to be strong risk factors for neonate unplanned rehospitalizations. The nomogram derived from the prediction model can be used by clinicians on paper or as an online risk calculator.

Previous studies indicated that GA, BW, sex, race, LOS, diagnosis, Apgar score, maternal age, etc. were risk factors for neonate rehospitalization, but the studies were mostly based on maternity hospitals with neonates who were relatively healthy (4,23). Few studies have constructed prediction models based on neonates in the NICU (4,24). One study constructed a model for infants that incorporated 7 binary factors (Apgar score, BW, congenital abnormalities, maternal age, maternal smoking during pregnancy, maternal substance and alcohol use during pregnancy, and previous pregnancy) that predict infant mortality (25). However, risk factors for rehospitalization differ slightly from risk factors for mortality in neonates and there has been no research established a prediction model for neonate unplanned rehospitalization. Our present study focused on disease-based neonate rehospitalization in a specialized hospital and incorporated 4 comprehensive and easily available predischarge variables. The model performed well, with an AUC of 0.68 and a C-index value of 0.65 in the training and validation datasets.

This prognostic model has clinical relevance for neonatologists, enabling them to better identify neonates who are at high risk of rehospitalization. According to our neonate rehospitalization prediction model, 61.1% of the outcomes for discharged neonates could be predicted well (including both safe discharge and rehospitalization). The outcomes of 19.3% of the discharged neonates might be predicted incorrectly (predicted to have a safe discharge but eventually readmitted), putting the discharged infants at risk for health-related uncertainties. Therefore, follow-up for high-risk neonates could be achieved by short-term outpatient visits or maternal education with regard to adequate postnatal care and follow-up should be enhanced for neonates with total scores greater than 39 points.

Limitations

The study has several limitations. First, the datasets did not include information about rehospitalization in a different hospital nor the data from the hospital previously attended, which likely led to an underestimation of rehospitalization. Second, the patients were all consisted of the Han nationality and the data were collected from a single city, thus the application of the model in different races and regions should be assessed. Third, prospective studies are needed to further confirm the reliability of the nomogram.

Conclusions

By including 4 main risk factors for neonate rehospitalization, a nomogram for neonatal clinicians was generated. The model is a good predictor of neonate unplanned rehospitalization, and it performed well after validation. A total score that greater than 39 indicated the need for further follow-up.

The article’s supplementary files as