Modified paediatric preoperative risk prediction score to predict postoperative ICU admission in children: a retrospective cohort study.

OBJECTIVE: To integrate intrinsic surgical risk into the paediatric preoperative risk prediction score (PRPS) model to construct a more comprehensive risk scoring system (modified PRPS) and improve the prediction accuracy of postoperative intensive care unit (ICU) admission in paediatric patients.
DESIGN: This was a retrospective study conducted between 1 January and 30 December 2016. Data on age, American Society of Anaesthesiology physical status (ASA-PS), oxygen saturation, prematurity, non-fasted status, severity of surgery and immediate transfer to the ICU after surgery were collected. The modified PRPS was developed by logistic regression in the derivation cohort; it was tested and compared with the paediatric PRPS and ASA-PS by the Hosmer-Lemeshow test, the receiver operating characteristic (ROC) curve and Kappa analysis in the validation cohort.
SETTING: Hospital-based study in China. PARTICIPANTS: Paediatric patients (≤14 years) who underwent surgery under general anaesthesia were included, and those who needed reoperation due to surgical complications or stayed in the ICU preoperatively were excluded. MAIN OUTCOME MEASURE: ICU admission rate, defined as any patients' direct disposition from the operating room to the ICU immediately after the surgery.
RESULTS: A total of 9261 paediatric patients were included in this study, with 418 patients admitted to the ICU. In the validation cohort, the modified PRPS model fit the test data well (deciles of risk goodness-of-fit χ2=6.84, p=0.077). The area under the ROC curve of the modified PRPS, paediatric PRPS and ASA-PS were 0.963, 0.941 and 0.870, respectively (p<0.05), and the Kappa values were 0.620, 0.286 and 0.267. Analyses in the cohort indicated that the modified PRPS was superior to the paediatric PRPS and ASA-PS.
CONCLUSIONS: The modified PRPS integrating intrinsic surgical risk shows better prediction accuracy than the previous PRPS. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The new simple intrinsic severity of surgery category makes it easier to perform preoperative risk assessments.

The modified preoperative risk prediction score could only be applied to intensive care unit (ICU) admission (planned and unplanned ICU admission) because there was no information regarding unplanned ICU admission.

Limitation includes the shortage of the important information on indicators of transferring paediatric patients into ICU and reports of adverse events, which could help make more objective decisions than surgeons and anaesthesiology teams do.

Introduction

Perioperative morbidity and mortality are higher in children, especially in neonates and infants.1 2 Intensive care unit (ICU) admission offers a measure of additional safety and improves the survival rate for high-risk patients after operations.3 In recent years, several risk stratification tools have been developed to predict perioperative surgical risk to improve postoperative outcomes and facilitate resource allocation in paediatric patients.4 5

In our previous publication,4 we established the paediatric preoperative risk prediction score (PRPS) to predict postoperative ICU admission and death. However, the intrinsic surgical risk factor was not applied to the paediatric PRPS. It is well known that surgeries themselves carry risks for adverse outcomes beyond the influence of anaesthesia and patient comorbidities. Jason and colleagues recently defined the intrinsic risk of surgical procedures for perioperative adverse cardiac events in adults.6 To date, only one analysis of the intrinsic risk of surgical procedures in paediatric patients has been published. They performed a retrospective analysis of 367 065 surgical cases of paediatric patients from the American College of Surgeons National Surgical Quality Improvement Program database and found that paediatric risk stratification was improved by integrating the intrinsic risk of individual paediatric surgical procedures.7

The aim of this study was to integrate intrinsic surgical risk into the paediatric PRPS model to create a new and more comprehensive risk scoring system (modified PRPS) to improve the prediction accuracy of postoperative ICU admission in paediatric patients.

Methods

This study was developed in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis reporting guidelines.8

Patients

A retrospective cohort study of paediatric patients who underwent surgical procedures at the Second Affiliated Hospital and Yuying Children's Hospital of Wenzhou Medical University from January to December 2016 was performed by two independent examiners. The enrolled patients were ≤14 years old who underwent surgeries (both elective and non-elective surgeries) under general anaesthesia after informed written consent was signed by the parents. The exclusion criteria included patients who needed reoperation due to surgical complications or those who had stayed in the ICU preoperatively.

Study design

In the paediatric PRPS, the data of five preoperative predictors, including age, American Society of Anaesthesiology physical status (ASA-PS), prematurity, oxygen saturation (SpO2, before anaesthesia induction) and non-fasted status, were collected from the electronic anaesthesia records (variables defined as previously described).4 For the modified PRPS, we integrated a PRPS additional variable, the intrinsic severity of surgery, into the scoring system. The severity of surgery was graded into three classes: minor, moderate and major. All surgical patients fell into one of these categories based on a pre-set simplified criterion as follows: Class I (minor surgeries: extremities and body surface surgeries): orthopaedic surgery, arthroscopy, superficial tissue surgery, tonsillectomy/adenoidectomy, grommet/cochlear prosthesis insertion, mastoidectomy, strabotomy, circumcision, anoplasty, urethroplasty, inguinal herniorrhaphy and resection of testicular hydrocele; Class II (moderate surgeries: intraperitoneal surgeries): open abdominal procedure (abdominal organ, exploratory laparotomy, diaphragmatic hernia) and laparoscopic surgery; Class III (major surgeries: thoracic or intracranial surgeries): open thoracic or intracranial procedure: craniotomy (intracranial haematoma, hydrocephalus and neoplasms), thoracotomy (cardiac, pulmonary, oesophageal atresia, pericardiectomy and pyothorax surgery) and thoracoscopy.

The primary outcome of the study, ICU admission, including both planned and unplanned admission, was defined as all patients’ direct disposition from the operating room to the ICU for any reason immediately after the surgery. The final decisions for patients’ postoperative direct transfer to the ICU were generally made together by the anaesthesiologist and surgeon. The second outcome, perioperative mortality, was defined as death within 30 days after the surgery.

Statistical analysis

The data set (9261 patients) was randomly divided into two cohorts: a derivation cohort (consisting of approximately two-thirds of the sample) and a validation cohort (consisting of the remainder). Logistic regression was used to create the modified PRPS model to predict ICU admission after surgery in the derivation cohort. Then, the modified PRPS model was tested on the validation cohort. A risk score was derived for each patient by taking the sum of the model coefficients for the risk factors present. Differences among the groups were examined with the Cochran-Armitage test for trend. The accuracy of the modified PRPS model was assessed in the validation cohort by using the Hosmer-Lemeshow (H-L) test.9 The receiver operating characteristic (ROC) curve was used to measure discrimination; the cut-off point was determined by Youden’s index. ROC curves and kappa statistics were used to compare the accuracy of the modified PRPS with those of the paediatric PRPS and ASA-PS. A kappa value of 1 indicates perfect agreement, whereas a kappa value of 0 indicates agreement equivalent to chance.

All data were analysed with SAS software (SAS V.9.4; SAS Institute). The data are presented as the median (IQR), numbers and percentages.

Patient and public involvement

Because it is an observational and retrospective study, there is no additional risk or burden on paediatric patients. It has nothing to do with the patients’ priorities, experience and preferences. We searched all paediatric patients who underwent surgical procedures in our hospital from January to December 2016. We will put our study results (or the published article link) on our hospital website.

Results

Initially, 9315 patients were enrolled in the data set, and 54 patients were excluded because of missing information. Finally, two-thirds of the 9261 patients were assigned to the derivation cohort (n=6174), while the other one-third was used as the validation cohort to examine the fit of the model (n=3087), as shown in figure 1. The rates of ICU admission were 4.66% (288/6174) in the derivation cohort and 4.21% (130/3087) in the validation cohort. The perioperative mortality of the ICU admission patients was 12.15% (35/288) in the derivation cohort and 10% (13/130) in the validation cohort. No paediatric patients died in the operating room. The preoperative characteristics of the enrolled patients are summarised in table 1.

Figure 1

Flowchart of the study.

Table 1

Patient characteristics

Variables	Development data (n=6174)		Validation data (n=3087)
	PACU, n (%)	ICU, n (%)	PACU, n (%)	ICU, n (%)
Age
≥1 year	356 (6.05)	84 (29.17)	182 (6.15)	40 (30.77)
1 month–1 year	5509 (93.59)	116 (40.28)	2752 (93.07)	43 (33.08)
<1 month	21 (0.36)	88 (30.56)	23 (0.78)	47 (36.15)
ASA-PS
I	5121 (87.00)	44 (15.28)	2519 (85.19)	23 (17.69)
II	715 (12.15)	132 (45.83)	418 (14.14)	51 (39.23)
III	49 (0.83)	84 (29.17)	15 (0.51)	42 (32.31)
IV/V	1 (0.02)	28 (9.72)	5 (0.17)	14 (10.77)
Premature
No	5705 (6.92)	232 (80.56)	2850 (96.38)	90 (69.23)
Yes	181 (3.08)	56 (19.44)	107 (3.62)	40 (30.77)
Non-fasted
No	5806 (98.64)	242 (84.03)	2903 (98.17)	112 (86.15)
Yes	80 (1.36)	46 (15.97)	54 (1.83)	18 (13.85)
SpO2
≥90%	5798 (98.50)	251 (87.15)	2895 (97.90)	117 (90.00)
<90%	88 (1.50)	37 (12.85)	62 (2.10)	13 (10.00)
Severity of surgery
Class I	3434 (58.34)	26 (9.03)	1728 (58.44)	16 (12.31)
Class II	2393 (40.66)	128 (44.44)	1193 (40.34)	58 (44.62)
Class III	59 (1.00)	134 (46.53)	36 (1.22)	56 (43.08)
30-day mortality	–	35 (12.15)	–	13 (10)

ASA-PS, American Society of Anaesthesiology physical status; ICU, intensive care unit; PACU, post-anaesthesia care unit; SpO2, oxygen saturation.

The modified PRPS model development and derivation

The five variables (age, ASA-PS, SpO2, prematurity and non-fasted status) were recorded from the hospital information system as previously described in the paediatric PRPS. For the modified PRPS, the variable of intrinsic surgical risk was added and graded based on the increased risks associated with the location and range of the procedure. Therefore, this new model had six independent variables for predicting ICU admission, which was created by binary logistic regression analysis (table 2).

Table 2

Binary logistic regression analysis predicting the incidence of postoperative intensive care unit admission

Variables	B	SE	Wald	OR (95% CI)	P value
Constant	−6.750	0.306	486.963	–	<0.001
1 month–1 year	1.508	0.248	36.925	4.52 (2.78 to 7.35)	<0.001
<1 month	4.736	0.354	178.604	113.97 (56.90 to 228.27)	<0.001
ASA II	2.277	0.230	97.676	9.75 (6.20 to 15.31)	<0.001
ASA III	2.741	0.355	59.640	15.50 (7.73 to 31.07)	<0.001
ASA IV/V	7.092	1.273	31.044	1202.19 (99.21 to 14 568.08)	<0.001
Premature	1.038	0.331	9.819	2.82 (1.48 to 5.41)	0.002
Non-fasted	1.069	0.339	9.957	2.91 (1.50 to 5.65)	0.002
SpO2 <90%	0.963	0.467	4.247	2.62 (1.05 to 6.55)	0.040
Class III	4.836	0.334	210.182	126.00 (65.53 to 242.30)	<0.001
Class II	1.761	0.292	36.263	5.82 (3.28 to 10.32)	<0.001

ASA, American Society of Anaesthesiology; SpO2, oxygen saturation.

The model coefficients were used to develop a formula for a risk score as follows, where each variable was assigned a value of 1 if present and 0 if absent: Logit (P)=ln(P/1 P)=constant + risk score; risk score=[(1 month–1 year)*1.508] + (≤1 month*4.736) + (ASA II*2.272) + (ASA III*2.741) + (ASA IV/V*7.092) + (premature*1.038) + (non-fasted*1.069) + (SpO2 <90% *0.963) + (Class III *4.836) + (Class II *1.761). The predictor had its own control value, which was set as a score of 0 (age ≥1 year old, ASA I, SpO2 ≥90%, full term, fasted, Class I). The sum of the highest values from each predictor was 50 points. A point value was assigned to each predictor by consequently normalising them and converting them to integer scores. The modified PRPS system was then constructed, as shown in table 3.

Table 3

Risk score

Variables	Score	AUC
Age		0.963
≥1 year	0
1 month–1 year	4
<1 month	12
ASA-PS
I	0
II	6
III	7
IV/V	18
Premature	3
Non-fasted	3
SpO2
≥90%	0
<90%	2
Severity of surgery
Class I	0
Class II	4
Class III	12

ASA-PS, American Society of Anaesthesiology physical status; AUC, area under the ROC curve; SpO2, oxygen saturation.

Comparison with the paediatric PRPS and ASA-PS

When these calculations were used to produce a percentage of predicted ICU admission after surgery for each paediatric patient in the validation cohort, the area under the ROC curve (AUC) value for ICU admission rates was 0.963, indicating excellent discrimination. From the ROC curve, it was calculated that the cut-off point for the risk predictor score was 10 points. The P value for the H-L test was 0.077, indicating that the modified PRPS model was well calibrated in the validation cohort.

Both the paediatric PRPS and ASA-PS demonstrated moderately good discrimination when tested in the validation cohort, with AUCs of 0.941 (95% CI, 0.932 to 0.949) and 0.870 (0.858 to 0.882), respectively (figure 2A). However, among the paediatric patients who were admitted to the ICU, the ROC curve for the discrimination between dead and surviving paediatrics showed a similarly and relatively poor ability, where the AUC values of the modified PRPS, paediatric PRPS and ASA-PS were 0.759 (95% CI, 0.676 to 0.830), 0.758 (0.675 to 0.829) and 0.762 (0.679 to 0.832), respectively (figure 2B). The cut-off point for the risk predictor score was 19 points according to the ROC curve for the modified PRPS model.

Figure 2

Receiver operating characteristic curves for the modified preoperative risk prediction score (PRPS), PRPS and American Society of Anaesthesiology physical status (ASA-PS) for the validation cohort: (A) a randomly selected individual who had intensive care unit admission had an overall score higher than that of paediatric patients who had post-anaesthesia care unit admission; (B) a randomly selected individual who died had an overall score higher than that of paediatric patients who survived.

The accuracy was higher in the modified PRPS model than in the paediatric PRPS and ASA-PS (95.85%, 84.68% and 85.07%, respectively), with kappa statistics of 0.620, 0.286 and 0.267.

According to the results, the modified PRPS was built, as shown in table 4. Three risk categories (high, intermediate and low risk) were defined based on the cut-off values. As the score of the modified PRPS increased, the incidence of propensity for ICU admission increased monotonically (p<0.0001).

Table 4

Outcomes for paediatric patients undergoing surgery in relation to the modified PRPS.

Risk level	Score	Patients (n)	ObservedICU admission,n (%)	Predicted ICU admission,n (%)	Prediction probability %median (IQR)	P value
Low risk	<10	8283	37 (0.45)	42 (0.51)	0.40% (0.12% to 0.59%)	<0.001
Intermediate risk	10–18	690	138 (20)	139 (20.14)	8.63% (5.99% to 31.41%)
High risk	19–50	288	243 (84.3)	259 (89.9)	91.51% (87.89% to 98.12%)

ICU, intensive care unit; PRPS, preoperative risk prediction score.

Discussion

Principal findings

This study verified our previous paediatric PRPS with an excellent AUC of 0.941 in the validation cohort. Then, with the modification and update of the previous version, the new model, termed modified PRPS, offered better capability in the prediction of ICU admission after surgery in children.

The modified PRPS had six observing variables instead of the five seen in the paediatric PRPS, and the total of combined risk scores are 50 points in both versions. Compared with the paediatric PRPS, the modified PRPS merged the categories of ASA IV and ASA V into one category. By considering the clinical gravities, slight plus or minus changes in the point distribution among the different categories were made, which allowed the modified PRPS to be more rational and clinically practical.

By using the H-L test during the validation process, the modified PRPS displayed a better calibration, which suggested an improved discrimination when the ROC curve of the new model was compared with that of the paediatric PRPS and ASA-PS scores. In addition, the kappa statistic was used to compare the agreement of the observed and predicted ICU admission rates among the three scoring models. The modified PRPS had an accuracy of 95.85% with a kappa value of 0.62, which was in the substantial agreement range and significantly higher than that of the other two models. After considering the severity of surgery, both the AUC and kappa values were closer to 1, and the modified PRPS was more accurate and closer to perfect.

Comparison with other studies

A variety of assessment score formulas for perioperative risk prediction have been published.4 5 8 10–14 As a gold standard for evaluating a patient’s general health and comorbidities preoperatively, the ASA-PS has been widely used to predict perioperative outcomes in children, even if it was not initially intended to be used in children for the reason of lacking objective.10 The Paediatric Risk of Mortality Score11 and Paediatric Index of Mortality12 have been widely used to predict mortality for children, but the greatest limitation is that they are only used in ICUs. The Paediatric Risk Assessment score, including 13 preoperative variables, had excellent accuracy in predicting perioperative mortality in children.13 Statistically significant differences of stratified surgical subgroups were found in univariate analysis, but not in multivariate regression analysis. However, stratified subgroups of surgeries are not equivalent to the severity of surgery. The intrinsic risk of the surgical procedure was not included in the final model. Moreover, it was only applied to non-cardiac surgeries, not all types of surgeries. The American College of Surgeons National Surgical Quality Improvement Program Paediatric Surgical Risk Calculator is a tool to calculate the risk of complications and mortality for a variety of surgical procedures. However, it requires a fill-in of current procedural terminology code, which is not easily accessible for health care-givers in other countries outside of the USA.14

There is no uniform definition of what is considered ‘intrinsic severity of surgery’ in current studies. Considering the impact of surgery (type and complexity) on outcomes, some clinicians graded surgical severity according to their own criteria. In 1996, Arvidsson et al coded surgical interventions into a 4-point scale ranging from minor interventions to extensive procedures, according to the official Swedish classification of the interventions.15 In 2002, the surgical risk score adopted the British United Provident Association operative grade category as one risk factor to predict mortality in surgical patients.16 In 2004, by modifying the Johns Hopkins criteria, Donati et al simplified surgical severity to three grades and developed their own new model for predicting operative risk.17 Based on the Office of Population, Censuses and Surveys system codes, the Surgical Outcome Risk Tool graded the magnitude of surgical procedures into four severity categories in 2014.18 Stratified subgroup analysis was also commonly applied for the types of operation.17 19 In this study, we classified the intrinsic severity of surgeries into three different levels (minor, moderate and major levels), which was different from other surgical categories but easier to work with.

Limitations of the study

There were some limitations to the present study. First, our primary endpoint includes both planned and unplanned ICU admissions. Second, the decision-making for ICU admission mainly relied on clinical bias and was also influenced by regional culture, economic factors and so on. Third, a higher probability of dying was also associated with a higher probability of being admitted to the ICU, but they were not treated as equivalent in our results.

Conclusion

In summary, with an appropriate adjustment of the ASA assessment and the integration of surgical severity into the scoring model, the new modified PRPS exhibits a more accurate prediction result and better discriminates ICU admission immediately after surgery in paediatric patients.