Risk prediction models for emergence delirium in paediatric general anaesthesia: a systematic review.

OBJECTIVES: Emergence delirium (ED) occurs in approximately 25% of paediatric general anaesthetics and has significant adverse effects. The goal of the current systematic review was to identify the existing literature investigating performance of predictive models for the development of paediatric ED following general anaesthesia and to determine their usability.
DESIGN: Systematic review using the Prediction model study Risk Of Bias Assessment Tool (PROBAST) framework. DATA SOURCES: Medline (Ovid), PubMed, Embase (Ovid), Cochrane Database of Systematic Reviews (Ovid), Cochrane CENTRAL (Ovid), PsycINFO (Ovid), Scopus (Elsevier) and Web of Science (Clarivate Analytics), ClinicalTrials.gov, International Clinical Trials Registry Platform and ProQuest Digital Dissertations and Theses International through 17 November 2020. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: All randomised controlled trials and cohort studies investigating predictive models for the development of ED in children undergoing general anaesthesia. DATA EXTRACTION AND SYNTHESIS: Following title, abstract and full-text screening by two reviewers, data were extracted from all eligible studies, including demographic parameters, details of anaesthetics and performance characteristics of the predictive scores for ED. Evidence quality and predictive score usability were assessed according to the PROBAST framework.
RESULTS: The current systematic review yielded 9242 abstracts, of which only one study detailing the development and validation of the Emergence Agitation Risk Scale (EARS) met the inclusion criteria. EARS had good discrimination with c-index of 0.81 (95% CI 0.72 to 0.89). Calibration showed a non-significant Homer-Lemeshow goodness-of-fit test (p=0.97). Although the EARS demonstrated low concern of applicability, the high risk of bias compromised the overall usability of this model.
CONCLUSIONS: The current systematic review concluded that EARS has good discrimination performance but low usability to predict ED in a paediatric population. Further research is warranted to develop novel models for the prediction of ED in paediatric anaesthesia. PROSPERO REGISTRATION NUMBER: CRD42019141950. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is a first systematic review of risk prediction models for paediatric emergence delirium and adheres to recommendations made in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

An extensive systematic literature search was conducted through 11 databases including Medline (Ovid), Embase (Ovid) and Web of Science (Clarivate Analytics) from their inception to 17 November 2020.

The current systematic review employed the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies for data extraction and the Prediction model study Risk of Bias Assessment Tool which determines the usability of the prediction models identified in the systematic review.

Our systematic review revealed only one study reporting the performance characteristics of the Emergence Agitation Risk Scale, a predictive model with low usability for predicting the development of paediatric emergence delirium.

Our results serve to highlight the need for the ongoing development and validation of robust predictive models for paediatric emergence delirium risk.

Introduction

Emergence delirium (ED) is a common complication occurring in approximately 25% of paediatric general anaesthetics (GA)1 and is associated with significant adverse effects including injury to the patient and personnel, damage to incision sites, exacerbated parental anxiety and increased nursing requirements, further resulting in an increased burden on the healthcare system.2 3 Identifying which patients are at highest risk for developing ED will allow practitioners to effectively apply multimodal prophylaxis in order to decrease the incidence of this significant complication.

Several risk factors have been identified for the development of ED, including age, preoperative anxiety, type of surgery and type of anaesthetic given.2 4 Development of predictive scores which aim to integrate these risk factors to determine an individual’s overall risk for developing ED has been attempted. However, no prior systematic review has been conducted to determine the usability of these prediction models.

The aim of the current study is to conduct a systematic review to identify all existing prediction models for the development of ED in a paediatric population, assess model performance and determine the usability of these models for use in clinical practice.

Methods

The results were reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols (PRISMA) 2015 statement.5

Search strategy

An information specialist with experience in systematic reviews searched eight databases (Medline (Ovid), PubMed, Embase (Ovid), Cochrane Database of Systematic Reviews (Ovid), Cochrane CENTRAL (Ovid), PsycINFO (Ovid), Scopus (Elsevier) and Web of Science (Clarivate Analytics) starting from their inception. Furthermore, three databases were searched for relevant recently completed or ongoing research (ClinicalTrials.gov, International Clinical Trials Registry Platform and ProQuest Digital Dissertations and Theses International). All searches were conducted on 17 May 2019 and updated on 17 November 2020. Search strategies were built to contain three sets of terms reflecting our search questions including the prediction models, the target condition (ED, emergence agitation (EA)) and the patient population (paediatric patients undergoing GA). Since ED and EA are sometimes used interchangeably in older literature, both these terms were included in our search strategy. In addition, reference lists of relevant trials and reviews were scanned. Refer to online supplemental appendix 1 for search strategies for all databases.

Study selection and data extraction

Title, abstract and full-text screening were independently conducted by two reviewers (SK and BS) with conflicts resolved by a third reviewer (M-AP). Cohen’s kappa was calculated to quantify the inter-rater reliability.6

Inclusion criteria were: randomised controlled trials (RCTs), cohort studies or case–control studies examining paediatric populations (<18 years old) undergoing GA investigating preoperative or intraoperative predictive models for the development of ED or EA in the postanaesthetic care unit (PACU). Inclusion and exclusion criteria are detailed in the online supplemental appendix 2.

Study characteristics as well as primary and secondary outcomes data were extracted from relevant studies using a standardised data collection form in accordance to the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) framework7 and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklist.8 These included demographic information, anaesthetic characteristics (type and dosages of anaesthetic used for induction and maintenance, use and dosage of premedication); type of scale used (components, model development methodology, model evaluation methodology, time span of prediction, and intended moment for the use of the model), discrimination characteristics (area under receiver operating curve (AUC) or corresponding c-statistic with 95% CI) and calibration characteristics (calibration plots, ratio between predicted vs observed incidence of ED, or Hosmer-Lemeshow goodness-of-fit statistic).

Assessment of methodological quality

The methodological quality of the evidence was determined using the Prediction model study Risk Of Bias Assessment Tool (PROBAST) framework.9 This consists of first determining the usability of the risk prediction model based on the risk of bias and concerns of applicability across four domains (participants, predictors, outcome and analysis), followed by a determination of the model’s predictive performance (discrimination and calibration).9 The model was considered to be ‘usable’ if it had a low risk of bias, low concern about applicability, and good predictive performance of discrimination and calibration. Good discrimination is defined as AUC ≥0.8.10 11

Software

References were collected and deduplicated using Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia).

Patient and public involvement

It was not appropriate, possible or necessary to involve patients or the public in the design or conduct of this study, or to disseminate the results to them.

Results

Search results

The literature search yielded 9242 citations, which, following title, abstract and full-text screening, yielded one study that met the inclusion criteria. The PRISMA flow diagram is shown in figure 1. Four full-text articles were excluded due to irrelevant populations,12 model being applied post-operatively13 and lack of reported discrimination or calibration data4 14 (table 1). The eligible paper investigated the development and validation of the Emergence Agitation Risk Scale (EARS).15 Cohen’s kappa showed good agreement between the two reviewers (kappa=0.99).

Figure 1

PRISMA flow diagram for search and review strategy. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Table 1

Detailed explanations for studies excluded following full-text assessment

Study reference	Predictive model investigated	Population	Reason for exclusion
Beringer et al, 201414	Paediatric Anaesthesia Behaviour scale	Children aged 2–12 years undergoing general anaesthesia for dental extractions	No discrimination or calibration parameters were reported
Kain et al, 20044	Yale Preoperative Anxiety Scale	Children undergoing surgery with general anaesthesia	No discrimination or calibration parameters were reported
Oh et al, 201312	Automatic PREdiction of DELirium in Intensive Care Units (ICU)	Delirium in adult ICU setting	Irrelevant population
Sadhasivam et al, 200913	Perioperative Adult Child Behavioural Interaction Scale (PACBIS)	Children aged 3–12 years undergoing general anaesthesia for tonsillectomy and/or adenoidectomy	PACBIS measured in the recovery room concurrently with PAED scoreNo discrimination or calibration parameters were reported

PAED, Paediatric Anaesthesia Emergence Delirium.

Emergence Agitation Risk Scale

The included study, published in 2017, was a Japanese, single-centre, paediatric hospital-based study detailing the development and validation of the EARS. It comprised a total of 220 ASA class I or II patients with mean ages 4.1±1.8 years and undergoing minor surgery including tonsillectomy, adenoidectomy, myringotomy tube insertion, strabismus surgery, cryptorchidism repair or inguinal hernia repair under GA. Anaesthetic induction consisted of sevoflurane and nitrous oxide inhalation without premedication while maintenance was achieved with sevoflurane. Analgesia consisted of intravenous fentanyl, acetaminophen suppository and nerve blocks wherever indicated. The outcome of interest, EA, was measured in the PACU using the Paediatric Anaesthesia Emergence Delirium (PAED) scale with a cut-off of >12 and had an overall incidence of 36.4%.16 Study characteristics are detailed in table 2.

Table 2

Summary characteristics of studies included in systematic review

Study	Score name and composition	Population	Inclusion and exclusion criteria	Anaesthetic type	Outcome definition	Model development and validation	Model performance
15	Emergence Agitation Risk Scale (EARS): Age Operative procedure (strabismus surgery, tonsillectomy) Paediatric Anaesthesia Behaviour scale Anaesthesia time	n=220 (120 during development and 100 during validation phases)	Inclusion criteria: 1.5–8 years old; ASA 1–2; undergoing general anaesthesia for inguinal hernia repair, adenoidectomy, tonsillectomy, strabismus repair, tympanostomy tube, cryptorchidism, and other minor surgeryExclusion criteria: mental retardation; taking psychotropic drugs	Induction with N2O and sevoflurane, maintenance with sevofluraneAnalgesia with intravenous fentanyl and suppository acetaminophen, and nerve block where appropriate	PAED score >12 as assessed by trained nurse or anaesthesiologist in PACUIncidence:34.2% of patients in the development phase39% of patients in the validation phase	Development phase: Data from prior randomised controlled trial Multicollinearity between 10 candidate predictors: age, height, weight, sex, operative procedures (tonsillectomy, strabismus surgery, other), airway management (ETT, LMA), nerve block, fentanyl dose, anaesthesia time, PAB score Multivariate logistic regression with Akaike information criterion stepwise selection conducted to determine the optimal combination of 7 remaining predictors (age, sex, operative procedure, airway management, anaesthesia time, PAB score) Beta-coefficient was calculated for each predictor Validation phase: Prospective cohort data. Best cut-off point determined by Youden index Grey zone identified as range between the two points on receiver operating curve where sensitivity and specificity are 90%	Development phase:Discrimination: C-index 0.84 (95% CI, 0.74 to 0.94)Calibration: Hosmer-Lemeshow goodness-of-fit test non-significant (p=0.97)Validation phase:Discrimination: C-index 0.81 (95%CI 0.72-0.89)

ASA, American Society of Anesthesiologists; EARS, Emergence Agitation Risk Scale; ETT, endotracheal tube; LMA, laryngeal mask airway; PAB, Paediatric Anaesthesia Behaviour Scale; PACU, postanaesthesia care unit; PAED, Paediatric Anaesthesia Emergence Delirium scale.

Development of EARS was conducted retrospectively in a cohort of 120 patients previously enrolled in an RCT investigating the use of acupuncture in the prevention of EA in children.17 Logistic regression was used to test the ten candidate predictors including age, height, weight, sex, Paediatric Anaesthesia Behaviour (PAB) score, operative procedure, anaesthesia time, airway securing device (endotracheal tube or laryngeal mask airway), presence or absence of nerve block use, and total fentanyl dose. The Akaike information criterion stepwise selection was used to identify four independently associated predictors for final inclusion in the EARS (age, PAB score, anaesthesia time and operative procedure). β-Coefficients were calculated and converted to integer scores for each predictor, yielding a score range of 1–23.

The validation phase of the score was conducted separately in a prospective observational cohort of 100 patients.

Study findings

The development phase study population had mean age 3.7±1.7 years and incidence of EA of 34.2%. The c-statistic was 0.84 (95% CI 0.74 to 0.94) and Hosmer-Lemeshow statistic was non-significant (p=0.97), indicating adequate discrimination and calibration, respectively.

The validation phase study population had mean age 4.5±1.9 years and EA incidence of 39%. The c-statistic for the validation phase was 0.81 (95% CI 0.72 to 0.89). The optimum cut-off point was found to be 11, yielding 87% sensitivity and 61% specificity for the development of ED. The grey zone, delimited by the points of the scale where the sensitivity and specificity become 90%, was 10 to 13 and comprised 38% of patients. Calibration data was not reported for the validation phase.

Score usability

Usability analysis based on PROBAST criteria, revealed low concern of applicability across all three domains probed (participants, predictors, outcomes). However, there was a high risk of bias, primarily due to deficits in the analysis domain. In particular, the number of events (n=41) per candidate predictor (n=10) during the development phase was 4.1 (at least 10 and preferably 20 would be considered adequate). Furthermore, during the validation phase the population size was inadequate, yielding only 39 patients experiencing the primary outcome, which fell short of the recommended 100 patients experiencing the outcome of interest. The authors did not present a calibration plot to illustrate the goodness-of-fit associated with the statistically significant Hosmer-Lemeshow statistic. And lastly, the authors did not account for overfitting and optimism in the context of the small events per candidate predictor (online supplemental table S1).

Taken together, given the good discrimination performance but insufficient calibration evaluation, as well as the high risk of bias despite a low concern of applicability of the model to paediatric populations for predicting EA, the EARS was given a designation of low ‘usability’ (table 3).

Table 3

Prediction model usability assessment

Study	Predictive performance		Risk of bias	Applicability	Usability
	Discrimination	Calibration
Hino et al, 201715EARS model	Development:C-index 0.84(95% CI 0.74 to 0.94)Validation:C-index 0.81(95% CI 0.72 to 0.89)	Development:Hosmer-Lemeshow p=0.97Validation:Not reported	Overall high concern	Overall low concern	Low

EARS, Emergence Agitation Risk Scale.

Discussion

The current systematic review revealed only one scale targeted at predicting the risk of developing ED in children, a complication associated with significant morbidity occurring in approximately 25% of paediatric GA. The PROBAST assessment of usability indicated that the discriminative performance and applicability of the scale were both good, but insufficient calibration evaluation and deficiencies in the analysis placed the scale at high risk of bias, reducing its overall usability in clinical practice.

To date, this has been the only systematic review looking at prediction scales of ED in a paediatric population. The review was extensive, including eight major databases of peer-reviewed literature as well as 3 databases of protocols, with the search including both ED as well as EA, which have been interchangeably used in previous literature. The study protocol was published in the PROSPERO database before the start of the review. The review process used rigorous methodology, employing two independent reviewers and careful adherence to the PRISMA,5 CHARMS,7 TRIPOD8 and PROBAST9 guidelines for systematic reviews and prediction scales. While the scale was found to have low usability, the current study highlights potential required improvements in the methodology of future studies validating the EARS, with problems arising from high risk of bias due to the analysis phase. Particular attention should be given to the number of events per variable to avoid the risk of overfitting and underfitting of the model.

A considerable limitation of the current review was that only one study investigating a single predictive model was found, which met all the inclusion criteria. While this hampered our ability to run a meta-analysis on the predictive properties of the scale, a need for further validation in other settings is underscored, as is the need for the development of other scales in this domain. Literature in this field may have been previously lacking due to considerable variations in the definition of ED, with different diagnostic scales being used, although the PAED scale is the only validated scale to diagnose ED in children at the moment.16 Furthermore, several confounding factors are present in the postoperative period, which makes timely and accurate diagnosis of paediatric ED difficult in clinical and research settings.1–3 As such, strategic prevention of paediatric ED based on a precise and validated prediction scale is preferable to treatment when the event occurs.

The current systematic review highlights several potential clinical and research implications. The first is the need to identify validated risk factors for ED based on prior research, which can then be used to generate new prediction models using prospective cohort methodology to accurately and precisely identify those patients at the highest risk for developing this side effect.

Important lessons can be drawn from the three studies excluded in our systematic review due to a lack of reported discrimination and calibration parameters. Discrimination and calibration characteristics are essential features in the assessment of the usability of a predictive scale. Discrimination values such as the AUC or the C-statistic indicate how well a model differentiates those patients who are likely to develop a condition in comparison to those who are not likely to develop a condition. Calibration, on the other hand, as measured by a visual representation of the relationship between observed and predicted values and by the Hosmer-Lemeshow statistic, is a measure of the scale’s ability to predict the absolute risk of developing the endpoint in question. The results of this study indicate that future scale development and validation should include a systematic assessment of the predictive properties of such scales. Indeed, following the analytical guidance provided by the PROBAST guidelines would ensure the analytic rigour required to incorporate these scores into clinical practice.

Lastly, future research, quality improvement projects and cost–benefit analyses should focus on determining whether implementing such scales into clinical practice to help target prophylaxis and treatment of ED is adequate and acceptable to healthcare providers and whether it ultimately results in improved patient outcomes.

In conclusion, the current systematic review has revealed a single study of predictive scores for the development of paediatric ED. While this score showed low usability, our results do highlight the need for the development of more such scales as well as the requirement for methodologically rigorous validation of such scales.