Performance of an Electronic Decision Support System as a Therapeutic Intervention During a Multicenter PICU Clinical Trial: Heart and Lung Failure-Pediatric Insulin Titration Trial (HALF-PINT).

BACKGROUND: The use of electronic clinical decision support (CDS) systems for pediatric critical care trials is rare. We sought to describe in detail the use of a CDS tool (Children's Hospital Euglycemia for Kids Spreadsheet [CHECKS]), for the management of hyperglycemia during the 32 multicenter Heart And Lung Failure-Pediatric Insulin Titration trial. RESEARCH QUESTION: In critically ill pediatric patients who were treated with CHECKS, how was user compliance associated with outcomes; and what patient and clinician factors might account for the observed differences in CHECKS compliance? STUDY DESIGN AND METHODS: During an observational retrospective study of compliance with a CDS tool used during a prospective randomized controlled trial, we compared patients with high and low CHECKS compliance. We investigated the association between compliance and blood glucose metrics. We describe CHECKS and use a computer interface analysis framework (the user, function, representation, and task analysis framework) to categorize user interactions. We discuss implications for future randomized controlled trials.
RESULTS: Over a 4.5-year period, 658 of 698 children were treated with the CHECKS protocol for ≥24 hours with a median of 119 recommendations per patient. Compliance per patient was high (median, 99.5%), with only 30 patients having low compliance (<90%). Patients with low compliance were from 16 of 32 sites, younger (P = .02), and less likely to be on inotropic support (P = .04). They were more likely to be have been assigned randomly to the lower blood glucose target (80% vs 48%; P < .001) and to have spent a shorter time (53% vs 75%; P < .001) at the blood glucose target. Overrides (classified by the user, function, representation, and task analysis framework), were largely (89%) due to the user with patient factors contributing 29% of the time.
INTERPRETATION: The use of CHECKS for the Heart And Lung Failure-Pediatric Insulin Titration trial resulted in a highly reproducible and explicit method for the management of hyperglycemia in critically ill children across varied environments. CDS systems represent an important mechanism for conducting explicit complex pediatric critical care trials. CLINICAL TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT01565941, registered March 29 2012; https://clinicaltrials.gov/ct2/show/NCT01565941?term=HALF-PINT&draw=2&rank=1.

Electronic clinical decision support (CDS) systems or tools are infrequently used in the PICU, despite widespread data supporting their use in adult ICUs. Additionally, the use of CDS systems for multicenter research trials are often underutilized.2, 3, 4 With the use of an electronic CDS system, Children’s Hospital Euglycemia for Kids Spreadsheet (CHECKS), the National Institutes of Health-funded Heart and Lung Failure-Pediatric Insulin Titration (HALF-PINT) trial demonstrated that blood glucose control to a target of 80 to 110 mg/dL with IV insulin was associated with increased hypoglycemia and conferred no advantage compared with a blood glucose target of 150 to 180 mg/dL. Large multicenter trials provide the highest quality evidence that indicates the optimal approach to blood glucose control with IV insulin.5, 6, 7, 8 Furthermore, successful dissemination of such research results requires demonstration of a safe, feasible, and replicable intervention of blood glucose control that is generalizable and minimizes the cognitive or work burden of bedside nurses.9, 10, 11

In the adult ICU, the use of electronic CDS systems for blood glucose control in clinical care results in more consistent blood glucose target levels and fewer adverse events than do paper protocols., Before HALF-PINT, investigators implemented paper protocols or guidelines for IV insulin titration in the PICU,, largely because electronic CDS protocols for clinical care in the PICU are not accepted widely.15, 16, 17, 18 Historically, clinicians express concern about electronic-based tools due to alienation from bedside decisions and the possible introduction of unseen risk to the patient.,, Excessive time consumption, interruptions of workflow, and interface usability are other stated barriers to bedside electronic CDS system implementation.,

During the HALF-PINT trial, CHECKS served as the research intervention protocol and was exported to 35 PICUs, 32 of which enrolled patients in the study. A novel just-in-time training and monitoring of adherence was facilitated by a web-based system that aggregated data in real time and processed the data on a server located at the clinical coordinating center for the trial. These efforts contributed to an overall compliance with CHECKS recommendations of >98%.

We describe in detail the adoption of CHECKS for the management of blood glucose control during the HALF-PINT trial. We compare patient factors, glucose metrics, and outcomes between high and low compliance with recommendations groups. We discuss novel CDS tools and clinician interaction data and perform an electronic CDS tool framework analysis to better understand broad areas of electronic CDS system refinement to enhance the adoption of CHECKS for additional research. We detail the website used to facilitate training. We expand our discussion to encourage the use of electronic CDS systems as explicit protocols in future pediatric critical care clinical trials.

Study Design and Methods

Study Site and Subject Selection

In this secondary analysis of the HALF-PINT trial, patients at 32 sites who were treated with the CHECKS computer blood glucose control protocol were eligible for inclusion. The parent HALF-PINT trial included children aged 2 weeks to 17 years with heart and/or lung failure and hyperglycemia and randomly assigned patients to a blood glucose target of 80 to 110 mg/dL vs 150 to 180 mg/dL. Hyperglycemia and blood glucose levels were managed by the CHECKS tool for all study patients. The Boston Children’s Institutional Review Board approved the study with informed parental consent (IRB # P00002310). We excluded patients in this analysis who were treated on CHECKS for less than 24 hours. We first reviewed overall compliance with the protocol as reported from the parent trial and a priori determined compliance <90% to be suboptimal. We compared baseline characteristics, blood glucose management and insulin therapy, and clinical outcomes between patients with high compliance (≥90%) and low compliance (<90%).

Data Collection

Baseline characteristics included demographic information, medical history, reason for ICU admission, Pediatric Risk of Mortality III-12 score, and invasive care therapies at the time of randomization. The CHECKS program template for all patients was maintained as a single version on a Boston Children’s Hospital server. Randomization occurred via a web-application built into the study website, with the treatment group assignment, patient weight, and insulin concentration being verified against a table of accepted values before being passed into a copy of the program template and downloaded to a bedside laptop. Thereafter, data from the individual sites was synchronized hourly with the central server and aggregated into a composite Structured Query Language database. From the CHECKS database, we calculated days of insulin therapy, average daily insulin dose, number of average daily glucose measurements, time to and time in the target range, and time-weighted blood glucose average. Clinical outcomes included ICU-free days through day 28, the primary outcome of the HALF-PINT trial, ventilator- and hospital-free days through day 28, and hospital mortality rate at 28 and 90 days. We also collected site-level data on factors that might affect compliance, which included whether sites used insulin protocols before HALF-PINT and whether nurses and/or resident physicians had the authority to enact therapeutic changes based on CHECKS recommendations.

Exploratory Granular Analysis of the CHECKS CDS Tool Compliance

The CHECKS system provided detailed instructions in response to patient-specific data. Instructions included when to enter Continuous Glucose Monitor, which measures glucose concentration in interstitial fluid, and/or bedside glucose meter (Nova StatStrip; Nova Biomedical), which measures whole blood glucose, results into the algorithm. CHECKS then provided detailed explicit instructions on how to adjust the insulin infusion rate to reach target blood glucose range (e-Fig 1) or dextrose rescue doses to avoid impending, or treat active, hypoglycemia. If the CHECKS algorithm (proportional-integral-derivative) recommended a change in insulin delivery, the user was prompted to verify the value using the bedside glucose meter and to follow the instructions accordingly. Once a recommendation was made, the bedside nurse and/or physician had the option to accept or override recommendation. In the event of an override, the user was provided with a pop-up menu (e-Fig 2) with the option to choose from a predetermined list of possible reasons or a comment box in which to explain the reason the recommendation was not being followed. More detail regarding the CHECKS system is described by Steil and Reifman.

We further conducted an exploratory analysis (both quantitative and qualitative) of the CHECKS CDS tool compliance by blood glucose measurement value and by compliance group to better understand the factors associated with CDS tool compliance. We reviewed user-specified reasons for CHECKS overrides. Due to the inadequate specificity of the embedded responses, we conducted a qualitative review of all overrides in the low-compliance group.

CDS Tool Framework Analysis

A random sample of 10 files of patients in the low-compliance group was reviewed by the primary author (E. Hirshberg) who then derived a grouping of themes that consistently emerged as applicable to CHECKS recommendations that were overridden. This list of eight themes was then sent to the entire writing group for refinement and consensus. A total of 12 themes were agreed on by the writing group before clinical review and category assignment. Once the themes were agreed on, an independent and more detailed interrogation of each of the overrides in the 30 patients with compliance <90% was completed. A total of three clinicians independently reviewed each chart and each overridden CHECKS recommendation. Each override was coded to all themes that applied, and these themes were analyzed quantitatively for descriptive purposes.

The 12 themes further fit into the larger categories common to evaluation of CDS tools and based on a modified version of the user (clinician judgment, environment, or clinician beliefs) function (mechanical issues), representation (work flow), and task analysis (algorithm, interface) (UFuRT) classification, plus a fifth category described as patient factors., UFuRT framework provides both quantitative and qualitative assessments and was designed to elucidate context and potential usability issues of computer-based systems.,

Statistical Analysis

We used Wilcoxon rank-sum tests or Fisher exact tests, as appropriate, to compare baseline characteristics and glycemia and insulin therapy variables between compliance groups. We used proportional-hazards, linear, and logistic regression, as appropriate, with adjustment for age group and severity of illness to compare hypoglycemia and clinical outcomes between compliance groups. All probability values are two-tailed and have not been adjusted for multiple comparisons. Analyses were performed with SAS software (version 9.4; SAS Institute Inc).

Results

Baseline Characteristics and CHECKS Compliance

Of 698 patients who received the HALF-PINT protocol, 658 (94%) were treated with CHECKS for ≥24 hours with a total of 100,998 instructions (Fig 1). The median number of CHECKS instructions per patient was 119 (interquartile range, 64 to 194). We observed an overall compliance with CHECKS instruction of ≥90% in 628 patients (95%) with 31 of 32 sites represented, with a median of 19 patients per site (range, 1 to 74). We observed compliance of <90% in 30 patients (5%) that represented 16 of 32 sites, with a median of two patients per site (range, 1 to 5). We observed that patients in the low-compliance group were younger (P = .02), more likely to have insulin infusing (P = .002), and less likely to require vasoactive infusions (P = .04) at the time of randomization. We observed no differences in other baseline characteristics that included the reason for ICU admission and Pediatric Risk of Mortality III-12 scores between the low-compliance and high-compliance groups (Table 1). There were no significant associations between CHECKS compliance group and clinical outcomes (e-Table 1).

Figure 1

Flow diagram of patients who received the Heart and Lung Failure-Pediatric Insulin Titration protocol. CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet; HALF-PINT = Heart and Lung Failure-Pediatric Insulin Titration.

Table 1

Baseline Characteristics According to Compliance Group

Characteristic	High Compliance (n = 628)	Low Compliance (n = 30)	P Valuea
Age at ICU admission, y, median (IQR)	6.4 (1.7-12.8)	2.6 (0.8-9.3)	.02
Age group, No. (%)			.08
<2 y,	177 (28)	14 (47)
2 to <7 y	156 (25)	7 (23)
7 to <18 y	295 (47)	9 (30)
Female, No. (%)	291 (46)	18 (60)	.19
Black race, No./total (%)	154/606 (25)	6/27 (22)	.82
Hispanic ethnic group, No./total (%)	147/627 (23)	6/29 (21)	.83
Baseline cognitive impairment (Pediatric Cerebral Performance Category >1), No. (%)	203 (32)	7 (23)	.42
Baseline functional impairment (Pediatric Overall Performance Category >1), No. (%)	231 (37)	13 (43)	.56
Any known genetic syndrome, No. (%)	116 (18)	5 (17)	1.0
Primary reason for ICU admission, No. (%)			.56
Respiratory, including infection	327 (52)	20 (67)
Cardiovascular, including shock	94 (15)	2 (7)
Neurologic	58 (9)	3 (10)
Trauma	57 (9)	1 (3)
Postoperative care	44 (7)	2 (7)
GI or hepatic	28 (4)	2 (7)
Other	20 (3)	0
Insulin at randomization, No. (%)	83 (13)	11 (37)	.002
Glucocorticoid therapy at randomization, No. (%)	319 (51)	19 (63)	.20
Inotropic support for hypotension at randomization, No. (%)	319 (51)	9 (30)	.04
Invasive mechanical ventilation (endotracheal tube or tracheostomy) at randomization, No. (%)	621 (99)	29 (97)	.31
Extracorporeal membrane oxygenation at randomization, No. (%)	31 (5)	0	.39
PRISM III-12 score, median (IQR)	12 (7-19)	10 (5-16)	.28
Risk of death based on PRISM III-12 score, %, median (IQR)	10.0 (2.9-34.7)	7.1 (2.7-23.5)	.72

IQR = interquartile range; PRISM III-12 = Pediatric Risk of Mortality III score from first 12 h in the PICU.

P values for the comparison between groups were calculated with the use of Wilcoxon rank-sum tests or Fisher’s exact tests, as appropriate.

Glucose Metrics and Compliance

Patients in the low-compliance group were more likely to have been assigned randomly to the lower blood glucose target treatment group in HALF-PINT and to have received insulin therapy compared with patients in the high-compliance group (Table 2). The average number of daily blood glucose measurements and the time to reach blood glucose target range were higher in the low-compliance group compared with the high-compliance group, although the percentage of time spent in the blood glucose target range was lower in the low-compliance group. There were no differences in the time-weighted glucose averages or the occurrence of severe hypoglycemia (<40 mg/dL) or any hypoglycemia (<60 mg/dL) between the low-compliance and high-compliance groups (Table 2). Adjustment for study site did not change the results appreciably.

Table 2

Glycemia and Insulin Therapy According to Compliance Group

Variable	High Compliance (n = 628)	Low Compliance (n = 30)	P Valuea
Percent compliance, %, median (IQR)	99.6 (98.5-100)	80.8 (73.3-87.0)	<.001
No. of recommendations, median (IQR)	121 (65-195)	72 (53-165)	.03
Randomized to lower target treatment group, No. (%)	302 (48)	24 (80)	<.001
Treated with insulin therapy, No. (%)	500 (80)	29 (97)	.02
No. of days of insulin therapy, median (IQR)	3 (1-7)	4 (3-7)	.21
Average daily insulin dose, units/kg/d, median (IQR)	0.23 (0.01-0.87)	0.51 (0.24-0.74)	.02
No. of average daily glucose measurements, median (IQR)	11.9 (6.6-17.6)	18.1 (14.4-20.3)	<.001
Time to the target range, h, median (IQR)	2.5 (1.0-6.5)	5.5 (3.0-17.0)	<.001
Time in the target range, %, median (IQR)	75 (58-92)	53 (39-58)	<.001
Time-weighted glucose average, mg/dL, median (IQR)	113 (103-128)	116 (109-132)	.16
Severe hypoglycemia (any relatedness), No. (%)	21 (3)	2 (7)	.35
Any hypoglycemia (any relatedness), No. (%)	100 (16)	9 (30)	.07

IQR = interquartile range.

For hypoglycemia, P values for the comparison between groups were calculated with the use of logistic regression with adjustment for age group and Pediatric Risk of Mortality III score from first 12 h in the PICU score. For other variables, P values were calculated with the use of Wilcoxon rank-sum tests or Fisher exact tests, as appropriate.

Of 30 sites contributing site-level data, eight sites (27%) used insulin protocols before HALF-PINT, and 17 sites (57%) gave nurses the autonomy without requiring physician confirmation to make changes to insulin or dextrose infusions based on CHECKS recommendations. Fifteen of the sites engaged resident physicians to participate in HALF-PINT; 67% of them gave resident physicians the autonomy to make changes to infusions based on CHECKS recommendations. These site-level factors were associated with statistically significant differences in compliance, but these differences were not clinically meaningful (e-Table 2).

Granular Compliance Quantitative: Compliance by Measured Blood Glucose Level

The low-compliance group had significantly lower compliance with CHECKS recommendations across all blood glucose ranges (<60, 60 to 79, 80 to 109, 110 to 129, 130 to 149, 150 to 179, 180 to 199, and ≥200 mg/dL) (Fig 2). In the low-compliance group, compliance with CHECKS recommendations was ≤69% in the two extreme blood glucose-measured ranges <60 mg/dL and ≥200 mg/dL and in the 110 to 129 mg/dL and 130 to 149 mg/dL ranges. Across both compliance groups, there were 337 recommendations when the blood glucose was <60 mg/dL, with 89.0% compliance overall. In this range, the low-compliance group received a higher than recommended dextrose dose 28% of the time, and the high-compliance group received a lower than recommended dextrose dose 6% of the time. In the low-compliance group, for the 192 recommendations when the blood glucose was ≥200 mg/dL, insulin was given at the recommended dose only 69% of the time, at a lower than recommended dose 24% of the time, and at a higher than recommended dose 7% of the time.

Figure 2

Compliance within blood glucose ranges according to compliance group. CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet; HC = high compliance; LC = low compliance.

Analysis of CHECKS Overrides: Reasons Provided by Care Team

The CHECKS interface prompted the user, with a drop-down menu, for a reason to override an instruction and had a comment section (e-Fig 2). The most common reason for CHECKS recommendation override was “Other” (63%), followed by concern on the part of the care team for hypoglycemia or recent hypoglycemia (23%).

Override Theme and UFuRT Classification by Research Team

The 30 patients with <90% compliance had 731 overrides, each of which underwent individual review and classification into 12 override themes by three independent clinicians (Table 3). The most common reason (89%) for noncompliance with the protocol under the UFuRT system was bedside clinician beliefs without clinical evidence or the “user” classification: for example, “fear of hypoglycemia” without any documented hypoglycemia or clinical evidence to support the concern. Workflow and clinical circumstances were less common, with 31% falling under the “representation” classification. Noncompliance was related to patient factors only 29% of the time.

Table 3

Override Themes and User, Function, Representation, and Task Analysis Classifications

User, Function, Representation, and Task Analysis Classification	Override Theme	Override Theme,aNo. (%)	User, Function, Representation, and Task Analysis Classification, No. (%)
User	Clinician factors (reasonable)	513 (70)	649 (89)
	Fear of hypoglycemia	307 (42)
	Clinician factors (unreasonable)	299 (41)
	Key stroke error	13 (2)
Function	Mechanistic/equipment factors	63 (9)	166 (23)
	“Distrust”	57 (8)
	Actual hypoglycemia	50 (7)
Representation	Clinical scenarios	194 (27)	226 (31)
	Work flow issue	36 (5)
Task analysis	CHECKS algorithm problem	94 (13)	118 (16)
	CHECKS instruction misunderstood/interface issue	37 (5)
Patient factorsb	Patient factors	213 (29)	213 (29)

CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet.

Three independent clinicians assigned themes to each override; more than one theme could be assigned per override (median, 2 [interquartile range, 2-3] themes per override).

This category was added to the classic User, Function, Representation, and Task Analysis classification as the modification.

Discussion

We present the first granular analysis of multicenter use of an electronic bedside clinical decision support tool (CHECKS) as a therapeutic intervention protocol in 32 PICUs during a randomized clinical trial (HALF-PINT). The overall high compliance with CHECKS for both HALF-PINT treatment groups supports the continued development and use of electronic CDS systems for research. The CHECKS interface facilitated rapid just-in-time training for a successful, high compliance (median, 99.5% per patient) intervention protocol deployment. Lower compliance with CHECKS occurred infrequently (5% patients) and was associated with less favorable glucose metrics that included more frequent blood glucose measurements, less time in blood glucose target, and more time to reach blood glucose target. Although unfavorable glucose metrics could result from specific patient characteristics, a granular analysis of these patients uncovered that most override decisions (89%) fell under the user classification and only 29% were categorized as patient-specific factors.

The CHECKS tool was integrated easily into practice as a therapeutic intervention and enabled a better understanding of the interaction between an electronic CDS tool and the bedside clinician., Our results argue against the claim that electronic CDS systems alienate the clinician from the patient interaction, because clinician independence was indicated with the frequent check of “Other” as the reason for noncompliance., Our data support previous speculation that electronic CDS tools can work well across several environments and multiple patient factors as long as appropriate training, education, and support are provided. Experience with CHECKS in a given enrollment site with CHECKS correlated with statistically significant differences in compliance, which supports work by others who note that trust or familiarity on the part of a user plays an important role in successful electronic CDS system adoption.,

We chose a just-in-time training model for CHECKS because any individual within the full group of bedside nurses in a given ICU could be assigned to a small volume of patients enrolled in the trial, often with limited or no prior experience with CHECKS. Online training modules consisted of short instructional videos paired with computer-scored competency tests and provided the capability of maintaining detailed records for on-site training of bedside staff in operation of CHECKS and the study devices. The clinical staff’s interaction with the protocol occurred continuously, 24 hours each day, at multiple locations, which made low-latency study monitoring a challenge. A web-based system was used to allow data to be aggregated and reviewed as it was being acquired; lead study investigators and data monitoring staff were alerted automatically to key issues such as protocol deviations as they occurred. This ultimately facilitated rapid iterative refinement of CHECKS.

Electronic CDS systems have been incorporated internationally by hospitals for improved error reduction, diagnostic accuracy, and better patient outcomes; with our data, multicenter PICU trials can be added to this list. Lower compliance with CHECKS was associated with less favorable glucose metrics and with patients who were of younger age or receiving insulin infusions at the time of randomization, which confirms the role of CDS tools to augment, not to replace, clinical judgment., The CHECKS user interface did not capture completely reasons for noncompliance with the built-in drop-down menu, because nearly two-thirds of the reasons were marked as “Other.” This, in addition to the high number of user classified overrides, suggests areas for additional improvement in CHECKS and informs the design of future electronic CDS for therapeutic interventions during research.

CHECKS was able to provide pertinent information to the bedside clinician at the time of care delivery that limited deviations. Importantly, the majority of CHECKS instructions occurred in optimal blood glucose clinical range of 80 to 180 mg/dL (including the target study ranges), which underscores the effectiveness of CHECKS in achieving target blood glucoses. Interestingly, compliance was <70% in the low-compliance group at both blood glucose extremes (<60 and ≥200 mg/dL) and in clinically acceptable ranges just outside of the trial targets (110 to 129 mg/dL and 130 to 149 mg/dL). These data suggest that users invoked independent recognition of an instruction’s potential to move the blood glucose away from a clinically acceptable range and, in those rare cases, overrode the instruction. Our data further demonstrate the bedside clinician’s ability to include factors unanticipated by CHECKS in deciding to override a recommendation. As expected, users cited fear of hypoglycemia as a reason for noncompliance highlighting the decision support (and safety) element of any CDS system.

A common theme in electronic CDS tool deployment is the notion that individual users may create additional cognitive struggle for themselves. Interestingly, excessive clinician burden, and the associated information overload, has justified electronic CDS system use in the past.,,, Our granular analysis of reasons for noncompliance and UFuRT classification validate the observation by others that clinician frustration must be managed appropriately during electronic CDS development., CHECKS serves as an example of a well-designed electronic CDS system for a therapeutic intervention that was used favorably.,, Further incorporating clinician specialists’ feedback in designing and addressing important performance gaps may enhance widespread adoption of electronic CDS systems in pediatric critical care trials.

This study has several limitations. Low compliance occurred in only a small number of patients. The preprogrammed reasons for noncompliance were not exhaustive, and the comment section rarely was completed. Most of the patients in the low-compliance group were assigned randomly to the low blood glucose target range during the parent trial, and clinician beliefs about the blood glucose target were not captured completely in CHECKS. CHECKS deployment was part of a National Institutes of Health-supported research investigation, which may not mimic other research environments. Our application of the UFuRT analysis represents a modified version of this analysis and may not be generalizable to other electronic CDS systems.

Interpretations

A well-designed electronic CDS system as a therapeutic intervention can be used successfully in pediatric critical trials with minimal just-in-time training. The CHECKS user interface worked well and facilitated easy identification of circumstances outside of the commonly expected clinical scenarios, which provided clear areas for future CHECKS refinement. Granular analysis of instruction compliance provided information about patient factors, clinical factors, workflow, and cultural or user factors that impacted study processes. It is possible that the same information could be extrapolated and used for the successful adoption of CHECKS into clinical practice. This study presents a pivotal step in navigating the possibilities of electronic CDS systems for future pediatric critical care trials. We suggest that future directions focus on electronic CDS systems for other therapeutic interventions and include investigations that export electronic CDS systems used for research into clinical practice, which would support the successful integration of critical care trial evidence into clinical practice.

Study Question: What information can be learned from a secondary analysis of an electronic clinical decision support (CDS) system performance when used as a therapeutic intervention during a multicenter PICU clinical trial?

Results: Electronic CDS system per patient compliance was high (99.5%) across a 32-site multicenter PICU trial. Patients with <90% compliance had common features and spent less time in the therapeutic target. Surprisingly, overrides were more often due to bedside clinician factors (89%) rather than patient factors (29%).

Interpretation: Electronic CDS systems represent an important mechanism for conducting reproducible complex pediatric critical care trials. Analysis of CDS system and bedside clinician interaction provides important areas for refinement of CDS systems and lessons for successful translation of research results into clinical practice.