Evaluation of a digital system to predict unplanned admissions to the intensive care unit: A mixed-methods approach.

BACKGROUND: We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) which aims to identify hospitalised patients most at risk of reversible deterioration. HAVEN combines patients' vital-sign measurements with laboratory results, demographics and comorbidities using a machine learnt algorithm.
OBJECTIVES: The aim of this study was to identify variables or concepts that could improve HAVEN predictive performance.
METHODS: This was an embedded, mixed methods study. Eligible patients with the five highest HAVEN scores in the hospital (i.e., 'HAVEN Top 5') had their medical identification details recorded. We conducted a structured medical note review on these patients 48 hours post their identifiers being recorded. Methods of constant comparison were used during data collection and to analyse patient data.
RESULTS: The 129 patients not admitted to ICU then underwent constant comparison review, which produced three main groups. Group 1 were patients referred to specialist services (n = 37). Group 2 responded to ward-based treatment, (n = 38). Group 3 were frail and had documented treatment limitations (n = 47).
CONCLUSIONS: Digital-only validation methods code the cohort not admitted to ICU as 'falsely positive' in sensitivity analyses however this approach limits the evaluation of model performance. Our study suggested that coding for patients referred to other specialist teams, those with treatment limitations in place, along with those who are deteriorating but then respond to ward-based therapies, would give a more accurate measure of the value of the scores, especially in relation to cost-effectiveness of resource utilisation.

Introduction

Early Warning Score (EWS) systems have been adopted internationally to identify patients who deteriorate in acute hospitals. EWSs combine individual vital-sign observations abnormalities into a single score. These scores are easily calculated at the bedside and alert clinicians to ward patients at high risk of clinical deterioration. However, EWSs do not account for additional patient risk factors, limiting their sensitivity and generating false alerts. Recent studies have shown how scoring systems that use additional variables from the Electronic Patient Record (EPR) (e.g., laboratory results and comorbidities) outperform EWS systems that rely solely on vital signs.3., 4., 5., 6., 7., 8., 9., 10.

We developed the Hospital Alerting Via Electronic Noticeboard (HAVEN). To develop the HAVEN model, we used standard statistical methods to select variables identified with a modified Delphi panel of experts and systematic review of existing literature. HAVEN uses a machine learnt algorithm to combine patients’ vital-sign measurements with their laboratory results, demographics, and comorbidities into a single risk score. It was developed and externally validated to predict impending cardiac arrests and unplanned transfers to the Intensive Care Unit (ICU).

There are clear guidelines for developing, validating and reporting prognostic models in healthcare. The guidance includes using appropriate statistical metrics (e.g., discrimination, calibration) for assessing model performance. However, rigorous methods for assessing their clinical utility and identifying factors that could improve model performance are limited.15., 16., 17. Moreover, in contrast to EWS systems that alert when the score exceeds a threshold, HAVEN ranks all patients in the hospital from highest to lowest risk of having an adverse event.

The aim of this study was to discover additional variables, not recognised during the data-driven development process, that would improve the performance of the HAVEN risk score. To do this, we reviewed the medical records of misclassified patients, that is, patients ranked highly by HAVEN but who were not admitted to the ICU; or patients who were never ranked highly by HAVEN but had an unplanned ICU admission. The general hypothesis was that identification of misclassification, such as patients false positively identified as in need of ICU care, would provide data to refine future identification and classification of ‘at risk’ patients.

Methods

A protocol for the study was published in advance. The study is reported according to the STrengthening Reporting of OBservational studies in Epidemiology (STROBE) guidelines (Appendix, Table 1).

Setting

The study was conducted at the John Radcliffe Hospital (JRH) which is part of the Oxford University Hospitals National Health Service Foundation Trust. The JRH is an 800-bed hospital that serves a local population of around 800,000 and a wider tertiary referral population.

Design

This was an embedded, mixed methods study. Methods of constant comparison were used during data collection and to analyse patient data extracted from the EPR.19., 20.

Participants

Participants were identified using a web interface that displayed real-time HAVEN risk scores for all currently admitted JRH patients. At 9am, on 27 out of the 39 (i.e., researcher available) days between July 27th and September 3rd, 2019, eligible patients with the five highest HAVEN scores in the hospital (i.e., ‘HAVEN Top 5′) had their identifiers recorded.

Inclusion and exclusion criteria

We included all adult patients (aged > 16 years of age) admitted to the study hospital. We excluded patients on high acuity, obstetric or paediatric wards at the time of the HAVEN Top 5 being recorded. High acuity excluded wards were the: intensive care unit, coronary care unit, specialty respiratory ward, high dependency unit, obstetric and paediatric wards.

Data collection & analysis

Both quantitative and qualitative methods were used to collect and analyse data. Data were collected between 48 hours and 60 hours post the HAVEN alert being documented (i.e., day three). Patients in the top five highest scoring patients each day were sub-divided into True Positive or False Negative, depending on whether they were admitted to ICU in this 48-hour period (i.e., False Positive were not admitted to ICU, True Positive were admitted to ICU). Concurrently, patients admitted to ICU during designated study days without being in the five highest ranked HAVEN scores during the preceding 48-hours were categorised as False Negative. (11) Cardiac arrest was not used to define False Positive or Negative because of its very low event rate within the hospital population.

Medical note review

Data collection was conducted using medical note review, by a single researcher (JM) who was a consultant in critical care medicine. The process was a modified version of the methods published by Hogan et al. Data were captured using Microsoft Excel and stored in a secure local server.

Core data

Core patient-centred and system-level data collected for the study were agreed a priori. These included patient age, sex, admission date, primary diagnosis, admitting team, elective or emergency admission status, prior surgery, past medical history, treatment limitations, current medications, radiological imaging, discharge status and Clinical Frailty Scores. System-level variables included admission specialities (e.g., medical or surgical) and inter-hospital transfers.

Constant comparison

Data were analysed using methods of constant comparison (CC). CC is based on Grounded Theory, first developed by Glaser & Strauss in 1968. CC (also called ‘Theoretical Sampling’) is a qualitative research method where data collection, coding and classification iterates and evolves as the study progresses. This process continues until a replicable ‘theory’ is produced which optimally explains the relationships within the data, specific to a question being asked of the data.

Data coding

Data coding was conducted in three phases: Open, Axial and Selective coding. Open coding compartmentalised the data into discrete (i.e., single) blocks or concepts. Axial Coding grouped the Open Codes and informed the iterative data collection process. The Selective Code (i.e., the Theory) was developed via analyses of the Axial Codes, which in this case was the classification process that best explained the False Positive cohort in this study. The data coding process was conducted using Taguette© (an opensource software alternative to NVivo©) and Microsoft Excel.

Additional data

As Open and Axial Coding proceeded during the data collection process, additional data fields were added.

Sample size

Sample size was dictated by pragmatic considerations of researcher capacity and availability and is described in detail in the protocol. Theoretical saturation (which is the point at which no new information or concepts are gained from ongoing data collection) was not considered when deciding sample size.

Results

140 patients were identified for inclusion in the cohort with 134 patients having medical note data extracted (6 were excluded because of incomplete medical record keeping). The median HAVEN score was 82, with a variance of 44.2 (Appendix, Table 2). The mean, median and minimum HAVEN scores for each study day are shown in the Appendix, Table 3. Of the 134 patient, 129 were not admitted to ICU (i.e., False Positives) and five were admitted to ICU (i.e., True Positives). The 129 False Positive patients then underwent constant comparison (Fig. 1).

Fig. 1

Diagram of patient groups.

False positives

The median age for the 129 False Positive cohort was 67 years and 79 were male. Forty (31%) were admitted under general medical teams, 47 (36%) under specialist medical teams, 12 (9%) under general surgical teams and 30 (23%) under specialist surgical teams (e.g., neurosurgery, cardiothoracic, ear-nose and throat, plastics, orthopaedics and trauma). Twenty-four (19%) patients had less than two known comorbidities, 63 (49%) had 2 – 4 comorbidities, 28 (22%) had 5 – 6 comorbidities and 14 (11%) had more than six comorbidities. The most common reason for hospital admission was community acquired pneumonia (17 patients, 13%). 47 patients (36%) had active Do Not Attempt Cardiopulmonary Resuscitation Orders. These data are summarised in the Appendix (Table 4).

Qualitative analysis

The Open, Axial and Selective codes produced from CC are shown in Table 1.

Table 1

Primary, Axial and Selective Codes generated via methods of constant comparison.

Primary Coding	Axial Coding	Selective Coding
Comorbidities/Past Medical History (compensated/decompensated/many/single)	Context	Outcomes
Frailty (baseline/current)
Functional status (exercise tolerance/independence)
Clinical Status (stable/unstable (critically/non critically))
Admission diagnosis/source/type	Patient Group
Demographics
Location
Presenting Complaint – Admission/In hospital
Readmission – Hospital
Service provider
Surgery (elective/emergency/pre and post theatre phase)
Emergency (airway/CVS/neuro/resp/other)	Event
Clinical deterioration
Medications - Type	Interventions
Intervention (medical/surgical/radiological/other)
Investigation (routine/non routine/single/multiple)	Pathology
Investigation - ABG (normal/abnormal/reason)
Investigation - Labs (normal/abnormal/reason)
Vital Sign Observation Set (abnormal à normal)	Physiology
Vital Sign Observation Set (normal/abnormal)
Vital Sign Observation Set (normal à abnormal)
Documentation (medical/specialist/surgical review)	Opinion
Documentation - clinical instability/stability
Death – Expected/Unexpected	Outcomes
ICU admission/referral
Speciality Referral/Transfer (geriatrics/palliative/cardiology/respiratory/other)
Treatment limitations – Location/Reason/Type
Response to therapy – Adequate/Inadequate/Unclear

CVS: cardiovascular system, neuro: neurology, resp: respiratory.

Eight Axial Codes emerged as stepwise explanatory factors for why a patient scored as high risk by HAVEN but did not require a subsequent ICU admission. Context grouped Open Codes specific to unchangeable but relevant patient centred variables that act as coefficients for decisions during the patient’s admission. Patient Group grouped Open Codes were descriptors. Events were defined as episodes during the patient’s admission with an objectively observed and documented explanation for a transition from a more to less stable clinical state. Interventions were documented actions taken by treating clinicians as a response to Events. Pathology were recorded investigation results (arterial blood gas, laboratory, routine, non-routine (i.e. tests other than standard blood count, liver function, biochemistry and basic coagulation studies), single and multiple. In each case the data provided an objective marker of disease severity and trajectory. Physiology was vital-sign observation sets, grouped into baseline (normal/abnormal) and trajectories (normal → abnormal and vice versa). These data provided an objective marker of physiological response to Pathology. Opinion was documentation in the EPR medical note, stating a clinical opinion regarding Pathology, Physiology and overall clinical status. Outcomes were Open Codes that were the result of Events, Interventions and Objective Data. These included change in location, death, ICU admission, specialist referral, treatment types, limitations and response (or otherwise) to ward based therapy. Outcomes represented the grouped Open Codes with the most relevant and highest accumulation of objective data regarding the patient’s clinical state, progress and reason for their not requiring an ICU admission. The accumulation of objective patient data is represented in Fig. 2.

Fig. 2

The Axial and Selective Code (or Theory).

Outcomes were divided into four main groups. Group 1 were clinically unstable patients but were referred to another specialist service with higher monitoring and intervention capabilities (e.g., cardiology and respiratory wards) for ongoing management. Group 2 patients were clinically unstable but received timely ward-based treatments, such as intravenous fluids or antibiotics and did not need ICU referral or admission. Patients in Group 3 were clinically unstable but were commonly frail and/or had documented treatment limitations, where it had been agreed that transfer to ICU would not be in their best interests. Group 4 were clinically stable and had no imminent need for ICU admission. Specific descriptions of the Outcomes are shown in the Appendix (Table 5).

Table 2 shows the distribution of False Positive patients across the four groups described above (i.e. Outcomes). Of the 129 patients, 37 had been referred to non-ICU specialist teams (Group 1), 14 were female and the median age was 60 years. Group 1 had fewer patients with significant frailty (i.e. Frailty Scale Score of 5 or greater) (n = 11, 29%). 38 patients responded to ward-based therapies (Group 2). This group also had a low number of patients with significant frailty (n = 5, 13%), with 16 females and median ages of 65 years. 47 patients had treatment limitation (Group 3). This groups had a higher proportion of significantly frail patients (n = 38, 80%) and a higher median age of 77 years. Of the seven patients who were in the FHR cohort but were objectively well (Group 4), none were severely frail. Four had been discharged from acute care settings post cardiac surgery but had residual, stable derangements in physiological parameters that were documented but of low clinical concern.

Table 2

False High Ranked patients (n = 129) not admitted to ICU evaluated using the results of the thematic analysis (Table 2). *Rockwood et al.

	Group 1:Referred to Specialist team	Group 2:Responded to ward-based therapies	Group 3:Treatment limitations	Group 4:Objectively well
Total	37	38	47	7
Age, median	60	65	77	71
Female	14	16	17	4
Frailty Scale Score* 1–4	26	33	9	3
Frailty Scale Score* 5–6	10	5	18	4
Frailty Scale Score* ≥ 7	1	0	20	0

Specialist team referrals are shown in the Appendix, Table 6. Documented reason for treatment limitations are shown in the Appendix, Table 7.

True positives

Five True Positives patients were identified. All were males aged between 51 and 69. Three were general surgical (two with post-operative respiratory failure and one with a seizure), one was neurosurgical, with a subdural haematoma with respiratory failure and one was a medical patient with respiratory failure. All had Frailty Scale Scores of 1 – 4.

False negatives

Three eligible patients were admitted to ICU during the study period and not ranked within the Top 5 by the HAVEN model. One patient had sudden, acute respiratory failure and ultimately received palliative care in the ICU, one patient developed an acute delirium in the day following a carotid endarterectomy (which did not produce any physiological derangement). The third patient became narcotised (with low respiratory rate) on the second day after a total knee replacement and required a naloxone infusion. Patient numbers in this low ranked cohort were insufficient for classification via qualitative methods.

Discussion

Key results

This study is one of very few to prospectively evaluate a machine learning algorithm using a mixed-methods approach. During the four weeks of the study, only five patients were False Negatives (i.e., admitted to the ICU ranked outside the Top 5 by HAVEN). However, 129 (96.2%) patients were False Positives (i.e., ranked in the Top 5 by HAVEN but were not admitted to an ICU). Of these False Positive patients, the majority were objectively unstable but only a small proportion required admission to ICU.

Using methods of constant comparison, we demonstrated there are four main groups of patients that explain these findings. Firstly, patients requiring referral to other specialist teams (e.g., cardiology). Secondly frail and often elderly patients with treatment limitations. Thirdly, patients who were promptly treated on the general ward and therefore avoided requiring an ICU admission. Finally, but infrequently, clinically stable patients with no need for ICU involvement.

Strengths and limitations

To our knowledge, this is the first study to apply an embedded mixed-methods approach to evaluate a prognostic risk score for deteriorating ward patients. Machine-learning in healthcare is increasingly being studied for potential use, but with the notable exception for Escobar et al., examples of successful, real-world implementation are uncommon.27., 28. Our mixed-methods approach to evaluating HAVEN could provide a complementary approach to assessing similar clinical decision support systems. An important limitation of this study was not considering theoretical saturation when calculating the sample size. This was a pragmatic decision based on researcher availability. However, the Theory emerged between 70 and 80 medical record reviews and was supported through the subsequent data collection and analysis, so saturation did occur.

Once implemented, a key indicator of HAVEN’s utility will be increased efficiency in the work of rapid response systems (RRS). Ranking patients by risk was used in this study to simulate the potential implementation of HAVEN via RRSs. This approach deviates from most current EWS systems, which direct clinical activity via thresholds or triggers. However, this method is not without its limitations. Analysis of retrospective data showed between one and two patients were admitted daily to the ICU from the general ward in the study hospital. Thus, we anticipated between three and four ‘false results’ from the outset but considered this acceptable given the intended aim was not to undertake a formal evaluation of system performance but to identify reasons for HAVEN error. This study was not designed to robustly evaluate the performance of HAVEN, but to understand the source of false positives, which will guide future development of the system. The size and resource availability of the Rapid Response Team, as well as hospital risk tolerance, will most likely be important contributors.

Comparison to current research

In a recently published systematic review, Hu et al. found 29 studies that reported the development and validation of EWS systems. Of these, five were comparable with HAVEN in that they developed a prognostic model using continuous, EPR linked data (both vital-sign observation sets and additional data such a laboratory results and demographics) to predict unplanned ICU admission, although this was often as part of a composite outcome.5., 8., 9., 29., 30. Three of these systems were tested in implementation studies9., 29., 30. but none used qualitative methods to extract and evaluate why the model generated false results. We believe our study is the first of this kind and provides insights into how to improve efficiency. Other authors have described many aspects of digital, prognostic modelling for the deteriorating patient and methods for incorporating treatment limitations when evaluating prognostic risk scores like HAVEN and note the importance of capturing these data electronically.

Implications for future research

Our results suggest several factors that could improve interventions targeting deteriorating ward patients:

The one third of patients with clinical instability, higher median age and high frailty scores represent and important sub-group. These patients require different but no less important escalations in care to specialist geriatricians, or other units with specialised expertise. In HAVEN, generating a Frailty score relied on having had a previous admission. Developing the utility to estimate frailty in all patients within the system will form part of future work. Numerous frailty measures are found in the literature but to our knowledge only one validated digital frailty score has been published.

Incorporating treatment limitations will be a requirement to improve efficiency when prognosticating for patients who would benefit from an ICU admission. The JRH did not have an electronic document capturing this data, so the HAVEN system was not able to incorporate it, however work is underway to add this information to the user interface and algorithm.

Why some patients respond to ward-based treatment and others do not is poorly understood. This group is important to identify because they are demonstrably salvaged by prompt intervention. All current metrics used in developing and validating early warning tools miss-classify these patients, which may lead to scores being developed that are less likely to recognise the very patients to whom clinicians should be called. Our study results suggest this distinction may inform how to reduce alert fatigue and inefficiency.

Novel outcomes and objective criteria that guide escalations in care, may reduce the bias associated with traditional, system-dependent outcome measures (e.g., unplanned ICU admission) in the evaluation. Novel outcomes, including specialty referrals and need for specific treatments (e.g., renal replacement threshold), should be considered when deriving and validating prognostic models in future. Broadening outcomes is the subject of future work. Additionally, better recognition of patient sub-groups (e.g., sepsis) and common patterns of deterioration, may also inform model development.

Ranking patients according to risk of risk for deterioration is a novel approach and requires further study.

Conclusions

HAVEN correctly identified clinically unstable patients on the hospital ward but only a small proportion required ICU admission. Qualitative analysis demonstrated that whilst these patients did not require ICU, they were correctly identified as objectively clinically unstable. Traditional, digital-only validation methods code this cohort as falsely positive in sensitivity analyses however our study showed this approach was limited in evaluating model performance. Our study suggested that coding for patients referred to other specialist teams and those with treatment limitations in place, along with those who are deteriorating but then respond to ward-based therapies, would improve the performance of similar models. We conclude that validating a method to accurately recognise and code for sub-groups of deteriorating patients from within the EPR would be an important next step in this field of research.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.