Red-flag sepsis and SOFA identifies different patient population at risk of sepsis-related deaths on the general ward.

Controversy exists regarding the best diagnostic and screening tool for sepsis outside the intensive care unit (ICU). Sequential organ failure assessment (SOFA) score has been shown to be superior to systemic inflammatory response syndrome (SIRS) criteria, however, the performance of "Red Flag sepsis criteria" has not been tested formally.The aim of the study was to investigate the ability of Red Flag sepsis criteria to identify the patients at high risk of sepsis-related death in comparison to SOFA based sepsis criteria. We also investigated the comparison of Red Flag sepsis to quick SOFA (qSOFA), SIRS, and national early warning score (NEWS) scores and factors influencing patient mortality.Patients were recruited into a 24-hour point-prevalence study on the general wards and emergency departments across all Welsh acute hospitals. Inclusion criteria were: clinical suspicion of infection and NEWS 3 or above in-line with established escalation criteria in Wales. Data on Red Flag sepsis and SOFA criteria was collected together with qSOFA and SIRS scores and 90-day mortality.459 patients were recruited over a 24-hour period. 246 were positive for Red Flag sepsis, mortality 33.7% (83/246); 241 for SOFA based sepsis criteria, mortality 39.4% (95/241); 54 for qSOFA, mortality 57.4% (31/54), and 268 for SIRS, mortality 33.6% (90/268). 55 patients were not picked up by any criteria. We found that older age was associated with death with OR (95% CI) of 1.03 (1.02-1.04); higher frailty score 1.24 (1.11-1.40); DNA-CPR order 1.74 (1.14-2.65); ceiling of care 1.55 (1.02-2.33); and SOFA score of 2 and above 1.69 (1.16-2.47).The different clinical tools captured different subsets of the at-risk population, with similar sensitivity. SOFA score 2 or above was independently associated with increased risk of death at 90 days. The sequalae of infection-related organ dysfunction cannot be reliably captured based on routine clinical and physiological parameters alone.

Introduction

Sepsis is defined as dysregulated host response to infection, resulting in acute organ dysfunction.[ This condition has been thoroughly studied in the intensive care unit (ICU), however, data from the general ward and emergency department (ED) setting is sparse.[ In addition, it is increasingly recognized that sepsis is even more prevalent and may be associated with greater mortality burden on the general wards.[ Controversy in sepsis research exists also regarding the best diagnostic and screening tool for sepsis outside the ICU.[ Development of a reliable tool is crucial as this condition still represents a major cause of morbidity and mortality.[

In addition to unclear sepsis prevalence and inaccurate identification tools, there has also been considerable debate regarding validity of sepsis definition used. We previously reported the results of a point prevalence study of all Welsh centers using the 2001 international consensus criteria for sepsis (SEPSIS-1) as well as comparison between SEPSIS-1 and the 3rd International Consensus Definitions for sepsis (SEPSIS-3) utilizing electronic data collection and real-time data monitoring.[ We found that between 4% and 5% of hospitalized patients had sepsis. Strikingly, different sepsis criteria identified different patient populations with a different 30-day mortality risk.[ While Sequential Organ Failure Assessment (SOFA) tool was found to be superior to systemic inflammatory response syndrome (SIRS) both in patient identification and predication of mortality outcome, it requires complete blood test analysis, potentially creating delays in patient treatment and some or many elements unavailable in resource constricted environments. The simplified quick SOFA (qSOFA) which was suggested for a ward-based use, however, was only able to identify about 10% of the at-risk population.[

Following the 2016 publication of NICE guidance NG51,[ the UK Sepsis Trust in communication with NICE launched a new screening tool based on NG51's “high” and “moderate” risk categories, which they have termed “Red Flag” and “Amber Flag” criteria respectively (See Table 1, Supplemental Content, presenting UK Sepsis Trust Red Flag Sepsis Screening Tool). The premise is that by identifying the sickest patients with the highest mortality risk early, treatment can be delivered as soon as sepsis-related organ failures are recognized, potentially reducing the mortality and the morbidity from sepsis. The Red Flag sepsis is based on clinical features and aims at triggering Sepsis Six bundle while confirmatory blood results are pending.[ The performance of Red Flag sepsis criteria has not been tested formally and analysis of their performance compared to SIRS, SOFA and a well-established track and trigger tool, the National Early Warning Score (NEWS) is needed.[

The primary objective of the study was to investigate the ability of Red Flag sepsis criteria to identify the patients at high risk of sepsis-related death in comparison to SOFA based sepsis criteria. Secondary objective was to compare the Red Flag sepsis to qSOFA, SIRS and NEWS scores and investigate patient characteristics to find factors influencing patient mortality.

Methods

Study design and participants

This was a large multi-center, point-prevalence study of at-risk populations of patients on general wards and ED. Fourteen hospitals in Wales, the United Kingdom with 24/7 consultant-level Emergency Department supervision and the facility to admit and treat any acutely unwell patient participated in the study. The study covered one 24-hour period (0800–0759 hours the following day) on October 18, 2017, during which we screened all patients presenting to the ED and being cared for in wards outside intensive care, pediatrics and psychiatry units. Our previous study was performed on October 19, 2016 which enabled us to analyze the validity of the study.[ The medical student data collectors recruited all patients with NEWS ≥3 in whom the treating clinical teams had a high clinical suspicion of an infection (documented as such in the medical or nursing notes). Patients were excluded if they were less than 18 years of age.

The details of the digital data collection platform developed for this study as well as description of the data collector training and performance during the study have been published previously.[ The data collected were obtained from medical and nursing records and included patients demographics and pre-admission characteristics, frailty (according to the Dalhousie Clinical Frailty Scale), physiological and laboratory data, presence of components of Red Flag, SOFA (where PaO2/FiO2 ratio was substituted with SpO2/FiO2 ratio as described by Pandharipande et al[) and SIRS sepsis criteria [ as well as input from the treating teams (use of screening tools by clinical team, antimicrobial prescription and delivery of the other “Sepsis Six” interventions [).[ A complete list of variables is presented in Supplementary Table 2. Presence of infection was determined by the investigators according to microbiology tests (blood, urine, respiratory, and wound cultures) and radiological imaging performed within the 48 hour period of the study. The amount of missing data was low and no assumptions were made for the missing data in line with similar recent critical care studies.[

Patients were grouped by the following clinical criteria: qSOFA group: qSOFA 2 or above; SOFA group: SOFA 2 or above; SIRS group: presence of 2 or more SIRS criteria; Red Flag sepsis group: presence of 1 or more Red Flag sepsis criteria. Due to the composition of the different clinical tools, patients could be grouped into none or more than 1 group. Our primary outcome was 90-day all-cause mortality.

The project was approved by the South Wales Regional Ethics Committee (16/WA/0071) and patients or legal representatives gave written informed consent. To facilitate linkage to national databases for the collection of follow-up data, patient identifiable data was collected and entered on to the secure data collection tool. The Defining Sepsis on the Wards project was prospectively registered with an international trial registry (ISRCTN86502304).

Statistical analysis

Categorical variables are described as proportions. Continuous variables are described as median and inter-quartile range. We compared the distribution of clinical and biochemical variables between survivors and non-survivors using Mann–Whitney U test or Chi-square test, as appropriate. A 2-tailed P-value <.05 was considered statistically significant. To assess the performances of the sepsis criteria to predict the primary end point, we calculated diagnostic performances (sensitivity, specificity, negative, and positive predictive values). We estimated the respective odds ratios (ORs) for the primary outcome within 90 days with a binary logistic regression with backward elimination model using mortality as a dependent variable. We determined goodness-of-fit of the model using the Hosmer–Lemeshow test. All statistical tests were calculated using SPSS 23.0 (SPSS Inc., Chicago, IL).

Results

In our study we screened 7055 patients over the 24-hour study period in the 14 Welsh hospitals. 459 patients had NEWS ≥3 and documented clinical suspicion of infection and were recruited in the study. Baseline characteristics are summarised in Table 1.

Table 1

Baseline characteristics of the patients for all recruited patients and comparing the non-survivors with survivors within 90-days.

Different sepsis criteria identify different patient populations

Out of 459 patients with NEWS ≥3 and high suspicion of infection, Red Flag criteria were present for 246 patients (53.6%), SOFA for 241 (52.5%), qSOFA for 54 (11.8%), and SIRS for 268 (58.4%). Some patients were identified by more than 1 criteria and 55 patients were not picked up by any criteria (Fig. 1). 131 patients had clinically proven infection. Among patients with SOFA 2 or above, 69/241 (28.3%) patients had a proven infection, in comparison to 75/246 (30.8%) patients fulfilling Red Flag criteria, 50/268 (18.7%) patients with 2 or more SIRS criteria and 13/54 (24.1%) patients with qSOFA 2 or above. Out of the 55 patients, who did not fulfill any of the sepsis criteria, 21 had clinically proven infection.

Figure 1

Patients identified using different scoring criteria. Red Flag sepsis, SOFA = sequential organ failure assessment score, SIRS = systemic inflammatory response syndrome criteria. SEPSIS-1 is defined by SIRS ≥2. SEPSIS-3 is defined by SOFA ≥2; Red Flag is defined by ≥1 Red Flag criteria. qSOFA was omitted in the diagram as patients identified by this criteria were also captured by SOFA score.

Different sepsis criteria were associated with different patient mortality

We analyzed the 90-day mortality for patients identified by each tool. First, we calculated the mortality for each score. The mortality was highest for patients identified by qSOFA (31/54, 57.4%), followed by SOFA (95/241, 39.42%), then Red Flag Sepsis (83/246, 33.7%) and SIRS (90/268, 33.6%). We calculated the diagnostic performances of each sepsis tool to predict death at 90 days in Table 2.

Table 2

Diagnostic performances of different sepsis definitions and clinical tools for the prediction of mortality at 90 days.

We also analyzed the mortality stratifying patients into groups of patients identified by one scoring criteria only, 2 scoring criteria and 3 criteria (Fig. 2). The mortality was highest for “Red Flag + SOFA” group (19/37, 51.4%), followed by “Red Flag + SOFA + SIRS” (43/103, 41.8%), “SOFA only” (14/42, 33.3%), “SOFA + SIRS” (19/59, 32.2%), “SIRS only” (17/57, 29.8%), “SIRS + Red Flag” (11/49, 22.5%). Interestingly “Red Flag only” mortality was (10/57, 17.5%), which was lower even than the mortality of patients not picked up by any criteria (11/55, 20%). We performed a separate analysis of the mortality for the “qSOFA only” group—patients identified by qSOFA but not SIRS or Red Flag criteria, not taking the presence of SOFA criteria into consideration. This revealed that qSOFA identified only 4 patients with 100% mortality rate.

Figure 2

Patient mortality depending on the scoring criteria. RF = Red Flag sepsis, SIRS = systemic inflammatory response syndrome criteria, SOFA = sequential organ failure assessment score.

We investigated whether lower mortality of the screening tool is associated with earlier recognition and treatment of sepsis. The Sepsis Six completion was low for all the screening tools: Red Flag Sepsis 15.5% (38/256), SOFA 18.7% (45/241), qSOFA: 20.4% (11/54), and SIRS: 16.4% (44/268).

We also explored the rate of microbiology confirmed infection for each screening tool. We obtained information about the blood, sputum, urine, wound, and CSF cultures for 405 out of 459 patients. There were 152 blood cultures, 39 sputum cultures, 133 urine cultures, and 20 wound swab cultures performed. The rate of positive cultures was similar for each screening tool: Red Flag Sepsis 9.5% (21/222), SOFA 10.7% (21/196), qSOFA 8.2% (4/48), SIRS 10.1% (24/237).

Survival analysis

We also aimed at identifying potential factors that could have an impact on the survival. In addition to baseline patient characteristics presented in Table 1, we analyzed patient clinical results, patient reserve (clinical frailty score and implemented limitations of care), and input from the treating team (Table 3).

Table 3

Patient characteristics and management comparing the non-survivors with survivors within 90-days.

Analysis of patient observations and laboratory results identified some parameters as statistically significant but the differences in the respective medians were too small to acknowledge the analytes as clinically significant (See Table 3, Supplemental Content, which shows patient observations and laboratory results).

We used a binary logistic regression model to independently assess variables that in a univariate analysis were associated with mortality and we felt were clinically important. All selected variables described both patient pre-admission characteristics and the screening tools most predictive of patient mortality. Consequently, we included age, frailty score, DNA-CPR, ceiling of care order and SOFA ≥2.

We found that older age was associated with death with OR (95% CI) of 1.03 (1.02–1.04); higher frailty score 1.24 (1.11–1.40); DNA-CPR order 1.74 (1.14–2.65); ceiling of care 1.55 (1.02–2.33); and SOFA score of 2 and above 1.69 (1.16–2.47). The result of the Hosmer–Lemeshow test indicated good fit of the model.

Discussion

To our knowledge, this is the first prospective study comparing the diagnostic performance of “Red Flag sepsis criteria” to SEPSIS-1 and SEPSIS-3. We found that SEPSIS-1 criteria identified the most patients in at-risk population (58.4%), followed by Red Flag (53.6%) and SEPSIS-3 (52.5%). There was a significant overlap between the criteria, although 55 patients (12%) with high NEWS scores, clinical suspicion of infection and 20% 90-day mortality were missed even after application of all scoring criteria, similarly to previous studies.[

Red Flag sepsis criteria were developed to aid healthcare providers in the ED and on the general wards, as well as in the community to identify patients at high risk of deterioration. Notably, this process was undertaken without the benefit of a robust clinical dataset, in contrast with the SOFA score data used to develop the SEPSIS-3 definitions.[ Following the publication of the NICE guidance, a number of UK organizations implemented the Red Flag sepsis tool as their screening for patients who might have organ dysfunction due to infection and are high risk of death.[ Our results cast some doubt on the potential effectiveness of this approach. SOFA score 2 or above either on its own or in conjunction with other tools was consistently associated with high risk of death. Red Flag Sepsis criteria failed to identify almost half of this population, signaling deficit in its intended use as a tool sufficiently sensitive to identify patients at high risk of deterioration. Moreover, of the 4 different clinical tools, only SOFA score 2 or above was independently associated with increased risk of death at 90 days. We argue that based on our results, the Red Flag Sepsis tool should not be used in isolation as a triggering tool for Sepsis Six bundle as it can potentially miss up to 45% of patients who are at high risk of death following an infectious episode; furthermore, it is not independently associated with adverse outcome. Patients with Red Flag signs did not have more formal sepsis screening or more reliable Sepsis Six bundle delivery, nor had more microbiologically confirmed infections compared to patients identified by other clinical tools. Although it can be argued that the currently operational sepsis screening in Wales is based on the original SEPSIS-1 definition and use the SIRS criteria, our data suggest, that regardless of the clinical criteria used, patients at high risk of deterioration are not reliably screened. It also appears that despite clear warning signs present, their treatment is not universal, although appears to be improved compared to previous years.[

Investigating diagnostic performance of the different clinical tools is crucial as failure to recognize sepsis could lead to excess deaths but conversely, over-triage of suspected sepsis is likely to burden general wards and ED and risks detracting from care of other patients. As there is still no gold standard diagnostic test, clinicians rely mostly on nonspecific physiological and laboratory abnormalities among patients with suspected or definite infection.[ Regardless of the clinical tool used, either based on established organ failure scores or more intuitive clinical categories, less than a third of the patients who fall into these categories had clinically proven infection. This could be partly explained by the relatively low rate of microbiological sampling and radiological examinations. Importantly, patients who did not score on any of the clinical tools had similar rate of clinically proven infection. These results emphasize the need for more sensitive screening tools which could highlight the patients at true risk of deterioration secondary to an infectious insult. Intriguingly, a recent multicentre study was able to accurately utilize an algorithm incorporated to an electronic health record system, to predict the development of sepsis in multiple clinical environments.[ However, the true heterogeneity of patient, pathogen and disease-related factors in sepsis can probably only be appreciated with sophisticated analysis of host response at a molecular level.[ Creating a tool which could be useful for identification of sepsis and prediction of mortality due to this condition has important implications for clinical care, epidemiologic and clinical studies, public health surveillance, and quality improvement programs.[

Analysis of the survival revealed that completion of a screening tool and Sepsis Six bundle is associated with improved outcome, in line with previous findings, but case numbers are too small for a definitive comparison[ It is apparent that pre-admission trajectories are the most clinically significant predictors of patient survival, similar to recent findings in the critically ill population.[ In the ward and ED setting, older age and decreased patient reserve and expectation of the treating physicians (higher frailty score, higher rate of DNA-CPR and higher rate of ceiling of care implementation) strongly predicted patient mortality.

The strengths of our study include wide participation of centers and prospectively collected patient information. Our study has high internal validity as our previous 2 studies applied the same methodology and recruited similar number of patients in the same centers.[ Moreover, to our knowledge, this is the first study to investigate the diagnostic performance of Red Flag sepsis criteria and their real-life utility in clinical practice.

Our study has some limitations. First, in designing a dataset small enough to maintain data collector participation and data reliability, we could have missed some determinants of sepsis. Second, we only recruited patients who had NEWS score of 3 and above, and therefore we could also have missed patients with sepsis who had a lower score.[ However, recent data suggest that NEWS cut-off of 3 may be the optimal trigger to screen patients for sepsis in the ED,[ and a NEWS score of 3 is recommended as an escalation trigger by NICE and used in the Sepsis Trust's Red Flag Sepsis pathways.[

Conclusion

The different clinical tools captured different subsets of the at-risk population, with similar sensitivity and with similar precision for confirmed infection. SOFA, but not “Red Flag Sepsis” or SIRS based clinical criteria identify patients with suspected infection at high risk of deaths on the general ward. SOFA score 2 or above is independently associated with increased risk of death at 90 days. The sequalae of infection-related organ dysfunction cannot be reliably captured based on routine clinical and physiological parameters alone.

Acknowledgments

The authors want to thank Maria Hobrok, Igor Otahal, Vincent Hamlyn, Luis Macchiavello, Rhidian Jones, Orsolya Minik, Paul Morgan, Gemma Ellis, Ceri Brown, Chris Littler, Chris Subbe, Judith E Hall, Robert Lundin, Meshari Alsaeed, Hannah Williams, Arfa Ayob, Nor Farzana, Sweta Parida, David Lawson, Michal Mazur, Lezia D'Souza, Bethan Ponting, Terrance Lau, Ruairidh Kerrigan, Lucy Morgan, Roshan Vindla, Claudia Zeicu, Becky James, Amirah Amin Ariff, Wan Binti Wan Azzlan, Charlotte Collins, Elizabeth Wickens, Alisa Norbee, Aliya Zulkefli, Thomas Haddock, Megan Thomas, Matthew Lee, Akshita Dandawate, Holleh Shayan-Arani, Ellie Taylor, Oliver Kyriakides, Rachel Price, Ffion Haf Mackey, Emily Haines, Samuel Chun, Chantal Roberts, Alessia Waller, Laura Heekin, Kathy Wang, Rhianna Church, Shrina Patel, Marianne Broderick, Hannah Whillis, Daniel Craig Hathaway, Emel Yildirim, Caitlin Atkins, Adam George Mounce, Anoopama Ramjeeawon, Ndaba Mtunzi, Duncan Soppitt, Jay Hale, Jack Wellington, Robert Buchanan Ross, Danielle Lis, Sophie Stovold, Sam Vickery, Nia Jones, Alice O’Donnell, Monty Cuthbert, Osa Eghosa, Muhammad Karim, Lowri Williams, Louise Tucker, Thomas Downs, Ailsa MacNaught, Swagath Balachandran, Abbie Shipley, Jennifer Louise Kent, Talea Roberts, Samuel Tilley, Bethany Davies, Emma Withers, Krishna Parmar, Lucie Webber, Thomas Grother, Harry Smith, Rachel Watson, Natalie Hoyle, Rym Chafai El Alaoui, Omar Marei, Emma Kirby, Anna Gilfedder, Lydia Maw, Sarah O’Connor, Abigail Rogers, Zhao Xuan Tan, Clare Chantrill, Amal Robertson, Jonathan Foulkes, Rahana Khanam, Jomcy John, Isobel Sutherland, Sarah Hannah Meehan, Huria Metezai, Hannah Dawson, Eloise Baxendale, Karishma Khan, Pan Myat, Andrew Forrester, Oliver Moore, Hse Juinn Lim, Aimee Owen, Faris Hussain, Nima-banu Allybocus, Maneha Sethi, Umair Asim, Emelia Boggon, Ibrahim Alkurd, Genevieve Lawrence, Jade Brown, Lowri Hughes Thomas, Emily Murphy, Evie Lambert, Jeremy Guilford, Beth Payne, Mariam Almulaifi, Arwel Poacher, Sashiananthan Ganesananthan, Sara Tanatova, Jasmine Kew, Megan Eilis Clark, Ellen Hannay, Olesya Godsafe, Christina Houghton, Francesca Lavric, Rachel Mallinson, Hei Man Priscilla Chan, Eshen Ang, Niamh McSwiney, Yin Yin Lim, Zong Xuan Lee, Svetlana Kulikouskaya, Nur Zulkifli, Sheryl Lim, Lim Xin, Thomas Chandy, Abduahad Taufik, James Cochrane, Sioned Davies, Samuel Willis, John Lynch, Sieh Yen Heng, Alex Cooper, Henrik Graf von der Pahlen, Isabella Talbot, Robin Gwyn Roberts, Jessica Sharma Smith, Aisling Sweeney, Cerian Roberts, Paul McNulty, Peter Havalda.

Author contributions

Conceptualization: Maja Kopczynska, Ben Sharif, Tamas Szakmany.

Formal analysis: Maja Kopczynska.

Funding acquisition: Tamas Szakmany.

Investigation: Maja Kopczynska, Naomi Spencer, Sian Cleaver, Jessica Davis, Amit Kurani, Camilla Lee, Carys Durie, Jude Joseph-Gubral, Angelica Sharma, Lucy Allen, Billie Atkins, Alex Gordon, Llewelyn Jones, Amy Noble, Matthew Bradley, Henry Atkinson, Joy Inns, Harriet Penney, Carys Gilbert, Rebecca Walford, Louise Pike, Ross Edwards, Robyn Howcroft, Hazel Preston, Jennifer Gee, Nicholas Doyle, Charlotte Maden, Claire Smith, Nik Syakirah Nik Azis, Navrhinaa Vadivale, Ceri Battle, Ronan Lyons, Paul Morgan, Richard Pugh, Tamas Szakmany.

Methodology: Maja Kopczynska, Ben Sharif, Tamas Szakmany.

Project administration: Tamas Szakmany.

Software: Ben Sharif.

Writing – original draft: Maja Kopczynska, Tamas Szakmany.

Writing – review & editing: Ceri Battle, Ronan Lyons, Paul Morgan, Richard Pugh.