Scores for sepsis detection and risk stratification - construction of a novel score using a statistical approach and validation of RETTS.

BACKGROUND: To allow early identification of patients at risk of sepsis in the emergency department (ED), a variety of risk stratification scores and/or triage systems are used. The first aim of this study was to develop a risk stratification score for sepsis based upon vital signs and biomarkers using a statistical approach. Second, we aimed to validate the Rapid Emergency Triage and Treatment System (RETTS) for sepsis. RETTS combines vital signs with symptoms for risk stratification.
METHODS: We retrospectively analysed data from two prospective, observational, multicentre cohorts of patients from studies of biomarkers in ED. A candidate risk stratification score called Sepsis Heparin-binding protein-based Early Warning Score (SHEWS) was constructed using the Least Absolute Shrinkage and Selector Operator (LASSO) method. SHEWS and RETTS were compared to National Early Warning Score 2 (NEWS2) for infection-related organ dysfunction, intensive care or death within the first 72h after admission (i.e. sepsis).
RESULTS: 506 patients with a diagnosed infection constituted cohort A, in which SHEWS was derived and RETTS was validated. 435 patients constituted cohort B of whom 184 had a diagnosed infection where both scores were validated. In both cohorts (A and B), AUC for infection-related organ dysfunction, intensive care or death was higher for NEWS2, 0.80 (95% CI 0.76-0.84) and 0.69 (95% CI 0.63-0.74), than RETTS, 0.74 (95% CI 0.70-0.79) and 0.55 (95% CI 0.49-0.60), p = 0.05 and p <0.01, respectively. SHEWS had the highest AUC, 0.73 (95% CI 0.68-0.79) p = 0.32 in cohort B.
CONCLUSIONS: Even with a statistical approach, we could not construct better risk stratification scores for sepsis than NEWS2. RETTS was inferior to NEWS2 for screening for sepsis.

Introduction

Sepsis is a medical emergency, requiring early recognition and care. Its clinical features can vary and be vague and therefore difficult to detect. Sepsis is common, especially at Emergency Departments (ED), where it constitutes between 2–13% of encounters [1, 2].

Various risk stratification tools have been used in order to recognize and prioritize patients with risk of progression to sepsis. Any sepsis scoring system must be very sensitive, as this disease has both a high mortality rate and delayed treatment dramatically worsens outcome [3]. The National Early Warning Score 2 (NEWS2) is a modification of National Early Warning Score (NEWS) and is a risk stratification score for the probability of clinical deterioration of for example development of sepsis [4]. Although NEWS2 has the best accuracy for sepsis detection of the commonly used risk stratification scores, we have previously shown that a substantial portion of patients with sepsis goes undetected with a cut-off of NEWS2 ≥5 [5].

Risk stratification scores for deterioration have most commonly been constructed from analysis of the most abnormal vital signs in a given observation period [4, 6, 7]. Rapid Emergency Triage and Treatment System (RETTS) builds upon a vital sign based score and is widely used for triage at EDs in Sweden [7, 8]. RETTS uses a four-graded scale describing the levels of abnormal vital signs in combination with scores assigned for common Emergency Signs and Symptoms (ESS). RETTS is used to assign acceptable wait times before physician assessment, with ‘red’ the highest priority (S1 Table) [7].

More sophisticated but with greater complexity is to include biomarkers in the risk stratification scores. Lactate has been tested for inclusion in quick sequential organ failure assessment (qSOFA), but appears to offer no benefit [6]. Heparin-Binding Protein (HBP), another potential biomarker, is a granule protein which is released by neutrophils in response to bacterial products and neutrophil adhesion [9-13]. It has been found to be a superior biomarker than lactate to predict the development of sepsis in the ED [14].

We wanted to explore different approaches for sepsis risk stratification tools i.e. a statistical approach and the inclusion of biomarkers and symptoms in risk stratification tools.

By using two prospective, observational, multicentre cohorts of patients with blood drawn for biomarkers at presentation in an ED, the aims of this study were to a) develop a sepsis risk stratification tool based on the most predictive, minimal set of vital signs, lactate and HBP plasma levels and b) validate RETTS as able to predict both sepsis and subsequent 30-day mortality.

Materials and methods

Ethics

Ethical approvals were obtained from the regional ethical board in Lund (approval number 2010/205 and 2014/4), the regional ethical board in Bern (approval number KEK 315/14) and the regional ethical board in Vancouver (approval number H11-00505). All included patients gave written informed consent in cohort A. The study is in accordance with the approval and the informed consents. The study is also in accordance with the approvals and informed consents for cohort B. In cohort B, included patients all gave written informed consent or, if unable to give informed consent, next-of-kin was asked for permission. For patients that died without being able to leave informed consent, the use of data and samples was requested at the local ethics committee.

Patients

Cohort A (Suspected infection in the ED)

Data from an observational, multicentre convenience trial of biomarkers were used. Patients were included prospectively between 2011–2012 at EDs. The study has been described in detail elsewhere [14]. Patients at the Swedish sites were included in this analysis.

In summary, patients ≥18 years with a suspected infection and at least one of Systemic Inflammatory Response Syndrome (SIRS) criteria or self-reported fever or chills, were included at presentation.

The following data were registered at enrolment: data on demography, comorbid conditions, medication and vital signs. Samples for laboratory testing were ordered. Retrospectively, data on organ dysfunction, treatment, intensive care, infection diagnosis and 30-day mortality were gathered from medical records and a national death registry.

Cohort B (acutely ill undifferentiated ED patients)

Data from a multicentre, observational, convenience trial of sepsis biomarkers were used. Between 2015–2016, patients were enrolled at EDs in the study that has been described in detail elsewhere [15]. Patients who fulfilled at least one of the following requirements, were included: Respiratory rate >25 breaths per minute, heart rate >120 beats per minute, altered mental awareness, systolic blood pressure (SBP) below 100 mmHg, oxygen saturation (SaO2) <90%, or <93% if ongoing treatment with oxygen. Both infected and non-infected patients were included. Data on demography, comorbid conditions, medication and vital signs were registered at enrolment and samples for laboratory testing were ordered. Retrospectively, data on organ dysfunction, treatment, intensive care, infection diagnosis and 30-day mortality were gathered from medical records and a national death registry.

At inclusion, patients from the Swedish sites in cohort B were categorized according to the ESS algorithms for RETTS. Patients from the Swedish sites were also followed up for 30-day mortality.

Definitions

Sepsis was defined as a probable or verified infection based on clinical presentation, laboratory results, microbiological samples and radiologic examinations, and an acute organ dysfunction of no other apparent or pre-existing cause. We applied the organ dysfunction definitions from the former sepsis-2 definition since this was consensus criteria at the time the data were collected, although we did not require 2 SIRS criteria due to its lack of validity [16, 17]. For the definitions of organ dysfunction see S2 Table. Patients with infection that died or were treated at the intensive care unit within 72 hours were also regarded as suffering from sepsis. We applied this combined outcome since it is most probably sepsis that causes death or intensive care in infected patients but it can potentially be undetected. This combined outcome will hereafter be referred to as sepsis. To validate the combined outcome for sepsis, we also analysed risk stratification scores for predicting a maximal rise in SOFA score of 2 or more within 72h from admission together with infection (Sepsis-3 definition) in cohort B. However, in cohort A SOFA score was only available at baseline. Hence, the validation of the combined outcome for sepsis was made for the presence of sepsis at inclusion in this cohort.

For PaO2/FiO2 ratio for patients with SaO2 90–94% and for patients with COPD and SaO2 87–95% and simultaneous oxygen supply in cohort B, the Severinghaus equation was used [18].

Since acute neurological dysfunction was direct part of scores being evaluated, it was excluded as an organ dysfunction defining sepsis.

RETTS was defined according to S1 Table. ESS47 covers infection and categorize patients as red if presenting with petechiae and concomitant signs of infection.

NEWS2 was used as reference risk stratification score since it has the best accuracy for sepsis detection of the risk stratification scores widely used and validated (S3 Table)[5].

Biomarkers included in the construction of a candidate risk stratification tool were lactate and HBP. The biomarkers lactate and HBP were selected for their prognostic abilities in sepsis and the availability of point-of-care testing [14, 15]. Although in the study, HBP was analysed with Enzyme-Linked ImmunoSorbent Assay (ELISA) at a centralized laboratory. Lactate and blood sample analyses used for organ dysfunction were analysed at the clinical chemistry departments at each hospital.

Statistical methods

The least absolute shrinkage and selector operator (LASSO) method was used for construction of a candidate risk stratification tool. The LASSO method avoids correlating covariates from being included in a prediction model [19, 20]. LASSO was preceded by locally weighted scatterplot smoothing (LOWESS) regression for eligible parameters with reference to the outcome, sepsis including admittance to intensive care due to an infection or infection-related mortality within 72 hours from enrolment. This generated a smooth curve for selection of intervals for parameters to include in the LASSO regressions. Variables were dichotomized within the selected intervals and entered to the LASSO regressions. The LASSO included a 10-fold cross-validation with areas under the receiver operating characteristic curves (AUC) optimization and was iterated 50 times. All values optimizing AUC in more than 50% of the LASSO analyses and with a coefficient ≥0.05 were then entered in to a second set of LASSO regressions, unless the values were adjacent. If adjacent values, the one with the higher coefficient was chosen. Values included in more than 50% of the second set of LASSO regressions and with a coefficient of ≥0.05 within 1 standard error (SE) from max AUC were selected. These values were given a score proportional to their coefficients generated by the second set of LASSO regression and rounded to the closest integer. The cut-offs for these scores were set to also require scores from more than one parameter.

AUC, sensitivity, specificity and their 95% confidence interval (CI) were calculated. Odds ratio (OR) was calculated for the risk stratification scores relation to 30-day mortality. P-values were calculated with Chi2-test when comparing proportions and using the formula of Delong for comparison of AUC. P-values below 0.05 were regarded as significant.

Patients with missing values, among the vital signs or biomarkers included in the risk stratification scores or included in construction of a risk stratification score, were excluded in the primary analyses.

Multiple imputation of missing values that are part of the risk stratification scores were executed using predictive mean matching and logistic regression with 20 imputation sets and the performances of different risk stratification scores were calculated in a sensitivity analysis.

The performances of risk stratification scores were analysed in imputed data sets. AUC’s were calculated as medians of the pooled data and 95% CI’s for all imputed datasets.

Analyses were performed using glmnet package, R version 3.4.0 (The R Foundation for Statistical Computing) and SPSS software system version 23.0 (IBM, Armonk, NY).

Results

Cohort A

Patient inclusion, exclusion and classification as sepsis is outlined in Fig 1. A total of 506 patients with an infection and complete data on vital signs and laboratory analyses were included in the primary analyses in cohort A. 283 (56%) had at least one comorbidity, 12 (5%) were admitted to ICU and 4 (2%) died. 233 fulfilled the combined outcome referred to as sepsis (infection-related organ dysfunction, intensive care or death within the first 72h after admission). Of the 233 patients with the combined outcome, 228 had infection-related organ dysfunction, 4 were treated in the ICU and one died without organ dysfunction being registered. The two independent infectious disease physicians who reviewed the data attributed the death and the ICU-care for these five patients to infectious diseases.

Fig 1

Flow chart.

RETTS (Rapid Emergency Triage and Treatment System).

Patient characteristics are presented in Table 1. Table 2 compares RETTS to NEWS2 for sepsis. RETTS had lower AUC than NEWS2 for sepsis detection, 0.74 (95% CI 0.70–0.79) and 0.80 (95% CI 0.76–0.84), respectively, p = 0.05.

Table 1

Patient characteristics.

	Cohort A			Cohort B
	Without sepsis n = 273	With sepsis n = 233	p	Without sepsis n = 306	With sepsis n = 129	p
Age, median	52	70		72	77
Female, n (%)	106 (39)	118 (51)		142 (46)	64 (50)
Comorbidities n (%)
Diabetes	29 (11)	45 (19)	<0.01	54 (18)	33 (26)	0.06
Cardiovascular disease	31 (11)	78 (33)	<0.01	146 (48)	71 (55)	0.16
Renal Disease	15 (5)	30 (13)	<0.01	32 (10)	22 (17)	0.06
Liver Disease	3 (1)	1 (0)	0.40	6 (2)	7 (5)	0.05
Malignancy	20 (7)	24 (10)	0.24	46 (15)	17 (13)	0.62
Immunodeficiency	9 (3)	9 (4)	0.73	3 (1)	3 (2)	0.27
Respiratory Disease	21 (8)	32 (14)	0.03	69 (23)	38 (29)	0.13
No comorbidities	126 (46)	97 (42)	0.31	103 (34)	23 (18)
Organ dysfunction, n (%)				*
No organ dysfunction		5 (2)		154 (50)	0 (0)
Neurologic		37 (16)		64 (21)	32 (25)
Cardiovascular		186 (80)		80 (26)	83 (64)
Respiratory		61 (26)		59 (19)	81 (63)
Renal		25 (11)		52 (17)	25 (19)
Hematological		22 (9)		12 (4)	12 (9)
Hepatic		7 (3)		5 (2)	4 (3)
Intensive Care n (%)		12 (5)		20 (7)	12 (9)
3-days mortality n (%)		4 (2)		4 (1)	13 (10)

* Organ dysfunction without infection

Table 2

Accuracy of risk stratification scores for sepsis, 95% CI within brackets.

	RETTS RED	RETTS <RED	NEWS2 ≥5	NEWS2 <5	SEWS ≥7	SEWS <7	SHEWS ≥10	SHEWS <10
Cohort A
With sepsis	66	167	152	81	*	*	*	*
Without sepsis	12	261	55	219	*	*	*	*
Sensitivity	28 (23–35)		65 (59–71)		*	*	*	*
Specificity	95 (92–98)		80 (75–85)		*	*	*	*
AUC	0.74 (0.70–0.79)		0.80 (0.76–0.84)		*	*	*	*
p (NEWS2)	0.05		reference
30-day mortality	4	10	10	4	12	2	11	3
30-day survival	74	418	197	295	146	346	257	235
OR 30-day mort	2.3 (0.7–7.4)		3.7 (1.2–12.1)		14.2 (3.1–64.3)		3.4 (0.9–12.2)
Cohort B
With sepsis	74	55	108	21	108	21	107	22
Without sepsis	149	157	198	108	212	94	162	144
Sensitivity	57 (48–66)		84 (76–90)		84 (76–90)		83 (75–89)
Specificity	51 (46–57)		35 (30–41)		31 (26–36)		47 (41–53)
AUC	0.55 (0.49–0.60)		0.69 (0.63–0.74)		0.67 (0.61–0.73)		0.73 (0.68–0.79)
p (NEWS2)	<0.01		reference		0.63		0.32
30-day mortality	20	8	26	2	26	2	27	1
30-day survival	158	166	221	103	240	84	186	138
OR 30-day mort	2.6 (1.1–6.1)		6.1 (1.4–26.0)		4.5 (1.1–19.6)		20.0 (2.7–149.2)

* Not able to validate, derived in this cohort

Construction of a new risk stratification score

A LOWESS regression for eligible parameters with reference to the composite outcome (S1 Fig) resulted in the following values included in the first set of LASSO regressions as ordinal variables: 5-year-age groups between 40 and 90, heart rate from 60 to 140 in groups of five following frequencies, SBP from 70 to 120 in groups of two following mmHg, diastolic blood pressure (DBP) from 40 to 90 in groups of two following mmHg, respiratory frequencies from 20 to 40 in groups of two following rates and mental status classified as ordinal groups 1–5 according to whether normal, agitated, confused, drowsy or unconscious.

The LASSO regression was cross-validated and repeated with values dominating the first set of LASSO. The second set of LASSO regressions generated values that were given a score proportional to their coefficients. This resulted in construction of a new risk stratification scores (Tables 3 and 4) called Sepsis Early Warning Score (SEWS) for the model without HBP, and Sepsis HBP-based Early Warning Score (SHEWS) for a model with HBP. The cut-off values for risk of sepsis were set at 7 points for SEWS and 10 points for SHEWS.

Table 3

SEWS, Early Warning Score.

	1	2	3	4	5
Age	>45			>60	>80
Mental Status			Confused or drowsy
Respiratory Frequency					>24
SBP (mmHg)			<106	<100
DBP (mmHg)	<78	<58
Heart Rate		>110

Table 4

SHEWS, Sepsis Heparin binding protein-based Early Warning Score.

	1	2	3	4	5	6	7	8
Age	>45			>60	>80
Mental Status			Confused or drowsy
Respiratory Frequency					>24
SBP (mmHg)		<106				<100
DBP (mmHg)	<78		<56
Heart Rate		>110
HBP (ng/mL)			>26			>30	>48	>54

Sensitivity analyses

For 49 patients excluded with missing data, multiple imputation was performed and validation for RETTS was repeated in the imputed data sets. For missing data and demographics and outcome of patients with missing data, see S4 and S5 Tables.

Variables imputed were systolic blood pressure, diastolic blood pressure, heart frequency, respiratory frequency, temperature, mental status, SaO2, oxygen treatment, lactate, age and HBP. Other parameters in the imputation although not imputed were comorbidities and outcome. Predictive Mean Matching were used for multiple imputation and logistic regression for binary variables. The models were validated by plots of imputations and iterations.

Data were assumed to be missing at random, conditional on observed data in the imputation model.

For results of analyses using multiple imputation, see S6 Table. AUC’s were calculated as medians of the pooled data and 95% CI’s for all imputed datasets.

the analyses using multiple imputation yielded similar estimates as the analyses of the original data.

The validation of the combined outcome for sepsis was made for the presence of sepsis in this cohort, by analyzing risk stratification scores for detecting the combined outcome at inclusion compared to a presumed rise in SOFA score of 2 or more and infection (Sepsis-3 definition) at inclusion. When comparing risk stratification scores for detecting the combined outcome at inclusion to sepsis-3 at inclusion there were no differences except for SHEWS which did not perform as well in detecting sepsis-3 (AUC 0.79) as in detecting the combined outcome (AUC 0.86) and was not significantly higher than AUC for RETTS (S7 Table).

When excluding the 5 patients included in the composite outcome for sepsis but without registered organ dysfunction, as not possible to classify, the results were not significantly changed , RETTS AUC 0.75 (95% 0.71–0.79), NEWS2 AUC 0.80 (95% CI 0.77–0.84).

Cohort B

Data on patient characteristics are presented in Table 1. Of 435 patients, 184 (42%) had a diagnosed infection. 129 (30%) experienced sepsis within 72 hours from enrolment (Fig 1). All 129 patients categorized as sepsis had organ dysfunction and not only death within 72h or ICU without organ dysfunction being registered (composite outcome).

When compared to cohort A, these patients had more comorbidities 309 (71%) and were more often admitted to ICU 32 (7%) or died 17 (4%).

SEWS and SHEWS were evaluated for their ability to predict sepsis. For cross tabulations, sensitivity, specificity, AUC and OR see Table 2. The new score with HBP, SHEWS, yielded the highest AUC (0.73). The new score without HBP (SEWS) (AUC 0.67) was inferior to NEWS2 (AUC 0.69), although not significantly. RETTS (0.55) had the lowest AUC of the validated scores (p <0.01).

The patients from the Swedish sites (n = 354) were also classified according to the ESS. The ESS algorithms are used in combination with vital signs in RETTS and can give patients a higher priority due to symptoms. Only one patient was newly classified as red RETTS due to the ESS algorithm for infection. Eleven patients were classified as red due to other causes than infection, most often dyspnéa or chest pain with new left bundle branch block, ST-elevation or widespread, sudden pain with vegetative symptoms or unconsciousness. Neither did affect RETTS’ performance.

When validated among the sub group of (retrospectively diagnosed) infected patients (n = 182), the discriminating capacity of NEWS2 and SHEWS did not change significantly, AUC 0.72 (95% CI 0.64–0.80) and 0.74 (95% CI 0.66–0.82), but the AUC for RETTS rose to 0.61 (95% CI 0.52–0.70), which was still inferior to NEWS2, p = 0.02.

As in cohort A, multiple imputation of missing data for 54 patients excluded with missing data were performed. The analysis of performance of the different scores for the primary outcome was repeated in the imputed data sets.

For missing data and demographics and outcome of patients with missing data, see S4 and S5 Tables.

Variables imputed were systolic blood pressure, diastolic blood pressure, heart frequency, respiratory frequency, temperature, mental status, SaO2, oxygen treatment, lactate. Other parameters in the imputation although not imputed were comorbidities, outcome, age and HBP. Predictive Mean Matching were used for multiple imputation and logistic regression for binary variables. Data were assumed to be missing at random, conditional on observed data in the imputation model. The models were validated by plots of imputations and iterations.

AUC’s were calculated as medians of the pooled data and 95% CI’s for all imputed datasets.

The analyses using multiple imputation rendered similar result as the complete cases analyses, thus the imputation analysis did not change the relation of AUCs for RETTS, SHEWS, SEWS and NEWS2. For complete results see S6 Table.

To validate the combined outcome for sepsis, we also analysed risk stratification scores for predicting a presumed rise in SOFA score of 2 or more <72h together with infection (Sepsis-3 definition) in this cohort. When comparing risk stratification scores for predicting the combined outcome to sepsis-3 there were no differences except for NEWS2 which performed better in predicting sepsis-3 (AUC 0.79) compared to the combined outcome (AUC 0.69) (S8 Table).

Discussion

We used a statistical approach in constructing a new risk stratification score for sepsis. The new score, SHEWS, had the highest accuracy for detection and prediction of sepsis in the ED, although not statistically superior to NEWS2. Both SHEWS and NEWS2 performed significantly better than RETTS for sepsis detection, even though RETTS combines vital signs with symptoms for risk stratification.

RETTS has had little previous validation for sepsis detection, but our results are in concordance with the previous studies. Askim et al. demonstrated a sensitivity of 34% and specificity of 95% for red RETTS for detecting severe sepsis, in a cohort of infected patients at ED [21]. In the present study, RETTS had a sensitivity of 28% and a specificity 95% in cohort A and a sensitivity of 57% and specificity of 51% among infected patients in cohort B. A study of RETTS’ association with the final hospital diagnosis in children demonstrated sepsis to be the most frequent inappropriately classified, time-dependant condition [22].

Interestingly, when constructing a new risk stratification score for sepsis, the LASSO regression found, among others, exactly the same parameters and cut-off values that are included in the qSOFA score, with the exception of respiratory rate which differed slightly, ≥22 and >24 respectively [23]. This confirms the importance of the parameters and cut-off values selected for qSOFA in predicting sepsis, yet studies have indicated qSOFA to be too simplified [5, 6]. These parameters and cut-off values are also components of NEWS2 [4].

Perhaps it is not possible to reach higher AUC for sepsis recognition than NEWS2, using easy available vital signs and the biomarkers included in the statistical model. Other scores have not been able to demonstrate superiority over NEWS2 for prognostic accuracy for deterioration in infected patients [5, 24].

HBP increased the performances for the new risk stratification score, but we only used HBP and lactate as eligible biomarkers when constructing the new score. Other biomarkers for sepsis might have better additive effects.

One promising approach for a risk stratification score is the use of machine learning for continuous sampling of data and calculation of real-time scores. Targeted real-time early warning score has been demonstrated to perform well for both sepsis and septic shock and near real-time automated SOFA has proven to have a strong agreement with manual SOFA score calculation [25, 26]. Unfortunately, continuous sampling is hampered in the ED, due to the short observation period [25, 27].

Strengths of this study are the validation of the scores both among infected patients and among unselected patients at the ED with sepsis according to clinical assessment. A sepsis risk stratification score performs most likely better among infected patients, otherwise it is supposed to identify infection as well. However the initial assessment of whether the patient is infected or not has often proved to be wrong, why it is important to validate these scores among both infected and unselected ED patients [28].

A major limitation is that patients in the study that are considered as falsely classified as positive by the risk stratification scores, still can suffer from other time-critical conditions especially in the cohort which includes patients with infection as well as without. Also, the inclusion criteria for the cohort B are largely coherent with RETTS. These weaknesses might lead to RETTS being estimated as more sensitive but less specific. When analysing the sub cohort of infected patients in cohort B, the performance of RETTS was however not significantly changed. The low 30-day mortality resulted in difficulty to reliably assess the secondary outcome, to validate the risk stratification scores for 30-day mortality.

There were missing data at admission on variables for the validated risk stratification scores, although not a high proportion <7%. We performed multiple imputation as a sensitivity analysis to address this problem.

We assumed data to be missing at random. This is not testable, but becomes more reasonable within a model like ours that includes several characteristics, including predictors and the outcome. The probability that vital signs and laboratory values are missing is related to other parameters measured and hence missing at random is a valid assumption. Therefore, we can use multiple imputation to estimate the effects on the missing vital signs and laboratory values. Multiple imputation is commonly used when evaluating clinical risk scores [6, 29].

Another limitation is the use of the sepsis-2 definition. The proposed definition of sepsis-3 provoked a fierce discussion and the new definition has not by far been officially accepted by all professional associations. Even though we now believe sepsis-3 to be helpful for clinicians, it was not published at the time the data were collected and accordingly we use the sepsis-2 definition. We did however perform a sensitivity analysis which compared our combined outcome for sepsis to sepsis-3 in cohort B and for detection of sepsis at inclusion in cohort A. The sensitivity analysis did not change the results.

RETTS is commonly used in Sweden, but its external validity outside of Sweden is limited. However, we wanted to explore different approaches for sepsis risk stratification tools i.e. the addition of biomarkers and symptoms.

It is not evident which statistics best reflect the performance of risk stratification scores. Risk stratification or triage scores used for sepsis detection needs a high sensitivity since the consequences of delayed diagnosis are severe. The scores are simple and cheap, although the “cost” of a false positive score is the risk of another patient being lower prioritized for clinical evaluation. In this study the positive predictive value was at lowest one third, so the number needed to evaluate never exceeded 3. Hence, the sensitivity remains the crucial metrics in this study which was far too low for RETTS for sepsis detection. The other risk stratification score performed better, but there is still scope for improvement.

Conclusion

Even with a statistical approach, we could not construct better risk stratification scores for sepsis than NEWS2. RETTS was inferior to NEWS2.

LOWESS curves.

(TIFF)

Click here for additional data file.

RETTS.

(DOCX)

Click here for additional data file.

Organ dysfunction definition.

(DOCX)

Click here for additional data file.

NEWS2.

(DOCX)

Click here for additional data file.

Missing data.

(DOCX)

Click here for additional data file.

Demographics and outcome of patients with missing data.

(DOCX)

Click here for additional data file.

AUC for different risk stratification scores in multiple imputation datasets.

(DOCX)

Click here for additional data file.

AUC for different risk stratification scores detection of combined outcome for sepsis compared to sepsis-3 definition, cohort A.

(DOCX)

Click here for additional data file.

AUC for different risk stratification scores prediction of combined outcome for sepsis compared to sepsis-3 definition, cohort B.

(DOCX)

Click here for additional data file.

18 Dec 2019

PONE-D-19-32440

Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS

PLOS ONE

Dear Dr Mellhammar,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Thank you for this interesting and well-done submission. Overall, this paper addresses an important topic. I like the premise and the results, in the sense that they highlight that a more complex (in this case, biomarker-based) approach to prognostication is not necessarily superior simply because of the complexity.

In addition to comments raised by the reviewers, I have two questions/concerns related to assumptions made in methods section. The first (and more significant), is the part of the definition for sepsis, as stated in lines 157-159. I wonder if this definition is overly broad, in the sense that there are many reasons for death in 72 hours or treatment in the ICU, in the setting of "infection" that are not directly sepsis. I think that further justification for this definition is required (can the charts for patients who qualified as sepsis by these criteria be reviewed?); if this is not possible, the potential ramifications of this limitation need to be discussed.

My second concern, although related to a secondary outcome, is about the decision to use multiple imputation for missing data in the prediction scores--I wonder if the data is actually missing not at random (MNAR). Patients who are less ill may have fewer tests/labs ordered or had VS documented less frequently (or they may be repeated less often) and thus existing values may be more deranged as they represent the sicker spectrum of the population. As such, what data does exist may not be truly representative of the entire population thereby potentially biasing the results of the multiple imputation data sets; this would be particularly true for lactate in this case. Perhaps this could explain why the two analyses (with/without) missing data produced similar results?

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for including your ethics statement:

'the regional ethical board in Lund (approval number 2010/205 and 2014/4).

the regional ethical board in Bern (approval number KEK 315/14)

the regional ethical board in Vancouver (approval number H11-00505).'

a. Please amend your current ethics statement to confirm that your named institutional review board or ethics committee specifically approved this study.

b. Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

3. In the ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records used in your retrospective study.

Specifically, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent.

If patients provided informed written consent to have data from their medical records used in research, please include this information.

4. Thank you for stating the following in the Competing Interests section: "Bertil Christensson, Per.Åkesson, and Adam Linder are listed as inventors on a patent on the use of HBP as a diagnostic tool in sepsis filed by Hansa Medical AB. All other authors have declared no relevant conflicts of interest."

We note that you have a patent relating to material pertinent to this article.

a. Please provide an amended statement of Competing Interests to declare this patent (with details including name and number), along with any other relevant declarations relating to employment, consultancy, patents, products in development or modified products etc. Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

b. This information should be included in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

5. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

6. Please upload a new copy of your Supporting Information Figure as the detail is not clear. Please follow the link for more information: http://blogs.PLOS.org/everyone/2011/05/10/how-to-check-your-manuscript-image-quality-in-editorial-manager/

7. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The study utilized sophisticated statistical techniques to develop risk stratification model for sepsis. there are several concerns from me.

1. The score developed in the study requires lactate and HBP, which is difficult to obtain at the very beginning. in particular the HBP is not routinely measured. thus, the authors must discuss that the applicability of the model is limited. lactate is not routinely measured for infection but suspected sepsis.

2. "Patients with infection that died or were treated at the intensive care unit within 72 hours were also regarded as suffering from sepsis."---do you validate this statement? patients can die from other reasons but with mild infection signs. For example patients can have severe brain injury with aspiration pneumonia, but after 48 hours after ICU entry he died due to the trauma.

3. The statistical modeling lacks reference, how did you choose the cutoff points based on LOWESS for continous variables (Zhang Z, Zhang H, Khanal MK. Development of scoring system for risk stratification in clinical medicine: a step-by-step tutorial. Ann Transl Med. 2017;5(21):436. doi:10.21037/atm.2017.08.22)?

4. With multiple imputation you obtain multiple dataset, how did you combine the results? for example different datasets can give you different model coefficients and AUCs.

Reviewer #2: Strengths: This study builds on the authors' previous work in an area of critical need (i.e. improved ED-based diagnostic or screening tools for sepsis). The authors utilize statistically-sound methods to accomplish their stated goals, especially with regards to the derivation of the new decision tools (SEWS and SHEWS).

Limitations:

- I am partially uncertain why the authors decided to include a validation of RETTS in this report, in a secondary aim that is only somewhat related to the primary aim of deriving the new scores (SEWS and SHEWS) and comparing them to NEWS. This reviewer does not practice in Sweden, however, where RETTS is (as I understand it) a commonly-used system and perhaps its importance to the paper is simply the relative ubiquity of RETTS in the clinical setting of study (i.e. Swedish EDs).

-In line with above, there is some limitation in external validity outside of Sweden. This is not in any way a disqualifying issue, but probably should be mentioned in the limitations section (if page-limits allow).

- The largest concern I have is the way the sepsis 2 and sepsis 3 definitions were used here. The authors create a criterion-standard definition of sepsis which is largely based on the organ dysfunction parameters of the Sepsis 2 definition, as well as additional criteria including dying in the ICU with an infection. They then performed a sensitivity analysis comparing this definition (referred to as the combined outcome in the manuscript) with a SOFA score > 2 to adjudicate whether their definition of sepsis was concurrent with the Sepsis 3 definition.

They cite that the Sepsis 3 definition was not around at the time of sampling these patients as the reason for using their definition. I find this to be a generally unsatisfactory justification. Namely, just because sepsis 3 had not been published at time of sampling, it does not follow that it cannot be applied in a retrospective study. If the reason for using a sepsis-2 based combined definition was simply because of the authors' concerns about the validity of Sepsis 3 (a reasonable viewpoint), then I would advise them to so state. Alternatively, if feasibility of SOFA in the ED was the concern that would also be valid, but that is not stated either. Given that a sensitivity analysis using SOFA was performed, feasibility would seem to not be a major issue. Finally, throughout the article the endpoint of SOFA > 2 is used to describe the Sepsis 3 definition, however this is not accurate. The actual sepsis 3 definition is a rise in SOFA >= 2 from baseline. This helps to prevent patients with chronic disease (e.g. chronic kidney disease, cirrhosis) from being automatically classified as "septic" as soon as they hit the door (i.e. without any actual acute worsening of their end-organ function). As an example, if the definition of SOFA>=2 was used (instead of increase in SOFA >=2) it would mean every patient with stage IV-V CKD with an infection would automatically be labeled as septic in the sample even if they had no acute organ dysfunction. It is unclear if this was considered by the authors, but is eminently important since the rates of chronic renal disease were significantly higher in the septic (by combined outcome) patients in cohort A, and rates or liver disease were higher in cohort B. It may be difficult to accurately assess baseline values for SOFA with a retrospective design (i.e. to adjudicate rise vs. baseline points on SOFA) but even if so this needs to be addressed as a limitation.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Nicholas E Harrison

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

4 Jan 2020

Dear Dr Ehrman,

Thank you for your letter and the comments from you and the reviewers on the manuscript entitled “Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS”.

We appreciate your engagement and interest in our work, we are pleased that the reviewers found the manuscript interesting and we are grateful for the several constructive and helpful comments. The manuscript has now been revised according to the suggestions by you and the reviewers. Our response to the specific points is given below:

Editors comment:

The first (and more significant), is the part of the definition for sepsis, as stated in lines 157-159. I wonder if this definition is overly broad, in the sense that there are many reasons for death in 72 hours or treatment in the ICU, in the setting of "infection" that are not directly sepsis. I think that further justification for this definition is required (can the charts for patients who qualified as sepsis by these criteria be reviewed?); if this is not possible, the potential ramifications of this limitation need to be discussed.

Authors reply:

Of the 233 patients with the combined outcome in cohort A, 228 had infection-related organ dysfunction, 4 were treated in the ICU and one died without organ dysfunction being registered. Of the 5 patients included in the composite outcome for sepsis but without registered organ dysfunction, 3 had culture-verified infection and 2 had probable infection. The two independent infectious disease physicians who reviewed the data attributed the death and the ICU-care for these five patients to infectious diseases. All patients with the combined outcome in cohort B had infection-related organ dysfunction registered. Perhaps this is an overly broad definition and these patients could have been regarded as not possible to classify, it would not change the result significantly, RETTS AUC 0.75 (95% 0.71-0.79), NEWS2 AUC 0.80 (95% CI 0.77-0.84). Detailed data has been added p 12, line 232-236, p 17, line 326-328 & 334-335.

Editors comment:

My second concern, although related to a secondary outcome, is about the decision to use multiple imputation for missing data in the prediction scores--I wonder if the data is actually missing not at random (MNAR). Patients who are less ill may have fewer tests/labs ordered or had VS documented less frequently (or they may be repeated less often) and thus existing values may be more deranged as they represent the sicker spectrum of the population. As such, what data does exist may not be truly representative of the entire population thereby potentially biasing the results of the multiple imputation data sets; this would be particularly true for lactate in this case. Perhaps this could explain why the two analyses (with/without) missing data produced similar results?

Authors reply:

In the imputation models we included comorbidities, outcome, systolic blood pressure, diastolic blood pressure, heart frequency, respiratory frequency, temperature, mental status, SaO2, oxygen treatment, lactate, age and HBP. We assume that the missingness for variables in the analysis model can be assumed to be missing at random (MAR) conditional on observed data in the imputation model. The MAR assumption is not testable, but becomes more reasonable with imputation models that include a wide range of characteristics, including predictors, the outcome and auxiliary variables like in our model. The result section has been changed to make it clearer, p 16, line 313-314, and p 18, line 365-366.

Reviewer 1

Reviewers comment 1. The score developed in the study requires lactate and HBP, which is difficult to obtain at the very beginning. in particular the HBP is not routinely measured. thus, the authors must discuss that the applicability of the model is limited. lactate is not routinely measured for infection but suspected sepsis.

Authors reply:

We wanted to explore different approaches for sepsis risk stratification tools i.e. a statistical approach and the addition of biomarkers and symptoms to risk stratification tools. The biomarkers, lactate and HBP, were selected for their prognostic abilities in sepsis and the fact that, even if not generally available, there exists point-of-care testing. However, even if HBP increased the performances for the new risk stratification score, it was not statistically superior to NEWS2, why the result does not support the need for availability of HBP and lactate point-of-care testing for the use in these risk stratification tools for sepsis. The manuscript has been changed in order to clarify this, p 6, line 110-111.

Reviewers comment 2. "Patients with infection that died or were treated at the intensive care unit within 72 hours were also regarded as suffering from sepsis."---do you validate this statement? patients can die from other reasons but with mild infection signs. For example patients can have severe brain injury with aspiration pneumonia, but after 48 hours after ICU entry he died due to the trauma.

Authors reply:

As mentioned above, Of the 233 patients with the combined outcome in cohort A, 228 had infection-related organ dysfunction, 4 were treated in the ICU and one died without organ dysfunction being registered. Of the 5 patients included in the composite outcome for sepsis but without registered organ dysfunction, 3 had culture-verified infection and 2 had probable infection. The two independent infectious disease physicians who reviewed the data attributed the death and the ICU-care for these five patients to infectious diseases. All patients with the combined outcome in cohort B had infection-related organ dysfunction registered. Perhaps this is an overly broad definition and these patients could have been regarded as not possible to classify, it would not change the result significantly, RETTS AUC 0.75 (95% 0.71-0.79), NEWS2 AUC 0.80 (95% CI 0.77-0.84). Detailed data has been added p 12, line 232-236, p 17, line 326-328 & 334-335.

Reviewers comment 3. The statistical modeling lacks reference, how did you choose the cutoff points based on LOWESS for continous variables?

Authors reply: We agree and references has been added, p 9, line 193. The LOWESS curves were used to assess the relevant intervals for which dichotomization of the included continuous variables could be relevant. Each continuous variable was then dichotomized into several dummy binary variables with cut-offs spanning this relevant interval. These dummy variables were then entered into to LASSO regression. The LASSO-regression did then select the most relevant cut-offs based on statistical testing. Hence, the cut-offs for continuous variables were not manually chosen but were chosen through statistical testing in the LASSO model.

Reviewers comment 4. With multiple imputation you obtain multiple dataset, how did you combine the results? for example different datasets can give you different model coefficients and AUCs.

Authors reply: AUC’s were calculated as medians of the pooled data and 95% CI’s for all imputed datasets. This information has now been added p 11, line 220-221, p 16, line 315-316 and p 18, line 367.

Reviewer 2

Reviewers comment 1. I am partially uncertain why the authors decided to include a validation of RETTS in this report, in a secondary aim that is only somewhat related to the primary aim of deriving the new scores (SEWS and SHEWS) and comparing them to NEWS. This reviewer does not practice in Sweden, however, where RETTS is (as I understand it) a commonly-used system and perhaps its importance to the paper is simply the relative ubiquity of RETTS in the clinical setting of study (i.e. Swedish EDs).

-In line with above, there is some limitation in external validity outside of Sweden. This is not in any way a disqualifying issue, but probably should be mentioned in the limitations section (if page-limits allow).

Authors reply: As the reviewer state RETTS is commonly used in Swedish EDs for triage and has been introduced as base for sepsis alert systems where patients, with the highest priority according to RETTS, are prioritized and treated according to sepsis bundles (Rosen qvist M. et al. Sepsis Alert – a triage model that reduces time to antibiotics and length of hospital stay Infectious diseases. 2017;49(7):507-13). Apart from the clinical setting, with limited validity outside of Sweden, we wanted to explore different approaches for sepsis risk stratification tools i.e. the addition of biomarkers and symptoms. The limitation has been addressed in the discussion, p 22, line 433-435.

Reviewers comment 2. The largest concern I have is the way the sepsis 2 and sepsis 3 definitions were used here. The authors create a criterion-standard definition of sepsis which is largely based on the organ dysfunction parameters of the Sepsis 2 definition, as well as additional criteria including dying in the ICU with an infection. They then performed a sensitivity analysis comparing this definition (referred to as the combined outcome in the manuscript) with a SOFA score > 2 to adjudicate whether their definition of sepsis was concurrent with the Sepsis 3 definition.

They cite that the Sepsis 3 definition was not around at the time of sampling these patients as the reason for using their definition. I find this to be a generally unsatisfactory justification. Namely, just because sepsis 3 had not been published at time of sampling, it does not follow that it cannot be applied in a retrospective study. If the reason for using a sepsis-2 based combined definition was simply because of the authors' concerns about the validity of Sepsis 3 (a reasonable viewpoint), then I would advise them to so state. Alternatively, if feasibility of SOFA in the ED was the concern that would also be valid, but that is not stated either. Given that a sensitivity analysis using SOFA was performed, feasibility would seem to not be a major issue.

Authors reply: Data was prospectively gathered for vital signs and laboratory tests at arrival at the EDs and worst vital signs and laboratory results within 72 hours for cohort A. This is why we were able to perform a sensitivity analysis comparing the combined outcome with a presumed rise in SOFA score of 2 or more at arrival. We could however not calculate SOFA after arrival and found it important to evaluate the scores for prediction of sepsis and not only screening for sepsis at arrival, why we had to apply the sepsis-2 definition.

Reviewers comment 3. Finally, throughout the article the endpoint of SOFA > 2 is used to describe the Sepsis 3 definition, however this is not accurate. The actual sepsis 3 definition is a rise in SOFA >= 2 from baseline. This helps to prevent patients with chronic disease (e.g. chronic kidney disease, cirrhosis) from being automatically classified as "septic" as soon as they hit the door (i.e. without any actual acute worsening of their end-organ function). As an example, if the definition of SOFA>=2 was used (instead of increase in SOFA >=2) it would mean every patient with stage IV-V CKD with an infection would automatically be labeled as septic in the sample even if they had no acute organ dysfunction. It is unclear if this was considered by the authors, but is eminently important since the rates of chronic renal disease were significantly higher in the septic (by combined outcome) patients in cohort A, and rates or liver disease were higher in cohort B. It may be difficult to accurately assess baseline values for SOFA with a retrospective design (i.e. to adjudicate rise vs. baseline points on SOFA) but even if so this needs to be addressed as a limitation.

Authors reply: We agree, this was taken into account, but was poorly communicated in the manuscript. The text has been changed to a presumed rise in SOFA score of two or more, p 9, line 170, p16, line 321, p 18, line 372.

Additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

The naming has been changed, p 5, line 103, p 8, line 163, p 9, line 177, line 180, p 15, line 285, p 16, line 310, 318, p 17, line 330, p 18, line 365, 375, p 18, line 381 and supporting information files.

2. Thank you for including your ethics statement:

'the regional ethical board in Lund (approval number 2010/205 and 2014/4) the regional ethical board in Bern (approval number KEK 315/14) the regional ethical board in Vancouver (approval number H11-00505).'

a. Please amend your current ethics statement to confirm that your named institutional review board or ethics committee specifically approved this study.

b. Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

3. In the ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records used in your retrospective study.

Specifically, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent.

If patients provided informed written consent to have data from their medical records used in research, please include this information.

Authors reply to additional requirements 2 & 3: All included patients gave written informed consent in cohort A. The study is in accordance with the approval and the informed consents. The study is also in accordance with the approvals and informed consents for cohort B. In cohort B, included patients all gave written informed consent or, if unable to give informed consent, next-of-kin was asked for permission. For patients that died without being able to leave informed consent, the use of data and samples was requested at the local ethics committee. The Ethics Statement and the manuscript has been changed, p 7, line 133-135 p 8, line 152-155.

4. We note that you have a patent relating to material pertinent to this article. Please provide an amended statement of Competing Interests to declare this patent (with details including name and number), along with any other relevant declarations relating to employment, consultancy, patents, products in development or modified products etc.

Authors reply: Bertil Christensson, Per.Åkesson, and Adam Linder are listed as inventors on a patent on the use of HBP as a diagnostic tool in sepsis filed by Hansa Medical AB WO2008151808A1. This does not alter our adherence to PLOS ONE policies on sharing data and materials. All other authors have declared no relevant conflicts of interest

5. If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

Authors reply: The data in the study is based on patient material and since we still have a code key it is under the GDPR only considered to be pseudo-anonymized and not de-identified. Furthermore, since there are many individual variables and sensitive patient information there is a possibility that patients might be identified due to their comorbidities, patient characteristics and time of encounter. Hence, we were not granted an ethical permit to publish individual data but merely publishing data on group level. This is also clearly stated in the information to patients which the participants have signed. We consider uploading the full data set as publishing the data and due to the ethical restrictions imposed on us this is unfortunately not possible to do. In order to assure compliance with the information given to the patients, data can be shared upon request from ethics committee registrator@etikprovning.se, 0046104750800 (Swedish Ethical Review Authority) and from Cantonal Ethics Committee Zurich info.kek@kek.zh.ch.

6. Please upload a new copy of your Supporting Information Figure as the detail is not clear. Please follow the link for more information: http://blogs.PLOS.org/everyone/2011/05/10/how-to-check-your-manuscript-image-quality-in-editorial-manager/

Authors reply: A new copy of the Supporting Information Figure has been uploaded

7. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Authors reply: Captions for Supporting information files has been included p 23, line 464-475.

We feel that this revised manuscript is much improved compared to the previous version and hope that you will find it suitable for publication.

Again, thank you for your interest and engagement in our work.

Hoping for a positive response,

Sincerely,

Lisa Mellhammar

Submitted filename: Respons to Reviewers.docx

Click here for additional data file.

13 Jan 2020

PONE-D-19-32440R1

Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS

PLOS ONE

Dear Dr Mellhammar,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

The manuscript is markedly improved overall, but there remain some concerns about the multiple imputation models. While there may not be a strictly "correct" answer, addition of further discussion about why data was assumed to be MAR rather than MNAR and potential limitations and/or ramifications of this decision would strengthen the paper. Can you provide a reference for the sentence in bold below? If so, this would be very nice addition to the paper.

The MAR assumption is not testable, but becomes more reasonable with imputation models that include a wide range of characteristics, including predictors, the outcome and auxiliary variables like in our model.

This additional text could be included in discussion of the limitations.

==============================

We would appreciate receiving your revised manuscript by Feb 27 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: my previous comments were adequately addressed, WELL DONE job.

The rebuttal letter is good and my comments were well addressed.

Reviewer #2: I would like to see a little more addressing the imputation, specifically the R code and more detailed demographic and outcomes data for those patients with missing data, preferably stratified by the type of data element missing. Also, a "worst-case" sensitivity analysis specifically involving lactate (since this was missing so frequently, and directly relates to a major aim), may be worth performing.

I am concerned that little in the edits seem to have substantively addressed the comments regarding imputation. The change added a sentence saying that it was assumed that data was missing at random. I found this unsatisfactory, since a large portion of reviewer and editor feedback raised was very specifically directed at needing to account for the possibility that data was missing not at random.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Zhongheng Zhang

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

29 Jan 2020

Dear Dr Ehrman,

Thank you for your letter, the comments from you and the reviewers and the opportunity to revise the manuscript entitled “Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS”.

The manuscript has now been revised according to the suggestions by you and reviewer 2. Our response to the specific points is given below:

Editors comment:

While there may not be a strictly "correct" answer, addition of further discussion about why data was assumed to be MAR rather than MNAR and potential limitations and/or ramifications of this decision would strengthen the paper. Can you provide a reference for the sentence in bold below? If so, this would be very nice addition to the paper.

“The MAR assumption is not testable, but becomes more reasonable with imputation models that include a wide range of characteristics, including predictors, the outcome and auxiliary variables like in our model”.

This additional text could be included in discussion of the limitations.

Authors reply:

Discussion about limitations of multiple imputation has been added to the discussion, including the text and a reference for the statement that the MAR assumption is not testable, but becomes more reasonable within a model like ours that include several characteristics, including predictors and outcome, p 21, line 428-436 [1].

It is also elaborated in the reply to reviewer 2 below.

Reviewer 2

Reviewers comment 1: I would like to see a little more addressing the imputation, specifically the R code and more detailed demographic and outcomes data for those patients with missing data, preferably stratified by the type of data element missing. Also, a "worst-case" sensitivity analysis specifically involving lactate (since this was missing so frequently, and directly relates to a major aim), may be worth performing.

I am concerned that little in the edits seem to have substantively addressed the comments regarding imputation. The change added a sentence saying that it was assumed that data was missing at random. I found this unsatisfactory, since a large portion of reviewer and editor feedback raised was very specifically directed at needing to account for the possibility that data was missing not at random.

Authors reply:

We agree and have made an effort to address the comments on multiple imputation.

First, we have tried to clarify that even though the aims of this study were to develop a sepsis risk stratification tool based on the most predictive, minimal set of vital signs, lactate and HBP plasma levels, this was performed in cohort A, where few values for lactate were missing.

In cohort B the new scores and RETTS were validated and lactate is not included in these scores, supporting information table IV.

To calculate worst case scenario for missing data is an approach for missing data which is a strong assumption that can give the sensitivity analyses a wide range, even with moderate number of missing outcomes.

As mentioned previously, we assumed data to be missing at random, which is not testable, but becomes more reasonable within a model like ours that include several characteristics, including predictors and the outcome.

The cause for that a vital sign or a laboratory value is missing is often that they were considered unnecessary according to the attending physician. The decision was then evidently made based on other available vital parameters and laboratory values and hence the probability of missing depends on other parameters present in the data set; i.e missing at random (Pr (R=0|Yobs, Ymis, �) = Pr(R=0|Yobs,�) [2].

Multiple imputation is therefore commonly used when evaluating clinical risk scores [3-5].

We have elaborated the discussion about limitations of multiple imputation, 21, line 428-436 and added a table on demographics and outcome of patients with missing data to the supporting information (supporting information table V).

The multiple imputation was performed in SPSS with the code below.

*set the random seed

SET MTINDEX=2000000.

*Analyze Patterns of Missing Values.

MULTIPLE IMPUTATION HR SBP DBP mental_status temperature SaO2

RR oxygen_treatment hbp lactate age

/IMPUTE METHOD=NONE

/MISSINGSUMMARIES OVERALL VARIABLES (MAXVARS=25 MINPCTMISSING=0) PATTERNS.

*Impute Missing Data Values.

DATASET DECLARE newimputeddata.

DATASET DECLARE iterationhistory.

MULTIPLE IMPUTATION HR SBP DBP oxygen_treatment RR

temperature mental_status SaO2 outcome hbp lactate

age sex comorbidities_1 comorbidities_2

comorbidities_3 comorbidities_4 comorbidities_5 comorbidities_6

comorbidities_7

/IMPUTE METHOD=FCS MAXITER= 100 NIMPUTATIONS=20 SCALEMODEL=PMM INTERACTIONS=NONE SINGULAR=1E-012

MAXPCTMISSING=NONE

/CONSTRAINTS age( ROLE=IND)

/CONSTRAINTS hbp ( ROLE=IND)

/CONSTRAINTS sex ( ROLE=IND)

/CONSTRAINTS outcome ( ROLE=IND)

/CONSTRAINTS comorbidities_1( ROLE=IND)

/CONSTRAINTS comorbidities _2( ROLE=IND)

/CONSTRAINTS comorbidities _3( ROLE=IND)

/CONSTRAINTS comorbidities _4( ROLE=IND)

/CONSTRAINTS comorbidities _5( ROLE=IND)

/CONSTRAINTS comorbidities _6( ROLE=IND)

/CONSTRAINTS comorbidities _7( ROLE=IND)

/MISSINGSUMMARIES NONE

/IMPUTATIONSUMMARIES MODELS

/OUTFILE IMPUTATIONS=newimputeddata FCSITERATIONS=iterationhistory.

We feel that the addition of discussion on multiple imputation assumption has improved the manuscript compared to the previous version and hope that you will find it suitable for publication.

Again, thank you for your interest and engagement in our work.

Hoping for a positive response,

Sincerely,

Lisa Mellhammar

References

1. Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ (Clinical research ed.) 2009;338:b2393 doi: 10.1136/bmj.b2393[published Online First: Epub Date]|.

2. Buren SV. Flexible Imputation of Missing Data. CRC Press 2018

3. Seymour CW, Kahn JM, Cooke CR, Watkins TR, Heckbert SR, Rea TD. Prediction of critical illness during out-of-hospital emergency care. Jama 2010;304(7):747-54 doi: 10.1001/jama.2010.1140[published Online First: Epub Date]|.

4. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). Jama 2016;315(8):762-74 doi: 10.1001/jama.2016.0288[published Online First: Epub Date]|.

5. Sunden-Cullberg J, Rylance R, Svefors J, Norrby-Teglund A, Bjork J, Inghammar M. Fever in the Emergency Department Predicts Survival of Patients With Severe Sepsis and Septic Shock Admitted to the ICU. Crit Care Med 2017;45(4):591-99 doi: 10.1097/ccm.0000000000002249[published Online First: Epub Date]|.

Submitted filename: Respons to Reviewers.docx

Click here for additional data file.

3 Feb 2020

Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS

PONE-D-19-32440R2

Dear Dr. Mellhammar,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Robert Ehrman, MD, MS

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

5 Feb 2020

PONE-D-19-32440R2

Scores for sepsis detection and risk stratification – construction of a novel score using a statistical approach and validation of RETTS

Dear Dr. Mellhammar:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Robert Ehrman

Academic Editor

PLOS ONE