Combining procalcitonin with the qSOFA and sepsis mortality prediction.

To investigate whether procalcitonin (PCT) can improve the performance of quick sequential organ failure assessment (SOFA) score in predicting sepsis mortality, we conducted a retrospective multicenter cohort study with independent validation in a prospectively collected cohort in 3 tertiary medical centers. Patients with presumed sepsis were included. Serum PCT levels were measured at admission. Quick SOFA score and systemic inflammatory response syndrome (SIRS) criteria were calculated for each patient. PCT levels were assigned into 0, 1, and 2 points for a serum level of <0.25, 0.25 to 2, and >2 ng/mL, and added to the quick sepsis-related organ failure assessment (qSOFA) score. The incremental value of PCT to qSOFA was then evaluated by logistic regression, receiver-operating characteristic (ROC) curve, and reclassification analysis.In all, 1318 patients with presumed severe infection were enrolled with a 30-day mortality of 13.5%. Serum level of PCT showed a high correlation with qSOFA score and 30-day inhospital mortality. The area under the ROC curve was 0.56 for SIRS criteria, 0.67 for qSOFA score, and 0.73 for qSOFA_PCT in predicting 30-day mortality. The risk prediction improvement was reflected by a net reclassification improvement of 35% (17%-52%). Incorporation of PCT into the qSOFA model could raise the sensitivity to 86.5% (95% confidence interval 80.6%-91.2%). In the validation cohort, qSOFA_PCT greatly improved the sensitivity to 90.9%.A simple modification of qSOFA score by adding the ordinal scale of PCT value to qSOFA could greatly improve the suboptimal sensitivity problem of qSOFA and may serve as a quick screening tool for early identification of sepsis.

Introduction

Sepsis is a leading cause of mortality and morbidity globally.[ Early diagnosis of sepsis and early initiation of evidence-based bundle care can greatly improve the outcome of sepsis. Unfortunately, early and accurate diagnosis of sepsis is difficult. Sepsis is a complex clinical syndrome with a wide range of manifestations. Although the systemic inflammatory response syndrome (SIRS) criteria were part of the prior definition of sepsis, it has been shown to be unable to differentiate severe from uncomplicated infections. The latest Sepsis-3 definition describes sepsis as a life-threatening organ dysfunction caused by a dysregulated host response to infection,[ with life-threatening organ dysfunction defined as a change in the sequential organ failure assessment (SOFA) score of more than 2 points in intensive care unit (ICU) patients.[ In clinical settings outside the ICU where calculating the SOFA score is not routine, a simplified score—quick sepsis-related organ failure assessment (qSOFA)—was introduced as a screening tool for patients with sepsis.[

Since the introduction of qSOFA, concerns have been raised. None of the elements in qSOFA are specific for the detection of infection, and subsequent validation studies showed suboptimal overall discrimination and sensitivity (reported sensitivity 32%) under the recommended cut-off.[ In addition, diagnosing sepsis relies on the accuracy of the physician's clinical suspicion of infection.[ Infectious disease biomarkers such as C-reactive protein (CRP) and procalcitonin (PCT), on the contrary, have been shown to accurately predict infection and mortality.[ These 2 markers were accepted as part of the diagnostic criteria in Sepsis-2, but were not included in the Sepsis-3 definition.[ Thus, combining biomarker information with the qSOFA score would potentially enhance its ability to predict the mortality risk from sepsis. In this study, we sought to evaluate whether adding either CRP or PCT to the qSOFA score would improve its ability to predict inhospital mortality in a multicenter cohort of patients who presented with clinical symptoms of systemic infection.

Methods

Study design and locations

This was a multicenter retrospective cohort study performed at the Sichuan Provincial People Hospital (SCPH) in Chengdu City, First People's Hospital of Foshan (FSFPH) in Guangdong province of China, and National Taiwan University Hospital Yunlin Branch (NTUH) in Douliou city. All 3 hospitals are tertiary-care urban medical centers.

Study population

Patients were enrolled retrospectively using consecutive sampling of cases from each hospital from January 1, 2015 to December 31, 2016. All adult patients (≥18 years old) who presented to the emergency department or were admitted to the hospital floor were eligible for inclusion. Patients were included if they had symptoms that indicated systemic infection; and PCT (VIDAS BRAHMS PCT) and blood culture tests within 24 hours of admission. Exclusion criteria were missing data, transfer from other hospitals, leucopenia, do-not-resuscitate (DNR) orders, lost to follow-up, or history of pre-existing thyroid disease that may affect procalcitonin levels. Participating investigators from the 3 sites independently reviewed all retrieved medical records to confirm the presence of infection as a reason for admission. The study focused on qSOFA score, which targeted the sepsis patients outside the ICU. Therefore, we excluded patients who developed sepsis in ICU. A cohort that prospectively collected 493 consecutive emergency department (ED) patients with presumed sepsis was used for independent validation. Patients with presumed sepsis was defined patients who fulfilled at least 2 of the 3 SIRS criteria (temperature >38°C or <36°C, pulse rate ≥90 beats per minute, and respiratory rate ≥20/min) upon ED admission with a presumed diagnosis of systemic infection by treating physicians. This study was approved by the Research Committees and Institutional Review Boards for all institutions, and it met criteria for exemption from informed consent.

Data collection

The hospitalization course of all patients was followed from admission to discharge. Using a standardized data collection instrument, data were collected on patients (Table 1). The worst physiological and laboratory measurements within the initial 24 h of admission were noted. The endpoint, all-cause inhospital mortality, was determined by systematically reviewing hospital records resulting from each patient's index hospitalization. SIRS criteria variables include abnormal body temperature, tachycardia, tachypnea, and abnormal white blood cell count. The criteria for qSOFA include altered mental status, hypotension, or tachypnea. In addition, we defined septic shock as a systolic blood pressure <100 mm Hg requiring vasopressor therapy, and respiratory difficulty as a respiratory rate >22 breaths/min, an oxygen saturation <90%, or need for supplemental oxygen by either face mask or 100% nonrebreather.

Table 1

Characteristics of the derivation cohort from 3 participating hospitals from 2015 to 2016.

Measurement of PCT and CRP

Blood samples were collected within 24 hours of admission. PCT concentrations were measured using an immunoluminometric assay with a detection limit of 0.06 ng/mL (VIDAS PCT; bioMerieux, Marcy, France). CRP was measured using an immunonephelometric assay (Olympus, Osaka, Japan) with a detection limit of 0.2 mg/L.

Statistical analysis

Chi-square tests for categorical variables and Mann–Whitney U tests for continuous variables were performed for univariate comparison. The correlation between serum levels of biomarkers and clinical severity (defined by SIRS or qSOFA score) was examined with box plots. To evaluate whether PCT or CRP has incremental prognostic value independent of qSOFA variables, we employed a logistic regression model. The values of CRP and PCT were entered into the model as predefined ordinal scales to ease clinical application. We categorized the biomarker values into 3 ordinal classes (0, 1, and 2): 0 was assigned for PCT levels <0.25 and CRP <60, 1 for PCT levels from 0.25 to 2 and CRP from 60 to 120, and 3 for PCT >2 and CRP >120. We then compared the predictive values of SIRS, qSOFA, qSOFA plus ordinal CRP class (qSOFA_CRP), and qSOFA plus ordinal PCT class (qSOFA_PCT). The predictive value of each model was first calculated by the c-statistic, or area under the receiver-operating characteristic (ROC) curve (AUC), for which larger values indicate better discrimination. We compared the predicted probabilities and observed risk in each risk category.[

Next, we divided all patients into 4 predicted inhospital mortality risk groups: very low risk (0% to less than 5%), low risk (5% to less than 15%), high risk (15% to less than 30%), and very high risk (30% or greater). We then assessed whether the biomarker enhanced qSOFA models significantly reclassified patients into more appropriate mortality risk categories in comparison with the qSOFA model. The overall reclassification improvement was evaluated by the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI). The NRI was calculated by summing the proportion of participants across risk categories whose estimated risk shifts in the correct direction minus the proportion of participants whose risk shifts in the incorrect direction.[ Therefore, NRI was used as a measure to estimate any overall improvement in reclassification with the new model. Rather than using discrete risk categories, the IDI calculates the difference in discrimination slopes between the 2 models. Consequently, the IDI index demonstrates the improvement in both discrimination and reclassification.

Lastly, we calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of SIRS, qSOFA, and qSOFA_PCT in predicting inhospital mortality, and we calibrated each model to mortality rate with bar graphs. We validate the accuracy of qSOFA and qSOFA_PCT in an independent historical sample by calculating the sensitivity, specificity, PPV, and NPV. All analyses were performed with SAS Version 9.4 (Cary, NC), except for the NRI and IDI statistics, which were computed with R Statistical Software (Foundation for Statistical Computing, Vienna, Austria). A 2-sided P value <.05 was viewed as significant.

Ethical approval

This study was approved by the institutional review board of National Taiwan University Hospital.

Results

Patient characteristics and outcome

During the study period, 604 patients from NTUH, 503 from SCPH, and 515from FSFPH fulfilled the inclusion criteria. After the exclusion of 304 ICU patients, the final cohort included a total of 1318 patients (Supplementary eFig. 1). In all, 867 (65.8%), 208 (15.8%), and 752 (53.1%) patients were diagnosed with sepsis according to the SIRS criteria and qSOFA score, respectively. In all, 205 (15.6%) patients had clinical significant bacteremia, with a higher prevalence of gram-negative bacteremia (7.1%) than gram-positive bacteremia (6.1%). The median age of the study sample was 64.0 (interquartile range 47–75) years old. The overall inhospital mortality rate was 13.5%. The mortality of patients with clinical significant bacteremia was 19.5%.

Table 1 summarizes the baseline characteristic of patients. In general, nonsurvivors were older, had a higher burden of comorbidities and organ dysfunction, had more lower respiratory tract infections, had more bloodstream infection and polymicrobial infection, and had a greater proportion of patients with PCT, SIRS, or qSOFA scores higher than reference levels.

Comparison of correlation of laboratory markers and qSOFA

We examined the correlation between 3 laboratory markers (white blood cell count, CRP, and PCT) and the qSOFA score. The scatter plots of the 3 markers stratified by the 4 qSOFA classes are shown in Fig. 1. PCT has the highest correlation with qSOFA class, followed by CRP and white blood cell (WBC) count.

Figure 1

Serum levels of white blood cell count, CRP, and PCT in patients with different qSOFA severity classifications. CRP = C-reactive protein, PCT = procalcitonin, qSOFA = quick sepsis-related organ failure assessment.

Comparison of model performance-discrimination, model fit, and reclassification

Table 2 also presents comparisons of discrimination, overall fit, and reclassification statistics for 4 different models. For discrimination, defined as the AUC, qSOFA_PCT model had the largest AUC, followed by qSOFA_CRP, qSOFA, and SIRS. For overall fit, using Bayes information criteria, a lower score indicates better goodness of model fit with a parsimonious model. qSOFA_PCT had the best performance, followed by qSOFA, qSOFA + CRP, and SIRS. For reclassification, we made 2 stage comparisons. First, when comparing qSOFA to SIRS, both IDI and NRI showed qSOFA reclassified patients to a more appropriate mortality risk category than SIRS. Comparing qSOFA_CRP or qSOFA_PCT with qSOFA, we found that qSOFA_PCT could better reclassify patients as indicated by significantly higher NRI and IDI. qSOFA_CRP, however, failed to demonstrate a higher IDI than qSOFA, although the combined score still had better NRI than qSOFA alone. Using the qSOFA_PCT score, in all, 752 (53.1%) patients could be diagnosed with sepsis, in contrast to 208 (15.8%) patients by qSOFA score alone.

Table 2

The discriminating capability of different laboratory markers in predicting sepsis mortality, presented as the area under curve.

Model calibration

To visually compare model performance, we constructed bar graphs in Fig. 2 to present the mortality rate in each severity class for the 3 models (SIRS, qSOFA, and qSOFA_PCT). qSOFA_PCT showed the best calibration, followed by qSOFA. The mortality rate did not correlate with the SIRS classification. We further calculated the sensitivity, specificity, PPV, and NPV for SIRS, qSOFA, and qSOFA_PCT for inhospital mortality prediction (Table 3). We used a cut-off score of ≥2 for each criterion. Despite the poor performance in model fit and calibration, SIRS had the best sensitivity at the cost of the lowest specificity. qSOFA had the highest specificity, but lowest sensitivity. qSOFA_PCT ≥2 greatly enhances the sensitivity of qSOFA to 86.5% with a compromised specificity of 47.5%.

Figure 2

Observed mortality rate in SIRS, qSOFA, and PCT enhanced qSOFA severity categories. PCT = procalcitonin, qSOFA = quick sepsis-related organ failure assessment, SIRS = systemic inflammatory response syndrome.

Table 3

Comparison of the sensitivity and specificity of SIRS, qSOFA, and qSOFA_PCT in predicting inhospital mortality.

Validation

In the validation cohort, we confirmed qSOFA had a suboptimal sensitivity (39.1%) and high specificity (94.9%) in predicting 30-day mortality. Incorporation of PCT into qSOFA could improve the sensitivity to 90.9% at the cost of low specificity (50.3%).

Discussion

In our retrospective validation of the Sepsis-3 definitions using multicenter cohorts, we confirmed that the qSOFA score is superior in mortality prediction compared with SIRS in terms of discrimination, model fit, reclassification, and calibration statistics. We also demonstrated that PCT has better correlation with clinical severity than WBC count or CRP. Furthermore, we found that combining PCT and the qSOFA score by simply adding the ordinal scale of PCT to the qSOFA score can significantly enhance its mortality prediction capability in all dimensions of model performance indicators. Clinically, PCT enhanced qSOFA, or qSOFA_PCT, has the best sensitivity (86.5%) and can be served as a screening tool to quickly identify patients with sepsis who may benefit from early intervention. qSOFA alone has the best specificity (87.0%) and can subsequently serve as a quick confirmation tool to aid in the decision to pursue more invasive treatment.

The original Sepsis-3 definitions proposed using the simple qSOFA score as the initial screening tool, followed by the comprehensive SOFA score as the confirmation tool for sepsis. In the original work, the sensitivity of a qSOFA score ≥2 was reported to be low at 55%, albeit with a high specificity (84%), whereas a change of SOFA score of ≥2 had a higher sensitivity (68%), but a lower specificity (67%). The sensitivity and specificity profiles of qSOFA and SOFA, however, are contradictory to their proposed clinical use. A screening tool requires high sensitivity, whereas a confirmation tool requires high specificity. Our study, like other external validation studies, confirmed the low sensitivity and high specificity of the qSOFA score.[

Thus, we propose incorporating PCT levels into qSOFA to correct for its low sensitivity. The high sensitivity (86.5%) and high NPV (95.8%) of qSOFA_PCT ≥2 justify the combined score as an initial screening tool. Nonetheless, we would like to preserve the qSOFA as a confirmation tool for 2 reasons. First, it is validated as a simple but highly specific tool with a specificity (84%) higher than the proposed delta SOFA greater than 2 (specificity 67%). Second, in clinical settings outside the ICU such as the ED, the comprehensive SOFA score may not be easily obtained. In summary, we believe the newly proposed algorithm, using high-sensitivity qSOFA_PCT as a screening tool and the high-specificity qSOFA score as the confirmation tool, is the most optimal use of biomarker information and the best clinical decision rule in clinical settings outside of the ICU.

In addition to the contradiction between the sensitivity and specificity profiles of qSOFA and SOFA, another major concern regarding replacing SIRS with qSOFA is the absence of clinical indicators of infection in the qSOFA score.[ PCT has been shown to be a reasonably sensitive marker in differentiating sepsis from sterile SIRS. In a large meta-analysis with 3244 patients, PCT alone had a sensitivity of 77% and a specificity of 79% in the diagnosis of sepsis.[ The accuracy of PCT has been shown to be valid for various sites of infection and for different populations such as the elderly, patients with renal impairment, and patients with autoimmune disease.[ Therefore, the addition of PCT to the qSOFA score may complement its lack of infection indicators. However, the accuracy of PCT may be compromised in patients with neutropenia, hyperfunctioning thyroid cancer, and in patients with certain subacute infectious diseases such as infective endocarditis.[ The qSOFA_PCT should be used cautiously in these specific populations.

In clinical settings where testing for PCT is not readily available, the SIRS criteria would retain its value as a simple and low-cost screening tool. In multiple validation studies, SIRS ≥2 consistently demonstrates a higher sensitivity than qSOFA. We thus propose that when PCT cannot be obtained, SIRS should remain as a screening tool with qSOFA as the confirmation tool.

Although we showed that the AUC of qSOFA_PCT is higher than qSOFA alone, the confidence intervals overlapped. The limitations of the c-statistic, or AUC, as a measure of clinical model performance, have been discussed extensively in the literature. AUC is based exclusively on ranks, and it only measures how well the predicted values can rank order the responses.[ It may not be as sensitive as the likelihood function in choosing between models, and it is less clinically relevant as a calibration measure that directly cross-classifies the predicted risk categories with the observed risk. The magnitude of IDI, defined as the difference in discrimination slopes, has a direct interpretation.[ In our study, qSOFA_PCT, compared with qSOFA, had an IDI of 0.0097. This indicates that the new model increases the mean difference of predicted probabilities for death and nondeath by 0.97%. In the reclassification analysis, NRI is calculated as the sum of the net percentages of correctly reclassified patients with and without the event of interest. NRI is the favored metric when assessing the true discriminatory potential of a new predictor compared with other predictors. It captures the incremental strength of the new predictor after accounting for correlations with variables included in the baseline model. NRI values above 0.6 are considered strong, 0.4 intermediate, and below 0.2 weak. In our case, the NRI comparing qSOFA_PCT with qSOFA is 0.35, which suggests a medium number of patients were reclassified to more appropriate risk categories. This change was found to be significant (P = .00011).

Since its publication of Sepsis-3 criteria and qSOFA in 2016, numerous attempts have been made to assess the performance of qSOFA. A recent meta-analysis collecting 45 studies showed low sensitivity and good specificity when the qSOFA is used as a screening tool for sepsis (pooled sensitivity was 61%, and specificity was 72%).[ In contrast, SIRS criteria resulted in a pooled sensitivity of 88%, but with only 26% specificity. Our study demonstrates that combing PCT and qSOFA, rather than PCT and SIRS, can achieve the best sensitivity. The simple qSOFA_PCT score will help clinicians identify at-risk patients and those with high likelihood for deterioration. A prospective external validation of this simple score is needed to verify the generalizability of this modified score.

Our work has both strengths and limitations. We were the first in the literature to prove the added prognostic value of PCT to the qSOFA score by a rigorous statistical analysis. We further proposed a practical strategy for clinical use: the highly sensitive qSOFA_PCT score used as a screening tool, followed by the highly specific qSOFA score used as the confirmation tool. In addition, this multicenter design has a relatively large population, which increases the generalizability of our work. A key limitation of our study was the retrospective nature of this work. However, the patient characteristics of our study cohort are comparable with prospectively collected continuous samples. In addition, as these sepsis patients were not enrolled in ICU, common severity scores such as SAPS, APACHE II, or SOFA score were not available.

Conclusions

Our work confirmed that qSOFA has low sensitivity and high specificity in predicting sepsis mortality. The incorporation of the ordinal scale of PCT to the qSOFA model could enhance sensitivity and reclassify patients into risk groups that better reflect their actual short-term mortality risk. We propose using qSOFA_PCT as a screening tool followed by qSOFA alone as a confirmation tool for the identification of patients with sepsis in settings outside of the ICU.

Acknowledgments

We thank the staff of the Core Labs, the Department of Medical Research, and National Taiwan University Hospital for technical support, and Medical Wisdom Consulting Group for technical assistance in statistical analysis.

Author contributions

C.-C.L. designed the study, obtained funding, drafted the analytical plan, guided the statistical analysis, interpreted the data, and wrote the draft. H.Y., L.N., A.L., and K.L. helped to collect data in this study. D.Y. and M.-T.L. helped with language editing and provided critical comments. T.-C.H. conducted statistical analysis. Y.-C.H. analyzed the data, provided critical feedback, and authorized the final manuscript.

Data curation: Lu Nie.

Funding acquisition: Yenh-Chen Hsein.

Investigation: Aibo Liu.

Methodology: Chien-Chang Lee.

Project administration: Chien-Chang Lee.

Resources: Kuihai Wu.

Supervision: Yenh-Chen Hsein, Chien-Chang Lee.

Writing – original draft: Hua Yu, Chien-Chang Lee.

Writing – review & editing: Debra W Yen, Meng-tse G. Lee.