Validation of the mortality in emergency department sepsis (MEDS) score in a Singaporean cohort.

The emergency department (ED) serves as the first point of hospital contact for most septic patients. Early mortality risk stratification using a quick and accurate triage tool would have great value in guiding management. The mortality in emergency department sepsis (MEDS) score was developed to risk stratify patients presenting to the ED with suspected sepsis, and its performance in the literature has been promising. We report in this study the first utilization of the MEDS score in a Singaporean cohort.In this retrospective observational cohort study, adult patients presenting to the ED with suspected sepsis and fulfilling systemic inflammatory response syndrome (SIRS) criteria were recruited. Primary outcome was 30-day in-hospital mortality (IHM) and secondary outcome was 72-hour mortality. MEDS, acute physiology and chronic health evaluation II (APACHE II), and sequential organ failure assessment (SOFA) scores were compared for prediction of primary and secondary outcomes. Receiver operating characteristic (ROC) analysis was conducted to compare predictive performance.Of the 249 patients included in the study, 46 patients (18.5%) met 30-day IHM. MEDS score achieved an area under the ROC curve (AUC) of 0.87 (95% confidence interval [CI], 0.82-0.93), outperforming the APACHE II score (0.77, 95% CI 0.69-0.85) and SOFA score (0.78, 95% CI 0.71-0.85). On secondary analysis, MEDS score was superior to both APACHE II and SOFA scores in predicting 72-hour mortality, with AUC of 0.88 (95% CI 0.82-0.95), 0.81 (95% CI 0.72-0.89), and 0.79 (95% CI 0.71-0.87), respectively. In predicting 30-day IHM, MEDS score ≥12, APACHE II score ≥23, and SOFA score ≥5 performed at sensitivities of 76.1%, 67.4%, and 76.1%, and specificities of 83.3%, 73.9%, and 65.0%, respectively.The MEDS score performed well in its ability for mortality risk stratification in a Singaporean ED cohort.

Introduction

Sepsis is a severe and potentially life-threatening condition that represents a major cause of morbidity and mortality in both adult and pediatric populations. In 2017, at least 22.0% of all deaths in Singapore were attributed to sepsis from pneumonia and urinary tract infections.[ When sepsis does not result in mortality, complications, such as multi-organ dysfunction syndrome (MODS) are still a major cause of patient morbidity and healthcare burden. As the emergency department (ED) serves as the first point of hospital contact for many septic patients, early mortality risk stratification using a quick and accurate triage tool would have great value in guiding management.

Disease severity scoring systems such as the sequential organ failure assessment (SOFA) score[ and the acute physiology and chronic health evaluation II (APACHE II) score[ have been developed to assess severity of illness and stratify patients based on mortality risk. These systems were developed for use in the intensive care unit (ICU), and a major limitation hindering their practical application in the ED is that they require information that is often not readily available during a patient's time at the ED. Other quick scoring systems, such as the quick SOFA (qSOFA) score,[ National early warning score (NEWS), and modified early warning score (MEWS)[ have shown limited prognostic ability for septic patients.[ There is thus a need for a reliable scoring system intended for use in the ED, to better risk stratify septic patients, and to guide triage and management decisions.

The mortality in emergency department sepsis (MEDS) score[ developed in 2003, was derived and validated in patients presenting to the ED with suspected sepsis, and aims to risk stratify patients based on their risk for 28-day in-hospital mortality (IHM). It includes basic demographic, clinical, and laboratory variables that can be attained during a patient's time in the ED, which are weighted to give a final score and mortality risk assessment. In subsequent external validations and usage of the MEDS score in various populations, its predictive ability has been found to be promising. The MEDS score has achieved area under the receiver operating characteristic curve (AUC) values between 0.78 to 0.92 in its ability to predict IHM at various time points (majority being IHM at 1 month), outperforming clinical biomarkers, such as C-reactive protein, lactate, procalcitonin,[ and the other disease severity scoring systems such as the SOFA and APACHE II scores.[

We thus endeavor to conduct the first study validating the MEDS score in a Singaporean cohort, and to compare its predictive ability for mortality outcomes against the APACHE II and SOFA scores. Should the MEDS score perform well, its utility as an accurate and practical scoring tool in the ED is manifold. With regard to the individual patient, the MEDS score can inform decisions, intended aggressiveness of management, and patient candidacy for invasive monitoring and early goal directed therapy.[ In a research and administrative setting, the MEDS score can be utilized to perform risk stratified analysis, or for resource management.

Methods

Study design and setting

We conducted a retrospective data analysis on a convenience sample of patients presenting to the ED of Singapore General Hospital (SGH) between September 2014 to April 2016. The study was approved by SingHealth Centralised Institutional Review Board with waiver of patient consent.

SGH is a tertiary care hospital in Singapore, whose ED sees between 300 and 500 patients a day. Upon presentation at the ED, patients are triaged with the national Singaporean Patient Acuity Category Scale (PACS), a symptom-based triage system without strict physiological criteria. Patients are assigned a PACS score ranging from 1 to 4, reflecting urgency in need for physician consultation. Patients triaged to PACS 1 are critically ill, those to PACS 2 are non-ambulant, those to PACS 3 are ambulant, and those to PACS 4 are non-emergencies.

Study population and eligibility

Adult patients (aged 18 years and above) presenting to the ED with clinically suspected sepsis and fulfilling at least 2 of the 4 systemic inflammatory response syndrome (SIRS) criteria[ were recruited. The 4 SIRS criteria were temperature of more than 38°C or less than 36°C, elevated heart rate of more than 90 beats per minute, respiratory rate above 20 breaths per minute, and total white blood cell count of more than 12,000 cells per mm3 or less than 4000 cells per mm3. The study included patients triaged to PACS 1 or 2, and excluded patients who were lost to follow up or transferred to other hospitals within 30 days of initial ED presentation.

Data collection

Patient demographic data, comorbidities, clinical and laboratory parameters, and treatment received for the duration of the patient's ED stay were retrieved from the electronic medical records. The primary outcome was 30-day IHM following ED presentation, and secondary outcome was 72-hour mortality. Patient outcomes were determined based on data obtained from our comprehensive electronic data warehouse.

Parameters required for the generation of MEDS, APACHE II, and SOFA scores are listed in the Appendix. The MEDS score included 9 weighted variables, which were counted as either present or absent for the scoring of points. Number of points accumulated correlates with increasing risk of sepsis mortality. The MEDS score variables were the presence of terminal illness (6 points), age more or equal to 65 years (3 points), respiratory distress (3 points), septic shock (3 points), percentage bands on differential count more or equal to 5% (3 points), platelet count less or equal to 150,000 cells per mm3 (3 points), altered mental status (AMS) (2 points), physician suspicion of lower respiratory tract infection (LRTI) (2 points), and residence at a nursing home (2 points). APACHE II score variables were age, presence of chronic health problems, heart rate, respiratory rate, mean arterial pressure, temperature, Glasgow coma scale (GCS), Alveolar–arterial gradient or partial pressure of arterial oxygen (PaO2), arterial pH or serum bicarbonate, serum sodium, serum potassium, serum creatinine with reference to presence or absence of acute renal failure, hematocrit, and white blood cell count. SOFA score variables were mean arterial pressure or requirement for vasopressors, GCS, PaO2 over the fraction of inspired oxygen (FiO2), serum creatinine, serum bilirubin, and platelet count.

For the purposes of disease severity score calculations, we defined variables with reference to their original papers[ unless stated otherwise. In clarification of several MEDS score variables, a patient was considered to have an AMS if they had a GCS of less than 15, or if it was noted that they were ‘drowsy’ or ‘confused’ in their records. As our hospital does not routinely check for percentage bands on differential count, they were assumed to be normal if not reported. APACHE II score incorporates pH as one of its variables, but allows for the use of serum bicarbonate as a surrogate when pH data is unavailable.[ Both the APACHE II and SOFA scores utilized PaO2 as a component of their scores. As an arterial blood gas (ABG) was required for data on PaO2 and was infrequently performed, peripheral capillary oxygen saturation (SpO2) was used as a surrogate measurement where PaO2 was not available, a technique that has been described in the literature.[

Statistical analysis

Statistical analysis was conducted using SPSS version 25 (IBM corporation, Armonk, NY). Baseline characteristics and clinical parameters of patients were reported by primary outcome. Continuous variables were presented as means (standard deviation) and compared between groups with the Mann Whitney U test, while categorical variables were presented as numbers (percentage) and compared between groups with Pearson chi-squared test or Fisher exact test as appropriate. MEDS, APACHE II, and SOFA scores were calculated for each patient using the most abnormal clinical or laboratory value recorded for the duration of the patient's ED stay. For 15 patients who did not have their serum bilirubin values obtained, the median value of the other patients served as replacement in the calculation of their SOFA scores. Observed and expected mortality frequencies were compared across several MEDS score strata of increasing sepsis severity, with expected mortality frequencies derived from the original paper.[ Receiver operating characteristic (ROC) curve analysis was used to assess predictive performance of the scoring systems, and discriminatory ability for outcomes. Pairwise comparison of AUC values using the algorithm as suggested by DeLong and colleagues was conducted.[ For each scoring system, the sensitivity, specificity, positive, and negative predictive values were determined for its optimal cut-off point as indicated by the point on the ROC curve nearest to the upper left corner of the ROC graph.

Results

Patient enrolment and outcomes

A total of 368 patients clinically suspected to have sepsis in the ED were enrolled. One hundred nineteen patients did not meet criteria for SIRS and were excluded. Of the 249 patients included in the final analysis, 46 patients (18.5%) met the primary outcome of 30-day IHM (Fig. 1). Twenty-six patients (10.4%) met the secondary outcome of 72-hour mortality.

Figure 1

Patient enrolment flowchart with breakdown for 30-day in-hospital mortality (IHM). ED = emergency department, SIRS = systemic inflammatory response syndrome.

Baseline characteristics and clinical parameters

Table 1 displays univariate analysis of patient baseline characteristics and clinical parameters as stratified by primary outcome. Patients who met the primary outcome were older and more likely to have a history of ischemic heart disease as compared to patients who did not meet the primary outcome. Differences in gender, race, and several other comorbidities and medications taken were not found to be statistically significant between both groups.

Table 1

Baseline characteristics and clinical parameters.

On presentation at the ED, patients who met the primary outcome had a higher respiratory rate, lower temperature, and lower GCS as compared to patients who did not meet the primary outcome. The worst recorded systolic blood pressure, respiratory rate, and GCS for the duration of the patient's ED stay were also more abnormal in the group which met the primary outcome. Differences in presenting heart rate, presenting systolic blood pressure, source of infection, and disposition from the ED were not found to be statistically significant between both groups.

Analysis of MEDS score performance

Table 2 shows univariate analysis of MEDS score variables as stratified by primary outcome. Terminal illness, age more or equal to 65 years, respiratory distress, septic shock, AMS, and physician suspicion of LRTI were found to be present at an increased frequency in patients who met the primary outcome. Percentage bands on differential count more or equal to 5% (bandemia), platelet count less or equal to 150,000 platelets per mm3, and residence at a nursing home were not statistically different between both groups. Respiratory distress (odds ratio [OR] 8.80, 95% confidence interval [CI] 1.93–40.22) and terminal illness (OR 7.56, 95% CI 3.22–17.76) were strong predictors of the primary outcome.

Table 2

Analysis of MEDS score variables for prediction of 30-day IHM.

Table 3 shows patients as stratified into groups of increasing sepsis severity based on their MEDS score. For MEDS score 0 to 4, MEDS score 5 to 7, MEDS score 8 to11, MEDS score 12 to 14, and MEDS score ≥15, observed mortality frequencies were 0.0%, 3.7%, 10.6%, 33.3%, and 64.1%, respectively, and expected mortality frequencies were 0.9%, 2.0%, 7.8%, 20.2%, and 50.0%, respectively. Expected mortality frequencies were derived from the original paper.[ Although observed and expected mortality frequencies correlate well for strata of decreased sepsis severity (MEDS score 0–4, MEDS score 5–7, and MEDS score 8–11), they deviated to a larger extent for strata of increased sepsis severity (MEDS score 12–14, and MEDS score ≥15).

Table 3

Observed and expected mortality for various MEDS score strata.

Prediction of primary and secondary outcomes

Figure 2 shows the ROC curves of MEDS, APACHE II, and SOFA scores for prediction of primary outcome (30-day IHM). MEDS score was superior to both APACHE II and SOFA scores in predicting 30-day IHM, with AUC values of 0.87 (95% CI, 0.82–0.93), 0.77 (95% CI 0.69–0.85), and 0.78 (95% CI 0.71–0.85), respectively. On pairwise comparison of AUC values, discriminatory ability of MEDS score was superior and statistically significant (P ≤ .05) when compared to APACHE II score (Difference between areas = 0.10, 95% CI 0.04–0.17) or SOFA score (Difference between areas = 0.09, 95% CI 0.02–0.17). Discriminatory ability between APACHE II score and SOFA score was not statistically significant (difference between areas = 0.01, 95% CI –0.07 to 0.08).

Figure 2

Receiver operating characteristic (ROC) curves for prediction of 30-day IHM. Confidence intervals shown are for 95%. APACHE II = acute physiology and chronic health evaluation II, AUC = area under the ROC curve, IHM = in-hospital mortality, MEDS = mortality in emergency department sepsis, SOFA = sequential organ failure assessment.

Sensitivities, specificities, positive and negative predictive values for each score were calculated at its optimal cut-off point. MEDS score ≥12, APACHE II score ≥23, and SOFA score ≥5 performed at sensitivities of 76.1% (95% CI 61.2–87.4), 67.4% (95% CI 52.0–80.5), and 76.1% (95% CI 61.2–87.4), and specificities of 83.3% (95% CI 77.4–88.1), 73.9% (95% CI 67.3–79.8), and 65.0% (95% CI 58.0–71.6), respectively. Correspondingly, positive predictive values were 50.7% (95% CI 42.1–59.3), 36.9% (95% CI 30.1–44.3), and 33.0% (95% CI 27.8–38.7), and negative predictive values were 93.9% (95% CI 90.1–96.3), 90.9% (95% CI 86.8–93.9), and 92.3% (95% CI 87.7–95.3), respectively (Table 4).

Table 4

Performance of scoring systems for prediction of 30-day IHM.

For prediction of 72-hour mortality, MEDS score was superior to both APACHE II and SOFA scores with AUC values of 0.88 (95% CI 0.82–0.95), 0.81 (95% CI 0.72–0.89), and 0.79 (95% CI 0.71–0.87), respectively (Fig. 3).

Figure 3

Receiver operating characteristic (ROC) curves for prediction of 72-hour mortality. Confidence intervals shown are for 95%. APACHE II = acute physiology and chronic health evaluation II, AUC = area under the ROC curve, MEDS = mortality in emergency department sepsis, SOFA = sequential organ failure assessment.

Discussion

In this retrospective observational cohort study, we calculated disease severity scores for patients presenting to the ED with sepsis, and compared predictive ability for the primary outcome of 30-day IHM, and the secondary outcome of 72-hour mortality. MEDS score exhibited superior performance over APACHE II and SOFA scores on the prediction of primary outcome with AUCs of 0.87, 0.77, and 0.78, respectively (Fig. 2), and similarly for the secondary outcome with AUCs of 0.88, 0.81, and 0.79, respectively (Fig. 3). In the first utilization of the MEDS score in a Singaporean cohort, the MEDS score performed well in its ability for mortality risk stratification of septic patients.

Septic patients are commonly encountered in the ED, and utilization of the MEDS score can offer guidance on management decisions. In our study, univariate analysis of MEDS score variables have shown 6 out of 9 variables to be statistically different between patients who had 30-day IHM and those that did not (Table 2), with odds ratios ranging from 1.13 to 8.80. Observed mortality frequencies for various MEDS score strata provided additional information on how the MEDS score related to 30-day IHM in our patient population (Table 3). It may be noted that at higher scores, the observed mortality frequencies in our cohort deviated more widely from expected mortality frequencies. Previous MEDS validation papers have encountered similar phenomenon, and several have found that the MEDS score tends to underestimate mortality in cohorts of greater illness severity.[

Practical applicability of the MEDS score in the ED setting is also highlighted in this study by its ease of derivation as compared to the APACHE II and SOFA scores. Indeed, surrogate measurements had to be made in replacement of several APACHE II and SOFA score variables. These include SpO2 in replacement of PaO2, and serum bicarbonate in replacement of pH when data was unavailable. Furthermore, 15 patients did not have their serum bilirubin measured, and the median population value had to be used in replacement. The GCS variable utilized by both APACHE II and SOFA scores additionally, has been found to have only moderate interrater agreement in the ED,[ and does not differentiate between chronic and acute mental state deterioration. MEDS score by use of the variable AMS in place of GCS, which is taken in reference to the patient's baseline, may thus circumvent some of these issues. The usage of the APACHE II and SOFA scores in the ED may thus be hampered by their requirement for more comprehensive information.

Although the MEDS score has performed well in this study, implementation in the ED may still be faced with several challenges. The subjective variable ‘presence of terminal illness’, given twice the weight of other variables, has been shown in other studies to have the poorest agreement among all MEDS score variables.[ In this study, we have attempted to mitigate subjective interpretation by specifying the circumstances that it was to be used under, and by consistent usage of the term. Another variable which poses a challenge to usage of the MEDS score is bandemia. As our hospital does not routinely screen for bands on differential count, bands are assumed to be normal unless they were reported. Indeed, information on percentage bands were only available for four patients, and the variable bandemia was subsequently found not to be a statistically significant predictor of 30-day IHM on univariate analysis (Table 2). The lack of information on this variable may thus have impacted the predictive ability of the MEDS score in our study. ‘Nursing home resident’ as a variable was also not found to have predictive significance for 30-day IHM (Table 2), and this could have resulted from differences in patterns of health care utilization between the populations.

Knowledge of a patient's disease severity and prognosis affords the physician valuable information. The MEDS score by way of its ability to risk stratify and prognosticate septic patients, offers an opportunity for appropriate intervention to improve patient outcomes. Information provided by the MEDS score can also be influential in decision making for allocation of resources, and even for conversations among the healthcare team and caregivers. Our study has validated the usage of the MEDS score in a Singaporean ED cohort, and demonstrated its ability for mortality risk stratification of 30-day IHM and even 72-hour mortality. With superior predictive performance as compared to the other disease severity scores and practical ease of use, the MEDS score can be considered for implementation.

There are several limitations to this study. As patients included were more severely ill patients triaged to PACS 1 or up-triaged from PACS 2, generalizability to all septic patients presenting at the ED may be hindered. Further studies including patients across all triage severities would be required to validate our results. Second, in the original paper for derivation of the MEDS score,[ study inclusion was based on physician suspicion of infection as evidenced by acquisition of a blood culture, and subsequent hospital admission. Our study however, utilized physician suspicion of infection and the SIRS criteria. The SIRS criteria were used to define a true sepsis population, as opposed to one in which there was a physician suspicion of sepsis. Such an approach has also been utilized by other studies on the MEDS score.[ However, because of differences in inclusion criteria, study population characteristics and hence performance of the MEDS score between the studies may differ.

Conclusion

The MEDS score performed well in this validation study in a Singaporean ED cohort, outperforming established scoring systems, such as the APACHE II and SOFA scores on ROC analysis for prediction of 30-day IHM. This superiority extended also to the prediction of 72-hour mortality. With its ability to mortality risk stratify and prognosticate patients coupled with its practical applicability, the MEDS score can be considered for use in the triage and management of septic patients presenting at the ED.

Acknowledgments

The authors would like to express thanks and gratitude for all contributions made from doctors, nurses, and research assistants from the Department of Emergency Medicine, Singapore General Hospital.

Author contributions

Conceptualization: Jeremy Zhenwen Pong, Nan Liu, Marcus Eng Hock Ong.

Data curation: Jeremy Zhenwen Pong, Zhi Xiong Koh, Mas’uud Ibnu Samsudin.

Formal analysis: Jeremy Zhenwen Pong, Zhi Xiong Koh, Stephanie Fook-Chong, Nan Liu.

Funding acquisition: Jeremy Zhenwen Pong, Nan Liu.

Methodology: Nan Liu.

Project administration: Zhi Xiong Koh, Mas’uud Ibnu Samsudin.

Resources: Mas’uud Ibnu Samsudin, Stephanie Fook-Chong.

Supervision: Nan Liu, Marcus Eng Hock Ong.

Writing – original draft: Jeremy Zhenwen Pong, Nan Liu.

Writing – review & editing: Jeremy Zhenwen Pong, Zhi Xiong Koh, Mas’uud Ibnu Samsudin, Stephanie Fook-Chong, Nan Liu, Marcus Eng Hock Ong.