Use of SOFA score in cardiac arrest research: A scoping review.

BACKGROUND: The Sequential Organ Failure Assessment (SOFA) score is a commonly used severity-of-illness score in cardiac arrest research. Due to its nature, the SOFA score often has missing data. How much data is missing and how that missing data is handled is unknown.
OBJECTIVES: We conducted a scoping review on cardiac arrest studies using SOFA, focusing on missing data. DATA SOURCES: PubMed, Embase, and Web of Science. STUDY SELECTION: All English-language peer-reviewed studies of cardiac arrest with SOFA as an outcome or exposure were included. DATA EXTRACTION: For each study, quantity of missing SOFA data, analytic strategy to handle missing SOFA variables, whether/to what degree mortality influenced the amount of missing SOFA scores), SOFA score modifications, and number of SOFA measurements was extracted. DATA SYNTHESIS: We included 66 studies published between 2006-2019. Five studies were randomized controlled trials, 26 were prospective cohort studies, and 25 were retrospective cohort studies. SOFA was used as an outcome in 36 (55%) and a primary outcome in 10 (15%). Nine studies (14%) mentioned the quantity of missing SOFA data, which ranged from 0 to 76% (median: 10% [IQR: 6%, 42%]). Twenty-seven (41%) studies reported a method to handle missing SOFA. The most common method used excluded subjects with missing data (81%). In the 50 studies using serial SOFA scores, 11 (22%) documented mortality prior to SOFA measurement; which ranged from 3% to 76% (median: 12% [IQR: 6%-35%]).
CONCLUSIONS: Missing data is common in cardiac arrest research using SOFA scores. Variability exists in reporting and handling missing SOFA variables.

Introduction

The Sequential (or Sepsis-related) Organ Failure Assessment (SOFA) score is a severity of illness score that is commonly used in cardiac arrest research. SOFA, developed in 1994, uses laboratory and clinical data to categorize organ failure by summing six scores from individual organ systems: respiratory, cardiovascular, hepatic, coagulation, renal, and neurological.

Since the SOFA score requires values from six organ systems (Supplemental Table 1), some of which require a blood sample, missing data is common. For example, when experts met to revise sepsis guidelines in 2014–2015, which advocate using the SOFA score to diagnose sepsis in the ICU, the derivation dataset used to provide evidence for the Sepsis-3 update had significant missing data, especially for the hepatic, respiratory, and neurologic scores. Additionally, missing data was heterogeneous across the organ systems with 62% hepatic, 74% respiratory, 15% coagulation scores missing in non-ICU patients., The reason for missing data is unknown.

While there has been some interest in exploring this missing data in sepsis,,6, 7, 8 this has not been investigated in cardiac arrest, despite increasing use of the SOFA score in cardiac arrest research. Because the method for handling missing data can alter conclusions and influence bias, an understanding of both the existence and quantity of missing SOFA scores and the methods by which this missing data is addressed are vital in understanding the results of cardiac arrest research. Since we hypothesized that there would be a high proportion of missing data in studies using the SOFA score, that the level of documentation for missing data seen would be rare, and that there would be significant variability in how the missing data would be handled, we aimed to describe this issue in order to aid interpretation of past studies and to inform future studies. Therefore, we conducted a scoping review with two objectives: (1) to determine the degree and cause of missing SOFA score data and whether it varies by study type, and (2) to define methods used to handle missing SOFA score data and the frequency with which each method is used.

Methods

Search strategy

A search strategy was developed in consultation with a research librarian. The databases searched were PubMed, EMBASE, and Web of Science (Appendix 1 in Supplementary material). We did not search for unpublished data. Only studies published in English were included in this review. The search was not limited to a specific time period, as the SOFA score was not developed until 1994. The search was performed on May 7, 2019. The scoping review protocol was registered with Open Science Framework, which can be publicly accessed at https://osf.io/32wmr.

Inclusion criteria

This review included studies that primarily involved adult patients with cardiac arrest who survived their index arrest and were admitted to the hospital. The studies were required to have SOFA score as a primary/secondary outcome or as an exposure. Only human studies and peer-reviewed literature were included. Reviews, conference abstracts, dissertations, and opinion papers were not included. Only studies published in English were considered. In this scoping review, the key concept was the method used for handling missing SOFA score data in statistical analysis. Secondary concepts were the degree and cause (such as early mortality) of missing data in each study and whether missing data was documented.

Exclusion criteria

This review did not include abstracts, editorials, or gray literature such as dissertations or theses. Only full-length peer-reviewed manuscripts were included. Studies with a primarily pediatric population were excluded.

Study selection

Following the search process, identified citations were uploaded into Endnote (Clarivate Analytics, PA, USA) and duplicates removed. Titles and abstracts were then screened by two independent reviewers to assess if they meet inclusion criteria via Covidence (Covidence, Melbourne, Australia). Studies that met/potentially met inclusion criteria were retrieved in full and assessed in detail against inclusion criteria. Full text studies that did not meet the inclusion criteria were excluded; reasons for exclusion were provided. Search results were presented in a PRISMA flow diagram. Any disagreements between the reviewers were resolved by a third reviewer. The PRISMA-ScR checklist was used in drafting this manuscript (Appendix 2 in Supplementary material).

Data extraction

Data was extracted using a data extraction form (Appendix 3 in Supplementary material) including specific details about the trial, study population, and mortality and SOFA score outcomes as well as information about missing SOFA score data, any modifications to the SOFA sore, and statistical analysis of SOFA. Any disagreements that arose between the reviewers were resolved with a third reviewer as arbiter.

Statistical analysis

To compare the amount of missing data by publication, a Fisher’s exact test was used. Data was described as medians with interquartile ranges (IQR) or counts with proportions, as appropriate. A post hoc non-parametric test for trend was performed to analyze rates of documentation of missing data over year. A p-value <0.05 was considered significant. All analyses were conducted using Stata 14.2 (College Station, Texas).

Results

The initial search provided 408 abstracts for review after duplicates were removed. One hundred and forty of those abstracts underwent full-text review (kappa between reviewers: 0.91), and 66 were included in this scoping review (Fig. 1). The 66 studies were published between 2006 and 2019 and consisted of five randomized controlled trials, 26 prospective observational studies, and 25 retrospective observational studies (Table 1).

Fig. 1

PRISMA flow diagram.

Table 1

Characteristics of studies and their use of the SOFA score by study type.

	Type of study
	RCT (n = 5)	Observational study (n = 61)
Median number of subjects	794 (IQR: 139, 897)	143 (IQR: 52, 226)
SOFA used as primary outcome	1 (20%)	9 (15%)
SOFA used as secondary outcome	4 (80%)	32 (52%)
Missing SOFA
Quantity of missing SOFA documented	1 (20%)	8 (13%)
Median quantity of missing SOFA	35%	8% (IQR: 4%, 46%)
Missing serial SOFA
Serial use of SOFA	5 (100%)	45 (74%)
Quantity of missing serial SOFA data documented	2 (40%)	9 (12%)
Median quantity of missing serial SOFA	3%; 35%	12% (IQR: 7%, 20%)
Method for imputing missing SOFA
Exclude patients with missing SOFA	3 (60%)	19 (31%)
Maximum SOFA while patient was alive	0 (0%)	3 (5%)
Modified SOFA	0 (0%)	3 (5%)
Impute using earlier and later SOFA	0 (0%)	1 (2%)
Sensitivity analyses to test how missing SOFA was handled
Imputed value for death and floor discharge	1 (20%)	0 (0%)
Adjust for mortality difference prior to SOFA measurement	1 (20%)	0 (0%)

Missing SOFA

Only nine studies (14%)11, 12, 13, 14, 15, 16, 17, 18, 19 noted the quantity of missing SOFA data, which ranged from 0 to 76% (median: 10% [IQR: 6%, 42%]). In the studies with SOFA as an outcome, only four studies (11%) documented the amount of missing data; when SOFA was the primary outcome, missing data was only quantified in 2/10 (20%). In the 50 studies (76%) using SOFA at time points after baseline, only 11 (22%) mentioned the quantity of data that was truncated due to mortality or patient discharge prior to SOFA measurement (potential causes of missing SOFA scores); when mentioned, it ranged from 3% to 76% (median: 12% [IQR: 6%–35%]). There was no trend over time for increasing documentation of missing SOFA (p = 0.192). There was also no statistically significant difference in documentation by journal: in the four journals with the most articles represented in this study, the percent of articles that had documentation of missing data ranged from 10 to 29%, p = 0.471.

Methods to handle missing SOFA

Only 27 (41%) studies reported a method to handle missing SOFA (Supplemental Table 4). The most common method was to exclude subjects with missing data from analysis,,,,19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36 (81%). Other methods (non-exclusive of each other) included using the maximum SOFA score while subjects were alive,, (11%), modifying the SOFA score by excluding the neurologic component, which had the most missing values,, (11%), and using the subject’s earlier and later SOFA scores to impute the missing values (4%).

When SOFA was the primary outcome, only four studies (40%) reported a method to handle missing data.,,, Of these, three (75%) excluded subjects with missing data,, and one (25%) used a modified SOFA score to handle the subjects who did not have PaO2 data available within twelve hours of the given time point (3.2% of cases). In this case, the respiratory component of SOFA was calculated using hourly SpO2, a method based on an article by Pandharipande et al.

Modifications to the SOFA score

Thirty-one studies (47%) described a modification made to the SOFA score. In 9/31 (29%) studies, the “full” SOFA score including all six organ system components was used as well.

The most commonly described modifications were not using the neurologic component of the SOFA score,,,,,,41, 42, 43 (9; 29%), only using the cardiovascular component of SOFA,,,,,43, 44, 45, 46,, (11; 35%), using SOFA to indicate the presence of multiple organ dysfunction,,,,,,, (8; 26%), and categorizing patients according to a post-cardiac arrest syndrome (PCAS) severity score, (2; 6%). These can be found in more detail in Appendix 3 in Supplementary material.

Discussion

This scoping review describes the frequency and proportion of SOFA score data that is missing in cardiac research studies. In addition, we describe the strategies used, to date, to handle this issue. We found that the quantity of missing SOFA score data varied widely, ranging from no missing values to over three-quarters missing. We also found that authors uncommonly reported the amount of missing SOFA data that was present in their study, and less than half discussed a method to handle this missing data. Although there is ample data to show that excluding patients with missing data leads to potentially biased results and less power to show an effect,, this was the most common strategy described. As dropping observations with missing data is the default strategy for missing data in many statistical software packages, it could be that this strategy was employed in at least a few of the studies that did not document how missing data was handled.

Additionally, and potentially more troublesome, is excluding patients who die prior to the measurement of SOFA. These patients do not have missing data per se, because it is not possible for them to have a value, and we do know what happened to them. Excluding these patients could completely change the results of a study if the mortality is not balanced at the time point of interest. For example, if a new treatment reduces mortality but causes higher SOFA scores in the patients who would have otherwise died, the comparison treatment, although inferior, could look superior if the outcome was SOFA score and patients who died prior to measurement were excluded. This is because the treatment arm would have a higher mean/median SOFA score because it includes the patients who would have otherwise died but they are dropped from the comparison arm. This problem is not unique to cardiac arrest but is present in other critical illnesses, such as sepsis, with substantial early mortality.,, Additionally, SOFA score is not the only measure affected by this early mortality — other outcomes, such as biomarkers, and severity of illness scores, such as APACHE, have similar limitations. Future cardiac arrest research would be improved if the quantity of missing data, the reasons why this data was missing (if known), and the methods (if any) used to handle the missing data are clearly documented.

This study highlights trends in approaches missing SOFA data in cardiac arrest research. The first is the lack of documentation for how missing data is handled. Less than half of the included studies provided this information. This is problematic because it does not allow for proper assessment of study results and, as we found in this study, there is no indication that documentation has increased over time or that it varies by journal. In fact, according to the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) checklist, which was developed to improve the quality of the methods and standardize the documentation of observational studies, quantifying the number of participants with missing data and describing how missing data were handled is vital to give the audience the ability to assess strengths and weaknesses of studies. The second is the tendency to exclude patients with missing data, which can cause bias when the data is not missing at random, which could be especially troublesome when mortality is ignored. It also reduces statistical power and increases the chances of a Type II error.,

To our knowledge, this is the first study to examine strategies to handle missing SOFA score data in cardiac arrest research. Limitations of this study include the lack of detail on strategies used, which made it difficult to truly know what methods were used. Another limitation is the possibility that some studies were not included in this scoping review, although we worked with research librarians to ensure the broadest possible search terms.

Conclusion

The issue of missing SOFA data in cardiac arrest studies is not commonly acknowledged or handled — when addressed, the most commonly reported method is to exclude subjects with missing data. This may lead to bias in results.

Conflict of interest

The authors have nothing to declare.

CRediT authorship contribution statement

Anne V. Grossestreuer: Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review & editing, Investigation, Data curation. Tuyen T. Yankama: Investigation, Writing - original draft, Writing - review & editing. Ari Moskowitz: Writing - original draft, Investigation, Writing - review & editing. Long Ngo: Writing - review & editing, Supervision. Michael W. Donnino: Conceptualization, Writing - review & editing, Supervision.