Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data.

INTRODUCTION: Administrative data are used to generate estimates of sepsis epidemiology and can serve as source for quality indicators. Aim was to compare estimates on sepsis incidence and mortality based on different ICD-code abstraction strategies and to assess their validity for sepsis case identification based on a patient sample not pre-selected for presence of sepsis codes.
MATERIALS AND METHODS: We used the national DRG-statistics for assessment of population-level sepsis incidence and mortality. Cases were identified by three previously published International Statistical Classification of Diseases (ICD) coding strategies for sepsis based on primary and secondary discharge diagnoses (clinical sepsis codes (R-codes), explicit coding (all sepsis codes) and implicit coding (combined infection and organ dysfunction codes)). For the validation study, a stratified sample of 1120 adult patients admitted to a German academic medical center between 2007-2013 was selected. Administrative diagnoses were compared to a gold standard of clinical sepsis diagnoses based on manual chart review.
RESULTS: In the validation study, 151/937 patients had sepsis. Explicit coding strategies performed better regarding sensitivity compared to R-codes, but had lower PPV. The implicit approach was the most sensitive for severe sepsis; however, it yielded a considerable number of false positives. R-codes and explicit strategies underestimate sepsis incidence by up to 3.5-fold. Between 2007-2013, national sepsis incidence ranged between 231-1006/100,000 person-years depending on the coding strategy.
CONCLUSIONS: In the sample of a large tertiary care hospital, ICD-coding strategies for sepsis differ in their accuracy. Estimates using R-codes are likely to underestimate the true sepsis incidence, whereas implicit coding overestimates sepsis cases. Further multi-center evaluation is needed to gain better understanding on the validity of sepsis coding in Germany.

Introduction

Acknowledging that sepsis is the leading cause of death from infection and affecting more than 30 million patients globally [1], the World Health Organization declared the prevention, diagnosis and management of sepsis as leading priority in its member states [2, 3]. For most countries, population-level sepsis incidence and mortality rates remain unknown, thus this resolution urges to implement measures of specific epidemiologic surveillance and “to apply and improve the use of the International Classification of Diseases system to establish the prevalence and profile of sepsis” [2]. In the US and several European countries, estimates on sepsis incidence are commonly drawn from retrospective studies based on hospital claims data using different International Classification of Diseases (ICD) codes for case identification [4, 5]. Administrative data are also increasingly used to compare risk-adjusted mortality rates in different conditions between health care providers [6-8]. In Germany, both the Initiative for Quality Medicine [9] and the German Quality Network Sepsis [10] provide their participating hospitals quality indicators on hospital mortality based on diagnostics-related-groups (DRG) data. Various ICD combinations emerged which attempt to capture sepsis patients. While explicit coding approaches use ICD codes for septicemia/sepsis [11], implicit coding approaches link infection and organ dysfunction codes to mirror clinical sepsis criteria [5]. Depending on the underlying codes, estimates on sepsis incidence and mortality differ considerably [12]. Efforts were made to validate ICD case identification strategies compared to a gold standard of manual patient chart review [13]. These studies have shown good specificity, but poor sensitivity of sepsis coding in claims data [14]. Recent population based studies from Scandinavia suggest that ICD abstraction may result in an up to 6-fold underestimation of the incidence of traditional severe sepsis and also in comparison to the newly proposed sepsis definition designated as “sepsis-3” [15] compared to medical record review [16, 17]. Existing validation studies mostly rely on small sample sizes, selective populations and restrict their review to charts selected based on the presence or absence of relevant ICD sepsis codes [13], an important source of bias, since sensitivity could not reliably be estimated by this approach. This is why we aimed compare validity of different ICD 10 code abstraction regarding a gold standard of manual patient chart review not pre-selected for presence of sepsis codes. Furthermore, no data exists in Germany on the variations in incidence and mortality due to different identification of sepsis cases in administrative data. Above that, it is yet unknown how patients identified using new “sepsis-3” definitions will be coded in administrative claims, perhaps impacting estimates of sepsis epidemiology. Further aims of this study were therefore to compare ICD code abstraction strategies’ estimates on sepsis incidence and mortality by using German hospital claims data, and to assess the concordance of “sepsis-3” definitions with cases identified by retrospective chart review.

Materials and methods

ICD coding strategies for sepsis

ICD coding strategies for sepsis in claims data were selected based on a review of international studies applying ICD abstraction strategies (S1 File). Codes for five main coding strategies were selected and translated from ICD-9 to ICD-10-German Modification (GM): for sepsis: I) R-codes, II) explicit approach (all sepsis codes = microbiological sepsis codes and R-codes); for severe sepsis: III) R-codes, (IV) explicit and organ dysfunction codes and V) implicit approach (presence of infection and organ dysfunction codes, Angus method [5]). R-codes for sepsis (R65.0!), severe sepsis (R65.1!) and septic shock (R57.2) are defined according to modified ACCP/SCCM consensus criteria [18]. In Germany, R65.0 and R65.1 were introduced in 2005; R57.2 in 2010. The use of R65.0 is restricted since 2007 a) to cases with positive blood culture and 2/4 positive SIRS criteria or b) in case of negative blood cultures only when 4/4 SIRS criteria are present. For specific codes, see S1 File.

Validation of coding strategies

Validation of ICD coding strategies was performed against a gold standard of manual chart review of patients admitted to a large university hospital in Germany (Jena University Hospital). We planned the sample size for the validation sample based on considerations for the width of a two-sided 95% confidence interval (CI) for sensitivity of explicit sepsis codes. We expected a sensitivity of 10% [14] and aimed at a 5% margin of the CI. This resulted in a sample size estimation of n = 139 true cases with sepsis. Given an expected inpatient incidence of sepsis of 2.3% [5], we decided for a stratified sampling of patient charts. Cases were sampled from a 2 x 2 classification based on any need for ICU care after admission, and length of stay > or ≤ 6 days. The same number of patients was sampled from each stratum, thus resulting in higher sampling probabilities for patients with longer stay and ICU treatment. Based on considerations of the available resources to conduct the chart review, we sampled a total of 1120 charts from all adult hospitalizations admitted between 2007 and 2013. From the sample, every chart was reviewed for the presence of infection and sepsis independently by two investigators blinded to the patients’ administrative data using a structured protocol (S1 File), discrepancies were resolved by discussion and third review. Sepsis was defined according to modified ACCP/SCCM consensus criteria (“sepsis-1”) [19] based on all available patient data. Organ dysfunctions were evaluated for diagnosis of severe sepsis and were considered if most likely caused by sepsis. For each patient with infection, additional information on microbiology, ICU treatment, severity scores and further clinical data were extracted. In a separate review conducted in the same way, cases fulfilling the “sepsis-3” definitions were identified [15]. In case of missing clinical information, no Sequential [Sepsis-related] Organ Failure Assessment (SOFA) score points were assigned in the respective category. Furthermore, we identified patients with false negative or false positive ICD coding and evaluated reasons for incorrect coding based of in available information in the patient chart. The study was approved by the Jena University Hospital IRB (file 4333-02/15) and the Jena University Hospital data protection officer. The need for informed consent was waived, since only routinely obtained clinical data were used.

German national incidence and outcomes of different coding abstraction strategies

National sepsis incidence and mortality were analyzed between 2007 and 2013 using the nationwide German DRG statistics, which includes inpatient claims data from all acute-care hospitals in Germany and is accessible by the Federal Statistical Office (S1 File). We identified septic patients (≥18y) by strategies I)-V) applied to the primary and all secondary discharge diagnoses, since there is no chronological or hierarchical order of secondary diagnoses in Germany. Comorbidities were defined using the Charlson Comorbiditiy Index [20]. Organ dysfunction codes were classified as cardiovascular, respiratory, central nervous system, renal, metabolic, hematologic and hepatic [5]. We used procedure codes to identify patients with ICU treatment, ventilation and hemodialysis (S1 File).

Analyses

Interrater-agreement in the manual chart review was assessed by the S statistic for dichotomous data for the diagnosis of infection and by the S statistic with linear weighting for the diagnosis of sepsis (no sepsis, sepsis without organ dysfunction, severe sepsis/shock) [21], an improved alternative statistic to Fleiss kappa [22]. Characteristics of patients with and without sepsis were compared by standard descriptive statistics. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the coding strategies regarding the gold standard were calculated correcting for sampling weights. Confidence intervals on the 95% level (95% CI) were obtained by 2000 weighted bootstrap samples using the non-parametric percentile method. Cross-tabulation of cases identified by different strategies and the gold standard was done and depicted graphically using the UpSetR web application [23]. Patients with sepsis identified in national German DRG data were characterized by standard descriptive statistics. Incidence and mortality were compared between the years 2007–2013. The annual population-based incidence and mortality rate of sepsis was directly age- and sex-standardized to the German population structure as of 31st December 2010 based on nationwide population data provided by the Federal Statistical Office. Analyses were performed using IBM SPSS Statistics® 20.0 and 23 and the statistical package R [24].

Results

Validation of ICD code abstraction strategies

Of 1120 cases selected by stratified sampling 937 charts were available; 183 charts were inaccessible for accompanying parents, diagnostic or therapeutic day cases without inpatient charts or when charts were not available at the archive at the time of review. Inter-rater-agreements were high both for diagnosis of infection and sepsis (S [95% CI] of 0.87 [0.84, 0.90] and 0.93 [0.91, 0.95], respectively). 151 patients had sepsis, including 81 patients with severe sepsis/septic shock. For patient demographics, see Table 1. Among the strategies for sepsis and severe sepsis case identification, the explicit coding strategy yielded a substantially higher sensitivity value compared to R-codes, while having no decreased PPV (Table 2). The implicit coding strategy for severe sepsis showed the highest sensitivity, but yielded a substantial number of false positives, resulting in a positive predictive value of only 22.1%. After correction for sampling weights, we found an underestimation of cases by up to 3.5-fold by R codes and explicit coding. Implicit coding led to a 2.7-fold overestimation of severe sepsis cases. The overlap between cases identified by manual patient chart review and from administrative data is shown in Fig 1 and Figure B in S2 File, as well as an overview on reasons for false positive or negative coding.

Table 1

Demographics of patients identified by clinical chart review in a sample stratified by length of stay and ICU treatment (n = 937).

Characteristic	No infection	Infection w/o sepsis	Sepsis without organ dysfunction	Sepsis with organ dysfunction (severe sepsis)	Septic shock
cases	650	135	70	17	64
age (y, median [IQR])	61 [44–72]	70 [55–79]	70 [54–70]	68 [53–79]	70 [60–78]
male sex (%)	52	50	63	65	63
hospital mortality (%)	8.0	11.1	11.4	29.4	60.9
microbiological prove of infection (%)	n/a	62	62	71	83
blood culture rate (%)	0.2	30	46	76	94
positive blood cultures (%)	0	27	34	62	40
organ dysfunctions (number, median [IQR])	n/a	n/a	n/a	3 [3–4]	3 [2–4]
• encephalopathy (%)				43	50
• thrombocytopenia (%)				24	44
• respiratory failure (%)				65	52
• renal failure (%)				24	57
• metabolic acidosis (%)				24	66
organ replacement therapy according to OPS codes:
• mechanical ventilation (%)	19	33	51	56	89
• renal replacement (%)	5	12	6	18	44
focus of infection
• respiratory (%)	n/a	31	49	71	69
• abdominal (%)	n/a	19	21	12	39
• genitourinary (%)	n/a	30	16	12	27
• wound/soft tissue (%)	n/a	21	10	24	6
• pregnancy-related (%)	n/a	0	0	0	0
• device-related (%)	n/a	5	6	0	13
• other (%)	n/a	7	14	12	14
• unknown (%)	n/a	6	10	0	3

IQR = interquartile range

Table 2

Sensitivity, specificity, PPV and NPV of ICD abstraction strategies for sepsis and severe sepsis against a gold standard of manual patient chart review.

Characteristic	ICD abstraction strategies for sepsis(incl. severe sepsis)		ICD abstraction strategies for severe sepsis(incl. septic shock)
	R codes (R65.0!, R65.1!, R57.2)	explicit sepsis coding*	R codes (R65.1!, R57.2)	explicit sepsis coding +organ dysfunction	implicit sepsis coding (infection + organ dys.)
Sensitivity (95% CI)	22.1% (15.2–29.1)	25.7% (19.2–33.1)	25.1% (16.0–34.6)	41.9% (30.9–51.9)	59.0% (48.1–69.1)
Specificity (95% CI)	99.6% (99.1–100)	99.6% (99.1–100)	99.6% (99.1–100)	99.4% (99.8–99.9)	95.7% (94.4–97.1)
PPV	77.2%	78.5%	56.1%	59.6%	22.1%
NPV	95.8%	95.9%	98.5%	98.8%	99.1%

* explicit sepsis codes: A02.1, A20.0, A20.7, A21.7, A22.7, A24.1, A26.7, A28.2, A32.7, A39.2, A39.3, A39.4, A39.1, A40.-, A41.-, A42.7, A48.3, B00.7, A54.8, B37.7, B37.6, B49, A49.9, R65.0!, R65.1!, R57.2

Fig 1

Intersecting sets of sepsis patients ≥18y in Germany 2007–2013 by different ICD code abstraction strategies; horizontal bars represent the numbers of patients identified by a specific coding strategy; vertical bars represent the number of patients identified by only one or several coding strategies indicated by dots with connecting lines (all numbers divided by 1000, generated with UpSetR[23]).

Proposed new sepsis definitions

From 70 cases clinically defined as sepsis without organ dysfunction according to the “sepsis-1” criteria in the validation cohort, the majority (98%) had no organ dysfunction as defined by SOFA change and thus was classified non-septic according to the new definitions (Table H in S2 File). All severe sepsis patients (n = 17) met the “sepsis-3”-criteria. This was also true for septic shock patients (n = 64). However, n = 4 (8%) had no elevated lactate and thus did not meet the “sepsis-3”-criteria for septic shock.

National estimates on sepsis incidence rates and mortality

Between 2007 and 2013 there was a steady increase in hospital treated sepsis incidence for any ICD coding abstraction strategy. The annual increase was most pronounced for implicit sepsis coding (Fig 2). The average annual sepsis incidence in adults ranged from 231/100,000 persons using R-codes, 318/100,000 using all sepsis codes (explicit coding strategy) to 1006/100,000 using any combination of infection and organ dysfunction (Table 3, Table D in S2 File). There was an increase in the absolute number of annual hospital deaths (Table B in S2 File) for all abstraction strategies despite decreasing case fatality rates (Table C and Figure A in S2 File). The average number of hospital deaths with sepsis per year ranged from 71 to 200/100,000 population depending on abstraction strategies (Table 3, Table E in S2 File). Sepsis cases identified using different abstraction strategies had differences in demographics, comorbidities, length of stay and mortality (Table 3). Of cases identified by any of these strategies, 72% of all cases met only the implicit case definition and had neither R nor explicit sepsis codes.

Fig 2

Course of hospital treated sepsis incidence ≥18y in Germany per 100,000 persons ≥18y per year by ICD code abstraction strategies, adjusted to the German population ≥18y in 2010.

Table 3

Cases, incidence, mortality and demographics of sepsis patients ≥18y in Germany 2007–2013 by different ICD code abstraction strategies.

Characteristic	ICD abstraction strategies for sepsis(incl. severe sepsis)		ICD abstraction strategies for severe sepsis*
	R codes (R65.0!, R65.1!, R57.2)	explicit sepsis coding*	R codes (R65.1!, R57.2)	explicit sepsis coding +odf**	implicit sepsis coding (inf+odf**)
cases 2007–2013	1,098,222	1,511,510	580,778	941,957	4,785,511
incidence / 100,000 population ≥18y / year	231	318	122	198	1006
hospital deaths / 100,000 population ≥18y / year	71	87	57	77	200
age (y, median [IQR])	72 [62–80]	72 [61–80]	72 [62–80]	72 [62–80]	75 [65–82]
male sex (%)	56	55	58	57	52
mortality (%)	30.8	27.5	47.0	38.7	19.8
ICU treatment (%)	37.4	31.8	53	45.2	25.0
CCI (median [IQR]	3 [1–4]	2 [1–4]	3 [1–5]	3 [1–5]	3 [1–4]
hospital LOS (d, median [IQR])	15 [8–28]	14 [8–27]	17 [8–33]	17 [8–32]	13 [7–22]

* explicit sepsis codes: A02.1, A20.0, A20.7, A21.7, A22.7, A24.1, A26.7, A28.2, A32.7, A39.2, A39.3, A39.4, A39.1, A40.-, A41.-, A42.7, A48.3, B00.7, A54.8, B37.7, B37.6, B49, A49.9, R65.0!, R65.1!, R57.2

** organ dysfunction

CCI = Charlson Comorbidity Index, IQR = interquartile range

Discussion

Using manual patient chart review as gold standard in a single center validation study in a large tertiary care hospital in Germany, we found that current ICD abstraction strategies differ substantially in their accuracy of sepsis case identification in administrative data. There is a trade-off between sensitivity and positive predictive value across different strategies. Explicit coding strategies have a better positive prediction compared to implicit coding strategies, but show a limited sensitivity and may miss a relevant number of sepsis cases in administrative data. R-codes and explicit sepsis coding strategies may underestimate sepsis incidence by 3.5-fold and 3-fold, respectively. Severe sepsis incidence rates may also be underestimated by 2.2-fold and 1.4-fold when using R-codes or explicit strategy, whereas implicit strategies risk overestimation by 2.7-fold. Our findings are in accordance with the results of studies from Denmark, Sweden and the US, which found an underestimation of severe sepsis cases in administrative data [4, 16, 17]. In a recent population based study from Sweden [17], a minority of 15.6% of patients with clinically diagnosed sepsis according to “sepsis-3” was coded as sepsis. In the US, only 30.5% of sepsis cases identified by “sepsis-3” criteria in eletronic health records had an explicit sepsis code [4].

Under the tentative assumption that the coding practices in our single center study would be representative for all German hospitals causing an underestimation using explicit sepsis coding and an overestimation using implicit sepsis coding, the average number of severe sepsis cases for Germany for the period between 2007 and 2013 would range between 277-372/100.000 persons. This would still be considerably less compared to the 687/100 000 traditional severe sepsis and 780/100 000 “sepsis-3” definition cases for Sweden. A US-wide projection of a recent national report using electronic health record data of representative US hospitals resulted in 1.67 million cases which corresponds to an incidence of approx. 517/100 000 persons [4]. Using EHR data, incidence rates of severe sepsis remained stable over time between 2009 and 2014, as did the mortality defined as combined outcome of in-hospital death (15% in 2014) and discharge to hospice (6.2% in 2014). Comparing temporary dynamics of hospital discharge diagnoses in the same patient cohort, they found an increase in coded severe sepsis incidence by an annual mean of 10% and a decrease in mortality by a mean of 7% annually. Given these differences, the coding of less severe sepsis cases may be a main contributor to the trends observed using hospital discharge data and EHR-based clinical data may provide more objective estimates for sepsis surveillance.

Similarly, we observed a significant increase of age- and sex-standardized sepsis incidence in Germany irrespective of the coding strategy, which went along with a moderate decline in mortality. However, there are also important differences compared to the findings in Sweden [17] and the US [4]. First, incidence rates for severe sepsis are substantially lower in Germany. This may be due to regulations for the coding of sepsis to avoid overcoding: Patients without blood culture sampling cannot be coded as sepsis. In case of negative blood cultures, patients need to fulfill 4/4 SIRS criteria. Since in only about 2/3 of cases blood cultures are collected and only half deliver positive results [25], this may result in an underestimation of sepsis cases. Above that, 12% of severe sepsis patients have negative SIRS criteria [26] and thus are excluded from coding by the German coding regulations. In Germany, the overall incidence of hospital-treated infection, and the proportion of coded sepsis among patients with infectious diseases is substantially lower than in Sweden and the US: Nearly 30% of infection patients who received intravenous antibiotics in Sweden were retrospectively diagnosed as sepsis patients with organ dysfunction according to the new sepsis definitions (780 of 2425/100,000 population) [17]. In the US, this ratio was 15% (535 of 3480/100,000 population) [27] compared to 7% in Germany [28]. These differences may result from a considerable undercoding of sepsis cases and contribute to the lower incidence rates observed in our study. As our analyses have shown, the number of severe sepsis cases increased when sepsis was identified by the combination of infection and organ dysfunction (implicit strategy), which is less prone to coding bias. This confirms the recent results of a US-study showing that trends in sepsis over time match those of infection cases requiring mechanical ventilation [27]. In contrast other studies using administrative data [4, 29], mortality rates were higher in Germany and only declined moderately over time. These differences may arise from several factors, including the higher age of our patients (median 72 years compared to a mean of 66 years in the US [4]), longer hospital length of stay, potentially reflecting varying discharge policies between countries, and that only a minimal, stable proportion of patients was discharged to hospice (0.2% of severe sepsis patients [28] compared to 6.2% in the US [4]). The admission rates to ICU facilities were decreasing over time in severe sepsis patients. Furthermore, differences in health care, population structure and the lack of nationwide quality improvement initiatives for sepsis in Germany are possible factors that may contribute to the between-country differences in sepsis mortality over time.

With its broad, representative sample, this validation study has an unique approach and provides a comprehensive assessment of the accuracy of ICD case identification for sepsis. No preliminary selection of cases according to the presence or absence of relevant ICD codes was applied, minimizing the risk of selection bias [30] compared to previous validation studies, which reported heterogeneous validity of sepsis case identification strategies in administrative data [13]. Our results are consistent with a previous validation study by Iwashyna et al., yielding better positive predictive values for explicit, and higher sensitivity for implicit coding strategies [14]. Profound variability in sepsis incidence and mortality depending on the ICD abstraction strategy was observed in different studies [12, 31]. For implicit coding, higher incidence due to false positives and lower mortality rates result from a missing causal relation between infection and organ dysfunction. Explicit coding strategies identify fewer cases with higher mortality, because sepsis is not always coded properly especially when less severe [32, 33], and relevant organ dysfunctions are not coded as found in our analyses.

It should be clear that a perfect agreement between cases identified from administrative data and from clinical criteria cannot be expected [34]. When using a framework proposed by Angus et al. [34], it is obvious that both approaches have strengths and weaknesses in different domains. Their usefulness depends highly on the intended purpose, with an important role for administrative data based strategies especially in quality improvement [35].

Despite our comprehensive approach, our study has several limitations. Since the DRG-statistics is a nation-wide all-payer inpatient database, the accuracy of coding is likely to vary between hospitals, thus validation results are not be fully generalizable and a transfer of conclusions is only possible to a limited degree. Given specific coding regulations in Germany, results may not be applicable to other countries. Our validation was based on retrospective manual patient chart review rather than on prospective assessment and is therefore limited by the quality of documentation. Clinical diagnosis according to the current sepsis definitions including SIRS criteria as gold standard may miss 8–10% of septic patients as found in Australia [26].

Further multicenter evaluation is needed to obtain comprehensive data on coding quality, allowing better estimates of the national burden of sepsis and the development of a reliable surveillance coding strategy that agrees with clinical case identification.

Supplementary methodological information.

(DOCX)

Click here for additional data file.

Supplementary results.

(DOCX)

Click here for additional data file.