Blood Cultures and Molecular Diagnostics in Intensive Care Units to Diagnose Sepsis: A Bayesian Latent Class Model Analysis.

BACKGROUND: Confirmation of sepsis by standard blood cultures (STD) is often inconclusive due to slow growth and low positivity. Molecular diagnostics (MOL) are faster and may have higher positivity, but test performance can be inaccurately estimated if STD methods are used as comparators. Bayesian latent class models (LCMs) can evaluate diagnostic methods when there is no "gold standard." Intensive care unit studies that have used LCMs to combine and compare STD and MOL method performance and estimate the prevalence of sepsis have not been described. PATIENTS AND METHODS: Results from an ICU sepsis study that used both tests simultaneously were analyzed. Bayesian LCMs combined prior prevalence of sepsis, prior diagnostic characteristics of the two methods, and the study results to estimate the posterior prevalence and diagnostic characteristics. Sensitivity analyses were performed using objective (published studies) and subjective (expert opinion) prior parameters. Positive predictive values (PPVs) of the prevalence of sepsis were estimated for all combinations of test results.
RESULTS: The range of posterior estimates was: sepsis prevalence (0.38-0.88), sensitivities (STD: 0.2-0.35, MOL: 0.56-0.86), and specificities (STD: 0.87-0.99, MOL: 0.72-0.95). The PPV (sepsis) of both tests being positive was (0.72-0.99).
CONCLUSION: LCMs combined two imperfect methods to estimate prevalence, PPV, and diagnostic characteristics. The posterior estimates (STD sensitivity < MOL and STD specificity > MOL) seem to reflect the clinical experience appropriately. The high PPV when both methods show positive results can be useful for ruling in disease. HOW TO CITE THIS ARTICLE: Sampath S, Baby J, Krishna B, Dendukuri N, Thomas T. Blood Cultures and Molecular Diagnostics in Intensive Care Units to Diagnose Sepsis: A Bayesian Latent Class Model Analysis. Indian J Crit Care Med 2021;25(12):1402-1407.

Highlights

Sensitivity and specificity of diagnostic tests and the prevalence of sepsis can be estimated even in the absence of a reference test with Bayesian LCM. The utility of combining the results of two diagnostic tests and estimating a PPV is a unique feature of this method.

Introduction

Sepsis and antimicrobial resistance (AMR) are among the major challenges faced by intensive care unit (ICU) clinicians, with 48 million cases and 10 million deaths attributable annually to sepsis worldwide.[1] Sepsis and AMR management programs can appear to have contradictory messages with respect to antibiotic prescription but are actually addressing the two perspectives of the same issue.[2] The lack of rapid and reliable diagnostic tests leading to inappropriate antibiotic usage are among the causes of morbidity in patients with sepsis and in patients with infections due to pathogens resistant to the commonly used drugs.

The recent surviving sepsis guidelines recommend obtaining blood cultures within 1 hour of presentation to the hospital.[3] Blood cultures are essential for pathogen identification and selection of appropriate antibiotics. Standard microbiological blood cultures (STD) can have variable yields, long turnaround times, and low sensitivity, which contribute to inappropriate antibiotic therapy.[4] Molecular diagnostic methods (MOL) are alternatives that provide rapid results, but false positivity due to contamination with extraneous microbial DNA can be a limitation.[5] A blood culture is thus still considered as the “gold standard” test in the detection of bloodstream pathogens despite its low clinical positivity. Sepsis is defined as an organ dysfunction caused by the host response to an infection. The presence of infection and the severity of the host response are two distinct components that have to be combined to establish a diagnosis of sepsis.[6] There is at present no standard diagnostic test to define sepsis.[7] The “true” prevalence of sepsis is thus unknown (latent) and is a matter of clinical conjecture. In the absence of a good reference test, the comparative performance of newer MOL tests as well as disease prevalence may be inaccurately estimated by standard methods which assume that the reference test has the perfect sensitivity and specificity.[8] Latent class models (LCMs) extend standard methods by modeling the sensitivity and specificity of the reference test, in other words acknowledging that it is not a perfect test.[9] Bayesian methods to estimate disease prevalence and compare the parameters of diagnostic methods in the absence of a gold standard were first described in a seminal paper by Joseph et al.[10]

The performance of MOL methods against blood cultures by STD methods in an ICU has been compared using standard statistical analyses. These comparisons have assumed that STD methods have perfect accuracy, but the “true/latent” prevalence of sepsis and the performance characteristics of the blood culture tests themselves have not been assessed.[11] To the best of our knowledge, no ICU study has been published to date using LCMs to compare MOL and STD methods for the diagnosis of sepsis from blood cultures.

The aim of this study is to estimate the prevalence of sepsis and to assess the diagnostic characteristics (sensitivity and specificity) of STD and MOL tests independently and in combination, by using Bayesian LCMs in ICU patients suspected to have sepsis.

Methods

We extracted reports from an existing study (data on file) in which STD and MOL tests had been collected. This study had recruited patients from the ICU of St John's Medical College, Bengaluru, from April 2010 to September 2010 after institutional ethical committee approval (IEC: Ref-188/2008). Further consent for the performance of this study was also obtained (IEC: Ref-002/2020). In the original study, from ICU patients thought to have clinical sepsis either at admission to ICU or later, blood samples were drawn and simultaneously tested by STD and MOL methods.

Sample Collection

Blood samples were only drawn from a central venous cannula (CVC) by the operator at the time of their insertion for the management of sepsis. This was done as CVCs are inserted in ICU under sterile conditions following standard protocol, and optimal peripheral samples in ICU patients are often difficult to obtain. Samples taken after admission were from patients where CVCs were freshly sited as part of the management of new-onset sepsis.

The recommended volume of blood of 5–10 mL was collected and injected into the BacT/ALERT FA PLUS® (bioMérieux, Marcy l’Étoile, France) bottles after decontaminating the top of the bottle with alcohol. The bottle was transported immediately to the lab for incubation and processing. Another 5 mL of blood was collected in an EDTA vacutainer and sent for MOL testing.

STD methods employed aerobic cultures using the BacT/ALERT® (bioMérieux, Marcy l’Étoile, France) system. MOL methods were tested by the syndrome evaluation system (SES) (Dhanvantri Xcyton Diagnostic Private Limited, Mumbai). The tests were done in separate laboratories by staff who were unaware of the comparator method result. Clinical details requested in blood culture forms were available to performers of both the tests. The techniques of both procedures have been detailed in the studies of Altun et al. and Bhat et al.[12,13] Bacterial isolates from each of the methods were recorded. Patient characteristics and outcomes were extracted from the records.

Statistical Methods

Descriptive statistics were calculated to describe the sample. The association of severity of illness scores and the STD and MOL test results, with deaths in the ICU, was assessed by logistic regression techniques.

The Bayesian analysis was conducted and reported following standard (STARD-BLCM) guidelines.[14] A Bayesian approach to inference for the LCM was necessary in this context because the model was not identifiable, i.e., there was insufficient information in the observed data to estimate all the parameters of interest. As both STD and MOL are imperfect tests, there is a possibility that they may make false-positive and false-negative errors simultaneously, a phenomenon referred to as conditional dependence. To adjust for conditional dependence, we used the fixed effects model proposed by Dendukuri and Joseph.[9]

An informative prior distribution on at least some parameters was necessary to obtain a meaningful posterior distribution. We considered two plausible sources of prior information. Objective prior information on the prevalence of sepsis, sensitivity, and specificity of STD and MOL methods was obtained from the published literature in a study in which the prevalence of sepsis had been clinically determined.[11] Subjective priors for the same parameters were elicited separately from multiple experts in intensive care. The estimates from these sources were combined using the methods proposed by Moitra to generate prior distributions.[15] As a result of the nonidentifiability of the problem, we expect the posterior distributions to be highly sensitive to the prior distributions. Sensitivity of the posterior estimates to changes in the informative prior distributions (for STD and MOL sensitivity and specificity) was examined to compare objective priors from clinical studies (model A) and subjective priors suggested by experts in critical care (model B). In both models, a noninformative beta (1,1) prior was used for the prevalence of sepsis which reflected the clinical uncertainty in the diagnosis of sepsis and wide variations in prevalence estimates.[16]

Applying Bayes’ theorem, we obtained posterior distributions for the prevalence of bacterial sepsis and diagnostic accuracy of the two methods, by updating these prior distributions with data from the present study. Bayesian estimation was implemented using a Monte Carlo Markov Chain (MCMC) approach, resulting in a posterior sample for all statistics of interest (prevalence, sensitivity, specificity, and positive predictive value [PPV]).

Standard statistical analyses were performed using STATA v14 (StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP.). The Bayesian LCM was performed using R: [R Core Team (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, URL: http://www.R-project.org/] and WinBUGS version 1.4 (http://www.mrc-bsu.cam.ac.uk/bugs/). The beta distributions of the priors and programs for reproducing these results are presented in the supplementary material. The MCMC method was implemented by discarding 5,000 burn-in iterations, following which 45,000 iterations were retained. Convergence was assessed visually and by using the Gelman–Rubin statistic.

Results

Three hundred ninety-three blood samples were tested by both the methods, and after excluding 14 samples identified as repeats of the same subjects, 379 samples were analyzed. The median age was 50 years interquartile range (IQR) (31–65); there were 129 females and 250 males. The mean APACHE II score was 13, SD (10), median ICU length of stay was 6 days IQR (3–12), and the ICU mortality rate was 28.7%. The demographic patterns, severity of illness scores, and outcomes were similar across patients who were positive only in one or the other of the diagnostic methods.

The proportion of positive isolates by the MOL method (50%) was significantly higher than that by STD method (18%), (McNemar's exact = 32, p <0.001) (Table 1). Forty-nine (13%) samples were positive by both methods, 171 (45%) were negative by both methods for any bacterial isolate, and 159 (42%) samples showed discordant results, of which 141 (37%) were positive by MOL methods only and 18 (5%) by STD methods only (Table 1). The agreement between the two methods was small (kappa = 0.16).

Table 1

Cross-tabulation of results of both tests

Bacterial isolate results	STD–positive	STD–negative	Difference in proportions (McNemar's exact–p)
MOL–Positive	49 (13%)	141 (37%)	32%, p <0.01
MOL–Negative	18 (5%)	171 (45%)

STD, standard blood culture methods; MOL, molecular methods

The prior information ranges obtained following objective (A) and subjective (B) approaches were very different (Table 2). In both cases, the sensitivity of STD was inferior to MOL while the specificity of STD was superior to MOL. However, the median values were very different. The uncertainty range following the subjective approach was much greater.

Table 2

Posterior prevalence, sensitivity, and specificity of both methods

			Sensitivity median (CI)		Specificity median (CI)
Model	Estimates	Prevalence median (CI)	STD	MOL	STD	MOL
A	Prior		0.16 (0.03–0.4)	0.32 (0.1–0.63)	0.98 (0.94–1)	0.95 (0.87–0.98)
	Posterior (IN)	0.69 (0.58–0.84)	0.24 (0.18–0.31)	0.68 (0.56–0.8)	0.99 (0.95–1)	0.95 (0.86–0.99)
	Posterior (DE)	0.88 (0.67–0.99)	0.2 (0.15–0.27)	0.56 (0.47–0.71)	0.98 (0.94–1)	0.95 (0.86–0.99)
B	Prior		0.39 (0.09–0.75)	0.68 (0.13–0.98)	0.87 (0.7–0.97)	0.76 (0.44–0.93)
	Posterior (IN)	0.69 (0.58–0.84)	0.35 (0.23–0.66)	0.86 (0.68–0.99)	0.92 (0.87–0.97)	0.72 (0.56–0.92)
	Posterior (DE)	0.88 (0.67–0.99)	0.25 (0.12–0.61)	0.7 (0.38–0.98)	0.87 (0.76–0.96)	0.66 (0.46–0.91)

STD, standard blood culture methods; MOL, molecular methods; CI, credible intervals (2.5–97.5%); IN, conditional independence assumption; DE, conditional dependence assumption

The posterior estimates of the prevalence of bacterial sepsis and diagnostic accuracy of the tests are shown in Table 2 and Figures 1 and 2. The prevalence estimated by model B (0.38–0.43) is much lower than the estimates (0.69–0.88) of model A, and in both these models, the conditional dependence model had a higher estimate of prevalence when compared with the conditional independence model. The posterior sensitivity of STD methods is low (0.2–0.35) in both models A and B. The posterior specificity of STD methods is uniformly very high (0.87–0.99) in all models irrespective of assumptions. In Figure 1, it can be seen that posterior estimates of model A have a narrow range as compared to model B even when the median values are similar as in posterior estimates of STD specificity. The posterior MOL sensitivities in both models A and B with any assumptions are higher than the corresponding STD sensitivities.

Fig. 1

Graphical representation of prior and posterior estimates of sensitivity and specificity of the two diagnostic tests. A—Model A, B—Model B, X-axis: probability of parameter, Y-axis: density of parameter. Priors: — line, posterior independence assumptions: — — line, posterior conditional dependence assumption: — · · —line

Figs 2A and B

Graphical representation of the prevalence of sepsis and PPV (positive predictive value) when both tests are positive. A—Model A, B—Model B, X-axis: probability of parameter, Y-axis: density of parameter. Priors: — line, posterior independence assumptions: — — line, posterior conditional dependence assumption: — · · —line

The posterior PPVs for the prevalence of bacterial sepsis given all combinations of the results of both diagnostic tests are shown in Table 3 and Figure 2. The PPVs of both tests being positive are high (0.72–0.99) across all models and assumptions. The PPV in model B with the conditional dependence assumption is the least (0.72) and has the widest range (0.09–0.98) reflecting the uncertainty in the prior information. The PPV of either test being positive in isolation or both tests being negative is far higher for model A when compared to the corresponding assumption in model B, owing to the higher prevalence estimates under model A. In a logistic model to predict ICU mortality which included the APACHE II score, only the addition of both diagnostic methods being positive for microbial isolates as a covariate was significant [Wald test, Chi-square (10.7) p = 0.001] when compared to the addition of other combinations of test results.

Table 3

Positive predictive values of the combination of test results

		Positive predictive value–median (CI)
Model	Posterior estimates	STD and MOL +ve	Std +ve and MOL –ve	STD –ve and MOL +ve	STD and MOL –ve
A	IN	0.99 (0.98–1)	0.93 (0.73–1)	0.96 (0.87–0.99)	0.37 (0.2–0.66)
	DE	0.99 (0.95–1)	0.99 (0.83–1)	0.99 (0.92–1)	0.75 (0.34–0.99)
B	IN	0.9 (0.68–0.99)	0.34 (0.02–0.84)	0.58 (0.13–0.91)	0.06 (0.002–0.3)
	DE	0.72 (0.09–0.98)	0.36 (0.004–0.98)	0.61 (0.02–0.99)	0.2 (0.006–0.87)

STD, standard blood culture methods; MOL, molecular methods; CI, credible intervals (2.5–97.5%); IN, conditional independence assumption; DE, conditional dependence assumption

Discussion

This Bayesian LCM study has shown how results of two imperfect diagnostic methods can be combined to estimate the prevalence of sepsis, relying on prior information on sensitivities and specificities of each test. We found that the results were very sensitive to the choice of prior information. Nonetheless, the estimated PPV when the results of both methods of blood culture are positive is clinically meaningful.

Sepsis cannot be defined by a validated standard diagnostic test and is recognized by organ dysfunction caused by a dysregulated host response to infection.[7] The diagnosis of infection and its causal relation to organ dysfunction is often uncertain, and up to 40% of ICU patients with a diagnosis of sepsis may not have infection and sepsis.[16] The mismatch between published sensitivities and specificities of blood culture by STD methods (0.98–1), blood culture positivity rates of only 25%, and contamination rates as high as 50% suggest that STD blood culture methods are not ideal markers of infection or sepsis.[12,17,18] In contrast, the prevalence and posterior sensitivities of STD methods estimated by models A and B seem clinically appropriate. In a recent study, 70% of ICU patients were getting at least one antibiotic, the prevalence of proven/suspected infection was 0.54, and the blood culture positivity rate was 15%.[19] This discrepancy between therapeutic, clinical, and diagnostic laboratory classifications can possibly be decreased by combining two tests to both improve diagnostic accuracy and clinical classification. This could then provide pathways for improvement in the management of sepsis and possibly AMR. We have shown that combining the results of two imperfect tests using an LCM can produce meaningful estimates of diagnostic test characteristics and prevalence of sepsis.

A review has reported that MOL methods have a higher specificity as compared to sensitivity and thus have a rule-in value for the early detection of sepsis.[20] This review included studies that used clinical criteria, other microbiological data, or blood culture (STD) as a comparator, and the authors had pointed out that incorporation bias was likely. A meta-analysis by Dark et al. has compared MOL and STD methods and has explicitly used the blood culture (STD) as a reference standard and concluded that MOL methods seem to have higher specificity than sensitivity.[21] The results of our study do not strongly support the conclusions of these two reviews. The methodological issues underlying such types of analyses and the basis of these assumptions have been addressed in a paper by Dendukuri and Joseph.[9] The assumptions that STD and MOL methods are independent estimators of infection are unlikely, as both of these methods are estimating a common biological metric: the presence of bacteria/bacterial components. These studies do not address the central issue of assessing diagnostic accuracy when there is no gold standard or if the gold standard is imperfect. The use of blood culture as a reference standard by these reviews may lead to inaccurate estimates, and Bayesian LCMs used in this study can provide unbiased estimates. The sensitivity and specificity of diagnostic tests are not inherent and can vary with population characteristics.[22] The high sensitivity and specificity (>95–98%) of STD methods are rarely reflected in clinical settings.[19] The low posterior sensitivity and high specificity of STD methods and MOL sensitivity being much higher than STD sensitivity in both models A and B seem to reflect clinical reality more appropriately than published sensitivities and specificities.[12,17,23,24]

The sensitivity and specificity of diagnostic tests are useful clinically when combined with estimates of the prevalence of sepsis and test results to calculate the PPVs of the test results. The Bayesian posterior estimates of prevalence, sensitivity, and specificity of both tests lead to the direct estimation of PPV in our study and do not need any external inputs. We have estimated PPVs of combinations of both STD and MOL tests across all models and assumptions. The PPV of both methods being positive ranges from 0.72 to 0.99 by all models and assumptions suggests that this combination of results would be more suitable to rule in sepsis. Both in models A and B, the PPV of MOL tests being positive in isolation are quite high and more than the PPV of STD tests being positive in isolation suggesting a clinical utility of MOL tests in diagnosing sepsis. The addition of rapid diagnostic tests to STD tests has been suggested as a part of antibiotic stewardship but how to combine the information has not been formally addressed.[25] The Bayesian results of our study provide an intuitive understanding; the posterior estimates show that when both methods are positive the PPV is high (ruling in) as is seen in all the models. In a recent ICU study using molecular diagnostics, it has been claimed that the negative predictive value of the test is 99.5%, and clinicians would be comfortable withholding antibiotics based on such a report.[26] These predictions may be optimistic as the specificity was calculated assuming STD methods as a gold standard, and the prevalence had not been taken into account. Additionally, in models A and B (conditional dependence assumption), the PPV when both tests are negative is in excess of 0.2 (0.2–0.75), suggesting that sepsis cannot be excluded even with both tests being negative.

In our study, the mortality was significantly related only to the positivity of both tests unlike the study by O'Dwyer et al., where the presence of microbial DNA itself was associated with increased mortality.[27] However, in that study, the multivariate modeling p value (0.04) was very modest and did not support a significant contribution by the presence of microbial DNA.

A limitation of this study was that blood was collected from a central line. We felt that contamination rates in ICU would be minimized when collecting blood only at the time of insertion of CVC under optimal sterile conditions.[18] Conventional blood cultures have been standardized, while MOL methods differ in their laboratory implementations.[5] The results of our study cannot be generalized to other MOL methods. The posterior estimates are dependent on the chosen prior values and can be easily seen as the estimates of model A differ from those of model B. This could arguably be the strength of the Bayesian method where clinical subjectivity with respect to priors could be transparently combined with local data. Clinicians are natural Bayesians: diagnostic tests are neither ordered or interpreted with clinical agnosticism. Clinicians have the ability to synthesize clinical information and analytical laboratory tests to decide on therapy.[28] A review of bacterial sepsis has suggested that methods utilizing both the severity and diagnosis of infection may be a step forward.[6] The next step forward would be a prospective study similar to the study of Bloos, in which the utility of using multiple diagnostic methods combined with clinical definitions could be assessed using Bayesian methodology.[11]

Conclusion

The Bayesian LCM was able to combine the prior prevalence of sepsis and diagnostic characteristics of the two tests with the results of two blood culture methods to provide meaningful posterior estimates of the prevalence of sepsis and diagnostic characteristics of the tests. The high PPV when both test results are positive suggests its use as a “rule-in” utility.

Supplementary Material

All the supplementary material from Supplementary tables 1 and 2 are available online on the website of www.IJCCM.org

Orcid

Sriram Sampath https://orcid.org/0000-0001-7521-7106

Jeswin Baby https://orcid.org/0000-0003-0299-3619

Bhuvana Krishna https://orcid.org/0000-0002-0003-6797

Nandini Dendukuri https://orcid.org/0000-0002-2330-0976

Tinku Thomas https://orcid.org/0000-0002-1786-6076