Balancing Enthusiasm for Innovative Technologies with Optimizing Value: An Approach to Adopt New Laboratory Tests for Infectious Diseases Using Bloodstream Infections as Exemplar.

A number of exciting new technologies have emerged to detect infectious diseases with greater accuracy and provide faster times to result in hopes of improving the provision of care and patient outcomes. However, the challenge in evaluating new methods lies not in the technical performance of tests but in (1) defining the specific advantages of new methods over the present gold standards in a practicable way and (2) understanding how advanced technologies will prompt changes in medical and public health decisions. With rising costs to deliver care, enthusiasm for innovative technologies should be balanced with a comprehensive understanding of clinical and laboratory ecosystems and how such factors influence the success or failure of test implementation. Selecting bloodstream infections as an exemplar, we provide a 6-step model for test adoption that will help clinicians and laboratorians better define the value of a new technology specific to their clinical practices.

Over the last 10 years, a great deal of interest has been focused on exciting new technologies to detect infectious diseases with the goal of improving provision of care and patient outcomes. The need for faster and more accurate tests is clearly argued and well accepted. The debate intensifies, however, when we examine whether advanced technologies will meet our clinical needs appreciably to prompt changes in medical and public health decisions. Often, enthusiasm to adopt new test methods outpaces evidence to support their routine use. With rising costs to deliver care, a value-based approach to test adoption is urgently needed, particularly one focused on the laboratory-clinician-patient triad.

Globally, bloodstream infections (BSIs) are a leading cause of morbidity and mortality, and these infections are associated with significant healthcare costs. By market estimates, approximately 1 million individuals in the United States are diagnosed with BSI, costing the healthcare system nearly $1B annually. In response, new diagnostic platforms have been developed to diagnose BSIs faster with the objective of delivering greater clinical value and reducing overall costs.

Along with the technical performance of new technologies, in-depth analyses of clinical and laboratory ecosystems are essential to fully explore potential barriers to optimal implementation. The ecosystem encompasses the whole process from a physician's test selection to how results influence clinical decision making and then improve patient outcomes. Using BSIs as an exemplar, we provide a 6-step model for test adoption that will help clinicians and laboratorians better define the value of a new technology specific to their clinical practices Table 1.

Table 1.

Use of the Six-Step Model for Test Adoption

Step	Assessment
Clearly define the clinical syndrome	What is the disease state? What step(s) in the clinical or diagnostic process would be addressed by the new technology?
Understand current diagnostic standard and identify gaps	What are the strengths and weaknesses of the current test methods? Do gaps exist in our knowledge of the current standard's technical performance or its influence on clinical decisions?
Assess new technology relative to best practices of gold standard	Why is it necessary to perform this new test? When should this test be performed? How does this test improve on current diagnostic methods?
Define potential barriers to optimization	What laboratory resources are necessary to successfully implement the test? How will laboratory or clinical workflow change with the new test? What policies or infrastructure should be in place to optimize the test's value?
Establish that new information changes clinician behavior and benefits patients	How will test results be communicated to clinicians? What is the intended response to the laboratory result? Do clinicians need any special training about how best to interpret the new test results? Does clinical practice of antimicrobial empiricism undermine the value of the new test?
Examine clinical utility of new technology	How will the new test result affect outcomes measures such as patient morbidity/mortality, hospital costs, infection prevention, patient safety, multidrug resistance, etc? Would pathogen or virulence specific, innovative therapeutics optimize the value of the new test?

CLEARLY DEFINE THE CLINICAL SYNDROME BEING TESTED

Although seemingly self-evident, the clinical syndrome being tested needs to be clearly defined to prevent the use of ambiguous case definitions that adversely affect a test's performance. In the case of BSI, its definition historically has been linked to blood culture, and, hence, there is a tacit consensus that when referring to BSI, we think of bacteria and fungi as etiologic agents. We often use the terms BSI and sepsis synonymously. In doing so, we may consider blood cultures falsely negative in patients with sepsis syndrome from a viral infection or systemic inflammatory response syndrome (SIRS) from noninfectious causes [1]. A test's failure to detect microorganisms from blood may be secondary to disease pathogenesis rather than from poor technical performance of a diagnostic test. Hence, when considering test adoption for BSI, we need to clearly define the clinical problem that we intend to solve, and then we need to communicate the requisite performance characteristics of a diagnostic test to meet specific clinical needs. The clinical and diagnostic problems can be multifold including the following: (1) earlier detection and identification of bacterial and fungal pathogens from blood; (2) earlier detection of antimicrobial resistance of BSI pathogens; (3) prompt differentiation among viral, bacterial, mycobacterial, fungal, and parasitic causes of BSI; and (4) prompt differentiation between infectious and noninfectious causes of SIRS/sepsis-like syndromes. No single test can meet all 4 of these criteria, and, for purposes of this viewpoint, we will limit the discussion to tests targeting the first 2 topics.

UNDERSTAND CURRENT DIAGNOSTIC STANDARD AND IDENTIFY SPECIFIC GAPS

For well over a century, blood culture has served as the primary method to diagnose BSIs, and, over the years, automated backend systems have been developed to detect and identify bacterial and fungal pathogens from blood along with their antimicrobial susceptibilities. Reliance on this method is well rooted by the blood culture's performance characteristics and relatively low cost. Blood culture's ability to detect a range of bacterial and fungal loads from as low as <1 colony-forming units (CFU)/mL to >2000 CFU/mL is well known, clinically relevant, and a gold standard for technology assessments [2, 3].

The microorganism load in sepsis is commonly believed to vary with host factors (eg, age, immune status, and comorbidities), pathogen characteristics (eg, species, virulence mechanisms), clinical presentation (eg, septic shock), and source of infection (eg, intravascular catheter, urine). Yet limited research has quantitatively defined the variation of BSI microbial loads in diverse patient populations, and this knowledge gap presents challenges when assessing new technologies. What we do know suggests that new methods must have a lower limit of detection below 1 CFU/mL to be applied to all patient populations and, certainly, before they can replace blood culture.

Many experts claim that up to 30% of blood cultures are falsely negative [4], and they use this figure as supporting evidence when advocating for better tests. However, numerous studies and reviews have shown that the analytical and clinical sensitivity of blood culture is excellent for detecting >95% of viable bacteria and some fungi circulating in blood [5, 6]. Indeed, “viability” remains a limiting factor for performance. Blood culture has decreased sensitivity for those patients with BSI from fastidiously growing and uncultivable microorganisms, nonviable microorganisms from prior antimicrobial therapy, transient infection, and infections from other anatomic sites (eg, hepatosplenic candidiasis). The frequency of these conditions is unknown and presents a diagnostic gap for blood culture performance - how new tests can close this gap remains to be defined.

Publications about new technologies often lead us to believe that positive blood cultures generally do not provide actionable data for up to 3–5 days [7, 8]. However, blood culture and standard methods can provide full organism identification and susceptibility information as early as 10–24 hours without the addition of new and expensive technology. In general, the median time to positivity of blood culture bottles ranges from 8 to 18 hours for Gram-negative bacilli [9, 10], 13 to 21 hours for Gram-positive cocci [10, 11], and 18 to 31 hours for yeast [12, 13]. Methods applied to positive blood culture bottles such as (1) latex agglutination PB2a tests to identify methicillin-resistant Staphylococcus aureus (MRSA) and (2) direct inoculation of positive blood culture broth into automated susceptibility instrumentation and matrix-assisted laser desorption ionization time-of-flight (MALDI-TOF) can generate an identification and susceptibility information from 20 minutes to hours after a bottle flags as positive [14-17] (Table 2).

Table 2.

Comparison of Conventional Blood Culture and Emerging Blood Culture Technologies

Metric	Blood Cultures With Automated Identification Systems and/or MALDI-TOF	Emerging Bloodstream Infection Technologies
Limit of detection	<1 CFU/mL	1–300 CFU/mL
Spectrum of microorganism detection	Broad detection of many, including unexpected pathogens.	FDA-approved tests have targeted methods limiting the number of pathogens identified in the system.
Spectrum of resistance markers	Full susceptibility testing can be performed.	Limited to detection of resistance, not susceptibility.Targeted detection of resistance markers will miss emerging resistance methods.
Time to result	Most blood cultures are positive within the first 10–24 h of incubation and in combination with identification and susceptibility testing, results may be available as early as 1 h (ID from MALDI-TOF) to 6 h (direct susceptibility) from positivity.	Culture-based identification and resistance test methods are limited by same incubation time as conventional blood cultures. Some methods require batching that extends the TAT.In most cases, results will have to be confirmed by standard methods.
False positives	2.5% contamination rate	No indication that contamination rate will substantially improve over conventional blood culture.Unknown how a single positive results of a common contaminant will be addressed.
Ease of use	Performed in laboratories of all sizes and complexities.Highly automated.Work-up only positive blood culture bottles.	Tests from positive blood culture bottle are sample to answer.Tests direct from whole blood often require a high complexity environment and are more labor intensive because every specimen is processed.
Laboratory cost	Less than $100Negative test results (accounting for 90% or more of blood cultures) require little hands on time.	Greater than $100Without policy to screen for high-risk patients, 90% or more tests will be negative.

Abbreviations: CFU, colony-forming units; FDA, US Food and Drug Administration; ID, identification; MALDI-TOF, matrix-assisted laser desorption ionization time-of-flight; TAT, turnaround time.

Why does there continue to be a perceived gap in the time to result from a positive blood culture? Budget-constrained laboratories do not have 24 hours per day coverage to perform on-demand testing, and without changes in hospital resources and priorities, this situation will not significantly change. However, it is important to note that our current blood culture technology (with MALDI-TOF), combined with optimized workflow, is capable of detecting most cases of BSI caused by bacteria and yeast, and most results can be provided within 24 hours of blood collection (frequently as early as 10–15 hours) at an affordable price (<$100.00).

ASSESS NEW TECHNOLOGY RELATIVE TO BEST PRACTICES OF GOLD STANDARD

For BSIs, the key performance metrics (among many) to consider for a new BSI pathogen detection test are lower limit of detection, spectrum of microorganism detection, spectrum of resistance markers, false positives, time to result, and ease of use. The different types of pathogen-directed BSI tests are emerging rapidly, and each have special performance characteristics. The intent of this section is to provide a general, comparative guide that can be used for any test under consideration. For those interested in learning more about the test characteristics of each type of BSI test, we refer the reader to 2 timely review articles [18, 19].

Lower Limit of Detection

Emerging technologies have bifurcated the BSI market with those being applied directly from whole blood and others requiring blood culture enrichment. The reason behind this distinction is mainly due to emerging technologies being unable to reliably detect bacterial and fungal loads below 1 CFU/mL directly from whole blood, and their requirement for a sensitivity boost from culture enhancement. Regulatory-cleared and commercially available technologies requiring blood culture enrichment are mostly nucleic acid tests (NATs) based on fluorescent in situ hybridization, microarray, and multiplex polymerase chain reaction (PCR) detection; however, other methods such as automated microscopy and metabolomics are emerging. Direct from blood platforms are promising, but they too cannot achieve blood culture's lower limit of detection of 1 CFU/mL or less, and, generally speaking, the more universal methods (eg, next-generation sequencing, mass spectrometry) have poor lower limits of detection compared with their multiplex NAT counterparts.

Spectrum of Microorganism Detection

Although it is true that any organism can cause sepsis, 90% of cases of sepsis are caused by a limited number of pathogens. These pathogens are the target of most BSI assays. Both commercial and in-development platforms offer different spectrums of microorganism detection. Most NAT (including in situ hybridization) platforms from blood culture enrichment have limited number of pathogens on their panels (up to 30). Automated microscopy and metabolomic signatures from culture enrichment rival the wide spectrum offered by blood culture. Platforms from whole blood that apply universal primers for amplification such as multiple DNA target sequencing, next-generation sequencing, or PCR electrospray ionization mass spectrometry also can detect a wide spectrum of microorganisms but are limited by other factors.

Spectrum of Resistance Markers

Many molecular platforms include only a handful of resistance targets, or they may fail to detect resistance when unusual mutations are present. In collaboration with antimicrobial stewardship initiatives, resistance marker detection has demonstrated value for reduced antibiotic costs and decreased length of stay. However, detection of resistance only provides a narrow picture of the antibiotic profile of an organism. In addition, as new resistance markers emerge, many NAT assays will be limited in their ability to expand their detection of resistance. For next-generation sequencing and similar platforms, detection of resistance can be comprehensive, but these methods cannot distinguish low level, nonclinically relevant mutations or nonfunctional mechanisms from those with clinical importance. The emergence of automated microscopy techniques to accelerate the process of antimicrobial susceptibility testing shows promise and overcomes the limitations of detection of resistance markers.

Time to Result

It is clear that the main impetus for BSI test development is to dramatically shorten the time between patient presentation and detection of a bloodstream pathogen. Technologies that rely on blood culture enrichment provide results no sooner than the average 10–22 hours of incubation of blood culture bottles, and their advantages over MALDI-TOF are unclear. For most direct from blood platforms, results can generally be provided within 4 to 8 hours from specimen collection under ideal conditions, but they can require batch processing, be labor intensive, need technical expertise to perform, and require result interpretation. Unlike the ease of automated blood culture systems (especially for negative specimens), every specimen tested on molecular platforms requires hands-on processing, often leading to batching for workflow optimization and thereby diminishing the potential value for rapid results.

False Positives

No technology is free from the perils of false-positive results, and these diagnostic errors can have a financial cost, complicate medical decisions, and create greater clinical uncertainty. For molecular-based tests, false positives are caused by many factors such as cross-reactivity of probes and primers or contamination from the environment, reagents, or collection tubes (sterile but not necessarily DNA free). In addition, false positives can arise from improper skin preparation or colonized catheter hubs—a frustrating problem with routine blood culture, resulting in median 2.5% contamination rate [20] that would not necessarily be overcome by the adoption of molecular-based tests.

Ease of Use

Many nucleic acid-based tests from blood culture enrichment have easy-to-use platforms that are attractive to laboratories because they can be run by technicians with no need for result interpretation. Most direct from blood platforms, however, require technical expertise and are not highly automated in the format of sample to answer. Comparatively speaking, blood cultures are highly automated, requiring technologist response for only those bottles that flag positive (approximately 5%–9% of blood cultures). Hence, for a new molecular test, a technologist without prior knowledge of positivity would be required to process every specimen to provide a negative or positive result, increasing the work load to more than 10 times over blood culture.

DEFINE POTENTIAL BARRIERS TO OPTIMIZATION

Without changes in current practices in the management of patients with suspected BSI, many barriers that prevent optimal application of blood cultures and their results could persist with the adoption of novel methods. The first barrier is overutilization of blood cultures. Perhaps due to the perception (often erroneous) that blood cultures are associated with low costs and cause no patient harm, physicians increasingly order blood culture tests even when the likelihood of BSI is low. In the absence of a rational use policy, adoption of new tests, particularly those used directly from blood, could be costly to patients and present workflow challenges for laboratories.

Second, blood culture has been plagued by contamination rates, which have remained consistent over the last few decades despite routine phlebotomy training, better antiseptic technique, and improved collection procedures from indwelling intravascular devices. Adoption of molecular tests would not necessarily overcome this problem, contributing to diagnostic uncertainty.

Third, staffing of microbiology laboratories is often not 24 hours a day, 7 days a week, 365 days a year due to personnel and budgetary constraints. Historically, benefits of continuous coverage of a microbiology laboratory have been difficult to demonstrate mainly due to the lack of supportive infrastructure that facilitates communication of test results to clinical decision makers.

Finally, careful review of literature has shown that over the last 30 years, care models similar to today's antimicrobial stewardship programs and bioinformatics solutions have been recommended to effectively communicate blood culture results that would foster meaningful changes in management decisions [21-23]. Overall, novel tests alone cannot overcome these clinical and laboratory processes, and without policy changes and infrastructure improvements, these barriers will undermine the value of new BSI technologies.

ESTABLISH THAT NEW INFORMATION CHANGES CLINICIAN BEHAVIOR AND BENEFITS PATIENTS

Decisions to adopt tests with faster and better results should be guided by evidence demonstrating that test results influence treatment decisions (ie, change behavior) and improve outcomes. For BSI, most authors have demonstrated that decreasing the time to result reduces mortality and costs [24, 25], but there are some that find no impact on outcome with faster times to result [26]. Exploration of such conflicting evidence is important, and psychological, sociological, and cultural influences and biases may provide clues.

A reduction of the indiscriminate use of broad-spectrum antimicrobial therapy may be a shared objective among all stakeholders assessing a new technology, yet sometimes we fail to fully understand how information changes clinician behavior that would reduce their use. For instance, in certain clinical contexts, broader antimicrobial coverage may be necessary due to complicating conditions (eg, decubitus ulcer), and a physician will not change therapy regardless of test result. In addition, the value of a new test (ie, prompting therapeutic changes) can be diminished by the widespread practice of antimicrobial empiricism. Authors recently examined the drivers of empirical treatment for tuberculosis and postulated the need to fully characterize the practice of empirical treatment before adequately estimating the value of new diagnostic tests [27]. Many parallel arguments exist for BSIs. As part of our test assessment, we should examine the triggers to prescribe empirical antimicrobials, define the clinical contexts when therapeutic empiricism fails, and evaluate the timing and type of test result required to change management decisions that actually improve outcomes [28]. Our ingrained practice patterns for broad-spectrum antibiotics may persist from lack of alternative pathogen-specific treatments. In fact, the development of new classes of therapeutics that are pathogen- or virulence factor-specific or modulate specific host inflammatory responses may reframe the discussion about the diagnostic value of new technologies. Perhaps the push for novel diagnostics is a bit premature and is not being adequately counterbalanced by a pull from innovative pathogen-specific therapy. This therapeutic gap may potentially undermine the value of any rapid diagnostic test for pathogen detection.

In summary, fast and better detection of a pathogen may not change physician behavior and prescribing practices as dramatically as we hoped because we do not currently have a full understanding of the drivers in clinical decision making. Indeed, without any change in testing strategy, improved patient outcomes and reduction in antibiotic use have been demonstrated when effective communication channels between infectious diseases specialists and care teams were implemented as the singular intervention [29]. Finally, as stakeholders, we will need to factor the amount of time that physicians (and pharmacists) will need to gain experience and confidence in the performance of new tests in different clinical contexts before such tests meaningfully change practice patterns.

EXAMINE CLINICAL UTILITY OF NEW TECHNOLOGY

Comparing the performance characteristics of blood culture with new platforms may generate several questions. (1) Are lower limits of detection of 1 CFU/mL or less necessary for most BSI? (2) Does a rapid diagnostic BSI test that detects a limited numbers of pathogens provide a more alternative value than those tests providing more comprehensive identification and susceptibility information? (3) At what point does speed eclipse performance characteristics and cost?

A hypothetical test that rapidly detects MRSA bacteremia within 3–4 hours of clinical presentation using an assay with a lower limit of detection at 40 CFU/mL could have high clinical value when positive. To realize the clinical value, this scenario would also require ideal laboratory operations (24 hours a day, 7 days a week), immediate and effective result reporting, and an uncomplicated clinical context prompting definitive changes in therapy. Although it is easy to assess the clinical utility of a positive result, a negative result presents many more challenges. For reasons of limits of detection, inclusivity of target pathogens, or both, a negative molecular test result would not necessarily lead to changes in care because it cannot definitively exclude infection, greatly diminishing the clinical utility of the test. A good screening method to identify higher risk patients before processing a rapid BSI test could potentially improve the value proposition of testing directly from whole blood.

Despite our knowledge about the performance characteristics of new diagnostic tests, we do not have sufficient data to completely assess their clinical utility. To date, we still do not know the following: (1) the relative frequency of BSI at lower limits of detection of 1 CFU/mL or less for each clinical context; (2) the comparative values of limited and comprehensive rapid diagnostic BSI tests with different levels of technical performance, time to results, and ease of use; (3) the influence of 24 hours a day, 7 days a week laboratory operations and decision support coverage; (4) the point at which the benefit of a shortened time to result becomes null, particularly in the context of widespread use of empiric broad-spectrum antimicrobial therapy; and (5) programmatic requirements to influence behavioral changes in clinical decision making. More importantly, when we turn to the peer-reviewed literature for answers, we need to exercise the utmost rigor because studies between new and gold standard tests often compare poorly optimized laboratory testing and reporting workflow of current tests with optimal testing conditions of the new test. We acknowledge that there are challenges to obtaining robust, prospective BSI clinical trial data that may include difficulties in patient accrual, poor representation of all potential pathogens (in absence of contrived specimens), logistical barriers, and high costs. We propose that implementation of screening methods to identify high-risk patients for BSI, streamlined patient consent processes, universal acceptance of ethics review protocols, use of interoperable, relational databases with electronic medical records (necessary to measure outcomes on laboratory workflow, patient safety, infection prevention, cost, morbidity/mortality, multidrug resistance, etc), and multinational/multicenter participation will greatly enhance our abilities to generate data to support clinical utility of new BSI tests.

RATIONAL APPROACH TO ADOPTION

The need for better and faster tests to diagnose BSI appears to make sense, especially for many of us caring for critically ill patients with SIRS or septic shock. We widely cite the seminal article by Kumar et al [30] in 2006 to support our belief that mortality rises with every hour delay of effective therapy. It is interesting to note that, in contrast to Kumar et al's [30] findings, a study in 2012 found that delaying antimicrobial treatment in critically ill patients with suspected hospital-acquired infection did not worsen mortality [31].

After a decade of innovation for diagnosing BSIs, we have realized that culture-based testing set an extraordinarily high standard in its ability to detect, identify, and provide susceptibility information on a myriad of microorganisms and therapeutic agents. Although novel and commercially available diagnostic platforms are increasing the speed at which organisms are identified in patients with BSIs, these technologies have not replaced existing blood culture tests in the laboratory but, rather, are adopted as adjunctive tools to standard blood culture. We are slowly learning that improvements in technologies alone have only limited influence on altering clinical practice patterns and improving outcomes. When adopting new tests, these times call for balanced and critical thinking that should include a rigorous examination of how we make clinical decisions and how we apply diagnostic methods to achieve their fullest potential. We submit that an entire healthcare ecosystem needs to be addressed to optimize a test's value, and we look forward to reading published research of mathematical models or prospective, multicenter comparative effectiveness studies that close our knowledge gap and lead to more effective assessments of diagnostic technologies in the future.