The value and validation of broad spectrum biosensors for diagnosis and biodefense.

Broad spectrum biosensors capable of identifying diverse organisms are transitioning from the realm of research into the clinic. These technologies simultaneously capture signals from a wide variety of biological entities using universal processes. Specific organisms are then identified through bioinformatic signature-matching processes. This is in contrast to currently accepted molecular diagnostic technologies, which utilize unique reagents and processes to detect each organism of interest. This paradigm shift greatly increases the breadth of molecular diagnostic tools with little increase in biochemical complexity, enabling simultaneous diagnostic, epidemiologic, and biothreat surveillance capabilities at the point of care. This, in turn, offers the promise of increased biosecurity and better antimicrobial stewardship. Efficient realization of these potential gains will require novel regulatory paradigms reflective of the generalized, information-based nature of these assays, allowing extension of empirical data obtained from readily available organisms to support broader reporting of rare, difficult to culture, or extremely hazardous organisms.

Introduction

Emerging technologies based on bioinformatic analysis of genetic signatures, most of which are enabled by conserved site PCR, can detect and identify diverse microorganisms. Examples include 16S ribosomal gene sequencing, internal transcribed spacer (ITS) sequencing, PCR followed by electrospray ionization mass spectrometry analysis of base composition (PCR/ESI-MS), PCR followed by amplicon fragmentation and mass spectrometry analysis of fragment composition (MALDI-TOF MS), random or specific PCR followed by resequencing microarray analysis,, PCR amplification followed by single molecule sequencing,, and amplification-independent sequencing (metagenomic sequencing).- These technologies can characterize the bacterial, fungal, and viral components in many sample types without need for culture. If widely deployed in primary care facilities for diagnosis, these broad spectrum biosensor technologies could simultaneously identify clinically relevant etiological agents of infectious illness and rapidly detect biothreat agents and emerging pathogens, providing both diagnostic and biodefense capabilities. While stand-alone biodefense surveillance technologies may suffer from both funding exhaustion, as a result of yielding almost exclusively negative results, and geospatial limitations resulting from limited distribution and often static sampling capability, broad spectrum biosensors useful for point-of-care diagnosis would be used continuously and reach across the exposure area of the population using the health care facilities in which they were operated.

Broad spectrum biosensors are defined here as systems able to detect any member of a broad group of related organisms using nonspecific reagents and a standardized information acquisition and processing algorithm. A simple example is a PCR-based system that amplifies conserved bacterial loci such as 16S ribosomal genes, sequences the amplicons, and uses informatics-based signature analysis such as BLAST to provide identification. In terms of sensitivity and breadth of coverage, such systems are limited only by the ability of the chosen PCR primers to amplify detectable fragments from genetically diverse targets.,, The post-amplification signature analysis algorithms of such methods are generally able to assign membership of the detected organism to a particular genus, group, species, or strain, whether that organism is a common pathogen such as Streptococcus pneumoniae or a biothreat agent such as Bacillus anthracis. These tests can be conceptualized as being single tests with respect to detection but essentially infinite with respect to identification, limited only by the presence of appropriate database signatures allowing definitive bioinformatic matching. This offers a fundamentally different testing modality than current molecular biology standards.

Broad spectrum biosensors are general—they use the same small set of reagents, primers, and bioinformatic tools to detect and identify all organisms within their designed breadth of coverage. They are most appropriately and efficiently validated in a general fashion using representative species from across the breadth of coverage to determine the range and variance of overall detection sensitivity and accuracy of identification.

The Individual and Social Benefits of Broad Spectrum Biosensors

Broad spectrum biosensors have the potential to revolutionize modern medicine. As diagnostic devices, these instruments can improve treatment at the point of care and facilitate antimicrobial stewardship (antibiotic sparing) by allowing the rapid detection and identification of diverse pathogens,, whether they are cultureable or uncultureable and whether they are fully viable or inhibited by antibiotics., As medical countermeasures, they can enable biosecurity efforts by rapidly detecting diverse biowarfare agents.- As biosurveillance devices, they can serve as a source of critical information for public health agencies tasked with detecting, tracking, and characterizing emerging pathogens and epidemic agents.,, Moreover, they are capable of performing all of these activities simultaneously.,, The integration of diagnostics, public health epidemiology, and medical countermeasures activities may provide the only avenue by which the latter two activities can be performed with the coverage and temporal efficiency necessary for timely and effective reactions to emerging pandemics and bioagent attacks.

Modeling efforts suggest that emerging infectious agents must be detected and identified near the initial source of human infection within 24 to 48 h of the first case in order for quarantine or mass prophylaxis efforts to successfully prevent pandemic spread. Effective response times for bioagent attack events are likely even shorter, as most exposures happen simultaneously at the time of release. Most patients infected with bioagents or emerging pandemic agents will initially present to clinics with the symptoms of common respiratory, septic, or gastrointestinal illnesses., If the technological platforms used for diagnosis of common illnesses at hospitals and clinics could identify both common and rare agents, the likelihood of capturing initial cases and thus identifying events early enough to allow an effective response would be greatly increased.

The information provided by broad spectrum methods could be very useful at the point of care. Inclusive molecular testing would allow physicians to efficiently identify the diverse bacteria and fungi associated with common conditions such as bacteremia. Accurate and rapid identification, in turn, can help identify the organ or body compartment acting as the source of bacteremic conditions, thereby more quickly addressing the root of the problem rather than simply treating the proximal symptoms. Similarly, rapid identification of pneumonia and bacteremia agents can improve the efficacy of antibiotic treatment. The benefits of rapid identification of pathogens to the patient have been shown to include reduced mortality across a broad spectrum of infectious disease-associated conditions and decreased use of broad spectrum antibiotics.- For indications such as bacteremia and respiratory illness, the attendant symptoms may represent either routine illness or the earliest and most treatable stages of illness following exposure to a biothreat agent such as anthrax or an epidemic agent such as avian influenza. The use of diagnostic tools which are also capable of identifying these agents could provide invaluable indications of an attack or incipient epidemic,, serving both the immediate patients and those that might benefit from ensuing public health responses and exposure control efforts. The sterile or semi-sterile sample types associated with such indications (blood, bronchoalveolar lavage, etc.) are appropriate for universal biosensors designed to detect and identify one or a few organisms at a time, whether they be common pathogens, emerging pathogens, or biowarfare agents.

For nonsterile sample types, background contamination or commensal carriage of organisms may pose a challenge for truly universal technologies sensitive to entire families or kingdoms of organisms, such as 16S ribosomal sequencing or PCR/ESI-MS assays targeted to all bacteria or all fungi. Since broad spectrum biosensors rely on the same set of PCR primers to amplify homologous genetic elements of all targets, background contamination can generate competitive interference that obviates detection and identification of less concentrated organisms. For nonsterile samples, it may be necessary to utilize more specific biosensors targeted specifically at groups of highly pathogenic organisms or biowarfare agents,,, universal biosensors with highly sensitive organism-specific detection capabilities, or amplification-independent technologies such as metagenomic sequencing, all of which may currently be too laborious and time-consuming for routine diagnostics but are able to detect and identify both highly concentrated and rare minority components in samples with complex bioburden, like throat swabs.

Current Validation Paradigms Cannot Effectively Enable Broad Spectrum Biosensor Technology

Current molecular diagnostic methods define the value of a test on the basis of the method’s ability to detect and identify individual pathogens (analytes). Existing regulatory paradigms designed for such assays require exhaustive analytical and clinical testing for each reportable species.- These types of validations are reasonable and effective for equipment and methods that rely on unique reagents and processes to detect and identify each intended analyte, but they cannot be used to fully enable broad spectrum assays which may have hundreds or thousands of potentially identifiable targets.

We argue that broad spectrum biosensors can be characterized in a general fashion by testing representative analytes. The following sections outline a conceptual framework within which broad spectrum technologies might be effectively validated without overburdening developers, while providing appropriate support for the accuracy and precision of the resulting information.

Definition of Sensitivity and Specificity of Broad Spectrum Biosensors

Many broad spectrum nucleic acid-based approaches use PCR primers targeting highly conserved regions such as rRNA gene to amplify homologous sequences from related microorganisms. In essence, the detection step of a broad spectrum PCR-based biosensor is a single test for many “strains” of the same thing. Much like a single traditional PCR test may target all human adenovirus serotypes or all subtypes of influenza A, broad spectrum biosensors may target all bacteria or all fungi with a single primer pair or collection of primer pairs. These sorts of processes can be validated in a general way using a representative subset of the theoretically detectable analytes, much as a single PCR test that claims to detect all human adenovirus serotypes or influenza strains can be validated using a reasonable number of genetically diverse strains.,

In traditional PCR, the amplification reaction and downstream signal acquisition are the source of both sensitivity and specificity—analyte-targeted biochemistry defines the breadth of coverage and the ability to identify the analyte. For broad spectrum methods, the PCR and downstream analytical chemistry define only the breadth of coverage and limit of detection (sensitivity); it is the digital information processes that provide identification (specificity). Figure 1 illustrates the basic process for microbial identification using broad spectrum biosensors. Step A represents the bioinformatic analysis process used to design primer pairs for a select group of related organisms. The same bioinformatic processes are used to generate a database of pre-computed signatures that can be used in later steps for signature matching and organism identification. Step B represents the various biochemical processes used to generate detectable signals. These processes are similar or identical for all detectable organisms. The burden of identification and specificity rests on the bioinformatic analysis tools, including the database from Step A, the signal analysis tools and signature matching algorithms in Step C, and any reporting filters (defined in Step D) that may be necessary to provide appropriately actionable information. Step D represents a process of risk analysis and mitigation focused on maximizing the clinical relevance of data provided to users—including, for example, the adoption of cutoffs or non-reporting pathways for certain targets that are more likely to appear as contaminants than as pathogens, or the pipelining of certain data related to biothreats or potential pandemic agents to public health agencies rather than primary care providers.

Figure 1. Microbial identification schema for broad spectrum nucleic acid-based biosensors.

Breadth of Coverage and the Limit of Detection

Because broad spectrum biosensors target a range of organisms rather than a specific organism, their breadth of coverage and their limit of detection (LOD) are inextricably linked. For assays using PCR-based amplification, conservation of the nucleic acid sequences targeted by the PCR primers and the permissiveness of the PCR conditions together determine the phylogenetic range of amplification and the variability in LOD across that range. PCR-based biosensors are inherently limited by sequence conservation within a targeted group of related organisms. While they can be essentially universal within a recognized phylogenetic spectrum, the potential sensitivity (or lack thereof) to previously uncharacterized organisms outside of that spectrum cannot be predicted. PCR-based biosensors can, for example, be designed to detect and identify all bacteria or all influenza viruses, but they cannot be effectively designed to detect previously unrecognized viruses that do not share the conserved sequences upon which a biosensor’s primers are based.

To design a broad spectrum assay, genomic sequence data are analyzed and regions evaluated for conservation across all members of the targeted phylogenetic group. The criteria for primer selection generally follow the principles described in the CLSI MM18-A guideline document, adapted to suit a technology’s design constraints. As an example, some broad spectrum PCR/ESI-MS primer pairs target conserved regions of bacterial genes such as the 16S rDNA, 23S rDNA, rpoB, and tufB., Each pair of primers is evaluated for conservation of target sequences within the group it is designed to detect, and primers are designed to match all members of the group as equally as possible by minimizing the degree of biased mismatches to specific members of the group. The amplified sequence between the primers must offer sufficient variability to allow discrimination of specific organisms at the intended level of resolution. An example is shown in Figure 2.

Figure 2. Example of conserved primer targets for broad bacterial detection. The figure depicts an alignment of homologous 16S ribosomal sequences from diverse representative bacteria. The forward and reverse primer sequences are shown in the first row. The priming regions across the organisms are indicated with dots for conservation and nucleotide letters where the primer pair and the target sequence are not complementary. Colored coded sequences between the priming regions show the diversity in sequence content that provides species-specific identification information through chemical analysis of amplified fragments and subsequent signature matching to a reference database.

For broad spectrum amplification reactions, the only substantive biochemical difference between analytes is the degree of match of the primers to their target sequences (Fig. 2). We propose that representative organisms from across the designed breadth of coverage of an assay be used to measure the general limit of detection and to statistically determine the range and variance of sensitivity resulting from primer site variations. Sensitivity would not be measured independently for each possible analyte.

Specificity and Accuracy of Detection

broad spectrum tests gain specificity through two layers of filtering. The first is primer specificity. Primers are designed to amplify fragments only from organisms within the assay’s designed breadth of coverage. The second filter is bioinformatic. Because reported detections are limited to species for which the downstream analysis system contains specific signatures, mispriming and amplification of untargeted species will not result in false-positive identifications. The primary function of specificity testing for a broad spectrum biosensor should be to measure the phylogenetic resolution of the system provided by the information contained in the amplified genetic regions and reflected in the reference database.

The functional metric of specificity should be considered the “accuracy of identification”. A wide diversity of targeted species should be tested to measure the assay’s resolution as defined by bioinformatic signature acquisition and matching. Bioinformatic resolution can be demonstrated through the same process described for sensitivity—testing of a significant number of representative analytes will allow definition of the range and variance in accuracy (resolution) of the identification system across the breadth of coverage. The interpolation of general accuracy measures to untested analytes requires in silico analysis of reference database signatures to demonstrate that the amplified regions will produce unique signatures for each reportable organism and to demonstrate that the associated signatures are correctly annotated in the database. For organisms for which the target sequences are not available in existing published databases, signatures can be obtained by directly analyzing representative samples of the organism obtained from existing strain collections (see “Database Expansion” in Fig. 1).

A Tiered Validation Approach for Broad Spectrum Biosensors

Beginning with a well-designed universal approach to amplification, based on the principles described above and supported by an in silico demonstration of the theoretical breadth of coverage and resolution of an assay, we propose that broad spectrum biosensors could be validated in a tiered fashion by testing representative microorganisms to demonstrate the general capabilities and limitations of the system. In Table 1 we outline one possible example, in which different organisms are used to analytically validate different aspects of system performance. For the purposes of this discussion, a reportable organism is defined as any organism theoretically detected by an assay’s primers, represented in the assay’s database, and permitted to be reported to users by the filters defined in the assay’s reporting rules.

Table 1. Broad spectrum biosensor analytical validation strategy

Analytes	Definition of analyte group	Analytical studies	Purpose of tier
Tier I: Core organisms	Exercise all primers and utilize all reagents and conditions	• LOD determination• LOD confirmation• Carryover• Interfering substances• Sample stability• Reproducibility• Precision• Mixtures (if applicable)	Analytical validation of physical and biochemical processes
Tier II: Limit of detection	Representative of full breadth of assay, including genetic outliers	• LOD determination• LOD confirmation	• Measure and report LOD for tested organisms• Estimate range and variance of sensitivity to interpolate LOD for untested organisms
Tier III: Breadth of coverage	All available species on reportable organism list. Analysis for correct identification only	Accuracy of detection (tested near the upper limit of the general range of LOD measured in Tier II).	• Demonstrate correct identification for broad and diverse set of representative reportable organisms.• Estimate range and variance of identification resolution to interpolate accuracy for untested organisms.
Tier IV: In silico	All other reportable organisms for which analytes are unavailable.	Analyze unavailable or unquantifiable organisms in silico on the basis of primer match and amplicon novelty.	Demonstrate that untested targets are within the validated primer match range and that available signatures allow discrimination.

The proposed first tier of validation involves demonstration of the biochemical capability of the assay through full analytical testing of a set of analytes (“core organisms”) that together use all of the primer pairs and biochemical reactions of the test method. This work should be done using live organisms spiked in natural matrix, following the same standards of analytical testing applied to predicate molecular in vitro diagnostic (IVD) devices. The limit of detection should be determined for each of the core organisms, followed by traditional analytical testing of parameters including reproducibility, interfering substances, cross reactivity, and carryover, performed at near-LOD concentrations following applicable FDA guidance and CLSI standards.-

Testing of second tier “limit of detection organisms” would be performed in natural matrix to demonstrate the general sensitivity through dilution series testing of diverse organisms (including genetic outliers) that could be identified by the assay. This stage of analysis would be used to define the average sensitivity of the assay and the variance in sensitivity resulting from divergence in primer target sequences. Testing of the third tier of “breadth of coverage organisms”, along with data collected in Tiers I and II, would provide verification of the accuracy of identification provided by the amplicon analysis system and the downstream database and bioinformatic analysis software. Tier III organisms would include a larger group of diverse species chosen on the basis of either their critical importance for the suggested clinical application or their representation of phylogenetically distinct groups of organisms not evaluated in Tier II LOD testing. Tier III should include some organisms representing the worst-case scenarios in terms of primer match specificities, representation in the reference database, numbers of primers expected to yield amplicon, or other critical factors likely to make detection and identification challenging. These organisms should be tested in natural matrix at the upper end of the LOD range determined in Tier II testing. This general scheme has been demonstrated in a recent study showing that LOD testing of a limited number of bacterial species on a broad spectrum biosensor provided a reliable estimate of the LOD for other bacterial species. The observed variance in accuracy of identification across all organisms tested in Tiers I, II, and III can be used to estimate the accuracy of identification for all reportable organisms through statistical interpolation. For certain organisms of interest, direct testing is hazardous. For others, testing will be unduly onerous due to sheer numbers of reportable analytes or impossible due to a lack of available reference strain material. The ability to detect these organisms could be verified through in silico demonstrations of appropriate matches to primer sequences and the availability of unique signatures in the reference database.

Some Considerations Related to Broad Spectrum Biosensor Application

The universal nature of broad spectrum biosensors suggests that significant competitive interference may occur if there are multiple analytes in the same specimen, especially in cases where those analytes would require amplification by the same primers. For this reason some broad spectrum tests may work only in generally sterile or semi-sterile sample types where diversity tends to be low. This limitation is best compared with culture systems, in which microorganisms that grow on the same media types can overgrow or out-compete each other for growth resources. The general level at which co-infecting organisms or contaminants mask detection of other organisms or interfere with appropriate identification should be addressed through mixture experiments with a limited number of organisms.

Another challenge in the validation of broad spectrum biosensors relates to the comparator or reference methods used, particularly in clinical studies. Many microorganisms capable of being detected and identified with broad spectrum assays are inherently unculturable. Other microorganisms are difficult to culture or require special culture conditions. Even in the case of readily cultured species, nucleic acids may be present in the absence of viable cells, especially in patients for whom pre-emptive antibiotic therapy precedes sample collection. The use of PCR followed by bi-directional (Sanger) sequencing as either a direct comparison method or for evaluation of discordants compared with culture can mitigate some of these issues. However, the use of this technique has its own challenges. The only sequencing methods that can approach the breadth of coverage of broad spectrum biosensors are conserved-site sequencing methods currently in common use for post-culture molecular identification.,, In uncultured specimens, these methods share the issues of primer mismatches and associated variance in sensitivity with broad spectrum biosensors, compounded by a lack of redundancy (addressed in many broad spectrum methods by the use of multiple primer pairs). This is expected to render such methods less sensitive than broad spectrum assays for some analytes and possibly more sensitive for others. Typically, polymicrobial specimens would require cloning followed by sequencing in order to capture multiple analytes, which becomes extremely expensive and may still fail to yield signal from low-titer or minority components of mixtures.

For some broad spectrum assays, there may be inherent limitations in the sensitivity of the assay relative to culture methods derived from the volume of sample tested. For example, blood culture bottle analysis of whole blood for purposes of diagnosing sepsis and bacteremia utilizes 10 mL of blood per assay bottle. Many molecular assays use 50 µL or less of equivalent specimen extract in each PCR reaction. Hence, the ultimate theoretical sensitivity of these molecular methods is at least 200-fold lower than that of culture. For all of the above reasons, there are no optimal comparator methods for broad spectrum biosensors. We propose that such assays be characterized as best as possible with accepted comparator methods, without a priori requirements related to the demonstrated sensitivity and specificity.

Conclusions

Broad spectrum biosensors have the potential to revolutionize both medical diagnostics and public health/biothreat surveillance. These technologies allow clinical laboratories, surveillance agencies, and researchers to query single samples for hundreds of organisms simultaneously, freeing them from the need to restrict their investigations and laboratory diagnoses to a few most-likely etiologies. The universal nature of such assays means that routine diagnostics, epidemiological tracking, biothreat surveillance, and pathogen discovery work could occur simultaneously in near real-time at the point of care. To fully realize the potential of these assays and enable them within the world of applied medicine, the paradigms guiding diagnostic assay regulation must shift with the technology and allow for broader and more general forms of analytical and clinical validation. A simple linear extension of the current models does not provide a feasible path toward acceptance of technologies with hundreds of potential analytes. The validation procedure proposed here would ensure the reliability of broad spectrum technologies without placing insurmountable obstacles in the path of approval.