Use and evaluation of molecular diagnostics for pneumonia etiology studies.

Comprehensive microbiological testing will be a core function of the Pneumonia Etiology Research for Child Health (PERCH) project. The development stage of PERCH provided the time and resources necessary for us to conduct a comprehensive review of the current state of respiratory diagnostics. These efforts allowed us to articulate the unique requirements of PERCH, establish that molecular methods would be central to our testing strategy, and focus on a short list of candidate platforms. This process also highlighted critical challenges in the general design and interpretation of diagnostic evaluation studies, particularly in the field of respiratory infections. Although our final molecular diagnostic platform was ultimately selected on the basis of operational and strategic considerations determined by the specific context of PERCH, our review highlighted several conceptual and practical challenges in respiratory diagnostics that have broader relevance for the performance and interpretation of pneumonia research studies.

The development of a comprehensive microbiological testing strategy has been a core principle in the conception and design of the Pneumonia Etiology Research for Child Health (PERCH) project [1]. In formulating the most effective approach for respiratory diagnosis, we determined that a multiplex molecular diagnostic platform would be an essential component in our approach. Many of the technical and operational considerations encountered through this process proved relevant to the overall design of the project. We describe here the theoretical and practical challenges encountered in the evaluation and selection of a molecular platform for the diagnosis of pneumonia.

METHODS FOR THE DIAGNOSIS OF PNEUMONIA

As described elsewhere in this issue [2, 3], microbiological evidence of infection must be considered in the context of several fundamental difficulties found in respiratory diagnostics, including the frequent lack of access to the site of infection, the insensitivity of available tests, insufficient assay validation, and complexities in determining whether a detected pathogen has a causal role in the illness. The specific research-related demands of PERCH added to these constraints, requiring that our diagnostic strategy must exclude any prior assumptions regarding the likely importance of specific pathogens; must include a full range of respiratory tract specimens, including upper respiratory swab or aspirate, induced sputum, lung aspirate, bronchoalveolar lavage, and pleural fluid; must be comprehensive, yet realistic; must appropriately balance the demands of accuracy and efficiency; must account for both clinical and research ethical issues; and must be feasible for use and support at all participating field sites.

To begin the selection process, the PERCH investigators conducted an extensive review of the microbiologic diagnosis of respiratory infections. Using published and unpublished data, as well as user and developer experiences, our team prepared a strategic summary of the available technologies that could detect pathogens from respiratory tract specimens. We evaluated each major assay category, including traditional bacteriology and viral culture, direct antigen and immunofluorescent antibody detection, and nucleic detection acid tests. It was evident that molecular diagnostics should be among the mix of diagnostic tools required to meet the needs of PERCH.

Nucleic acid detection tests (NADTs) have a number of advantages over other diagnostic platforms for the evaluation of respiratory specimens [4]. They demonstrate superior sensitivity in detecting organisms that are fastidious, less viable, or present in only small amounts [5]. Molecular diagnostics can also be quickly adapted to detect evolving or emerging pathogens and are amenable to efficiencies of scale such as automation. They also allow the simultaneous detection of multiple targets (multiplexing), which in turn allows for testing by clinical syndrome and the detection of co-infections. NADT methods present less of a safety hazard for laboratory personnel compared with culture, typically require less time compared with bacterial culture, and require less technical capacity compared with viral culture. Given these advantages, NADTs have been extensively evaluated in the detection of several viruses and bacteria of the respiratory tract and have become the diagnostic tool of choice for many agents that are difficult to isolate [4].

Molecular diagnostic platforms are not without their disadvantages. Cost and complexity remain significant barriers to adoption in many laboratories, and NADTs often risk problems of laboratory contamination with amplified products, particularly if the assay procedure requires opening of the reaction tube prior to the target detection step [6]. Measures to limit contamination often require additional laboratory space that may not be available in resource-constrained settings. Nevertheless, NADT methods represent one of the more productive areas of diagnostics research, promising future improvements in automation and speed, smaller devices, improved cost efficiencies, and better detection of emerging pathogens [7].

REVIEW OF MOLECULAR DIAGNOSTICS

The focus on molecular methods for respiratory pathogen detection yielded a large variety of potential technologies for consideration in PERCH (Table 1). Polymerase chain reaction (PCR) technology is more common at research sites worldwide, can be adapted to various platforms, and easily allows for multiplex amplification. Multiplexing, in which several targets are assayed for simultaneously, is commonly employed in PCR-based assays and offers significant advantages over single-pathogen assays in terms of efficiency and pathogen coverage. Still, developers must overcome considerable complexities in harmonizing the reaction requirements of each individual target and limiting potential competition among the analytes. These factors may result in a measurable decrease in sensitivity compared with single-plex assays. Several techniques have been developed to address such factors, such as alterations in cycling protocols [8], nested primer combinations [9, 10], complex primer structures and concentrations [11, 12], and the use of nontraditional nucleotides [13-15]. Other NADT technologies, such as nucleic acid sequence-based amplification and loop-mediated isothermal amplification, have been used for the detection of respiratory pathogens, but experience with multiplexing is limited [16-19].

Table 1.

Multiplex Molecular Diagnostic Platforms Considered by the Pneumonia Etiology Research for Child Health Project

Platform (Commercially Available Examples)	Advantages	Disadvantages	Selected References
Liquid-phase suspension array (Luminex, MassTag, IBIS)	Capacity for >20 targets in 1 reaction	Requires specialized equipment; decreased sensitivity, particularly with co-infections; typically requires opening the reaction tube	[9, 10, 32–34, 54]
Solid-phase arrays (Infiniti, Virochip)	Can potentially detect thousands of targets	Requires higher target concentrations; expensive equipment and reagents	[24, 25, 30, 31]
Mutliplex real-time PCR (Fast Track Diagnostics)	Widely used technology; closed system; provides information on the quantity and quality of target	Limited multiplexing capabilities in each tube; expensive equipment and reagents	[35, 38, 39]
Multiple uniplex real-time PCR array (Taqman Low-Density Array)	No interactions between target primers or reactions; capacity for >20 targets per specimen	Microfluidic scale reduces sensitivity; expensive equipment and reagents	[36]

Abbreviation: PCR, polymerase chain reaction.

Technologies for target detection take on an even larger variety of formats. Older methods include agarose gel electrophoresis, reverse-transcription PCR enzyme hybridization assay [20-22] and enzyme-linked oligonucleotide capture [18]. More recently, solid- and liquid-phase array platforms have become more useful for the detection of multiple targets. Solid-phase arrays use a variety of formats, typically embedding target-specific oligonucleotides onto a glass or silicon microchip [23-31] to detect anywhere between dozens to hundreds of thousands of amplified sequences. Several respiratory diagnostic systems have been based on a liquid-phase technology using polystyrene microbeads (Luminex) [9, 10, 13–15, 32] or mass spectroscopy [33, 34] for amplicon discrimination. Although these approaches have greatly expanded the versatility and sensitivity of multiplex PCR, their complexity, specialized equipment, and high start-up costs have limited their widespread adoption to date. Moreover, these platforms typically require separate steps for amplification and detection, increasing both the workload and the risk of operator error or amplicon contamination.

Real-time PCR assays address these issues by combining amplification and detection in one reaction tube, thus facilitating automation and reducing contamination. In addition, this technique allows for the quantification of pathogens and the assessment of replication efficiency. As with conventional PCR, multiplex real-time assays are subject to competition and inhibition among primers [5, 35]. Real-time assays are also restricted in the number of reaction products that can be detected in parallel [35], although this problem can be partially circumvented using arrays of uniplex real-time reactions at very small volumes [36]. Successful in-house real-time assays directed against respiratory pathogens have been developed using uniplex [37] and multiplex [38, 39] approaches, but data on the performance of commercialized versions are not readily available.

Much of the effort in NADT development for respiratory diagnostics has focused on the detection of viruses, given the advantages of these techniques over conventional methods in terms of speed, sensitivity, and versatility for detecting this class of pathogens. Multiplex approaches for viral detection have become more common as technologies have improved (reviewed by [5, 7, 40, 41]). In addition, NADTs have now become the gold standard for the detection of Mycoplasma pneumoniae [42] and Chlamydophila pneumoniae [43] and a useful addition to antigen testing for Legionella species [44]. Multiplex assays for the detection of more traditional bacterial pathogens have not been studied as frequently in respiratory specimens, primarily because culture techniques are usually adequate for clinical practice. Moreover, molecular methods provide no additional advantage over culture in differentiating infection from colonization of the upper respiratory tract. Nevertheless, multiplex NADTs for bacteria such as S. pneumoniae,Haemophilus influenzae, and Streptococcus pyogenes have been evaluated in respiratory specimens [45] and will likely be incorporated into larger multiplexing assays.

CHALLENGES IN THE EVALUATION OF EXISTING DATA

Having conducted our survey of the field, we narrowed our list of candidate molecular diagnostic platforms even further on the basis of the unique needs of our research study. As with clinical laboratories, we closely examined factors such as cost, feasibility, quality assurance, capital investment, platform versatility, and future utility. In contrast with clinical laboratories, we considered issues such as the rapid return of results or regulatory approval for use in patient care to be less crucial to our objectives. Moreover, our approach emphasized comprehensive pathogen detection, rather than focusing primarily on pathogens relevant for clinical management or infection control. Finally, our selected platform would be deployed in low-resource settings, where requirements for a reliable or continuous power supply, adequate access to reagents, sensitivity to extreme environmental conditions, and access to technical support would be highly relevant.

As our appraisals progressed, we encountered many challenges in interpretation that are common to the field of diagnostics evaluation. Most basic among these was confusion regarding the usage of the terms “sensitivity” and “specificity,” and the evaluations needed to measure these parameters [46]. The distinction between “analytic” performance characteristics, as opposed to “diagnostic” or “clinical” performance characteristics, is essential for properly assessing the validation of any assay, but it is particularly true in the field of molecular diagnostics. For NADTs, analytic sensitivity refers to the lowest concentration of target that can be detected, whereas analytic specificity measures the ability of the test to exclude undesired targets despite similar genetic sequences. In contrast, diagnostic or clinical sensitivity of a nucleic acid detection test refers to the appropriate identification of all patients carrying the agent, and diagnostic or clinical specificity describes the assay’s ability to exclude uninfected patients. Clinical performance characteristics are subject to a number of factors, including the patient’s disease status, variations in the concentration of the target throughout the course of illness, inhibition by other substances present in the specimen, sample quality, sampling variability, and specimen degradation. Generally, assays should be tested against a reference or gold standard. For tests of microbial detection, the reference standard typically is culture, but molecular diagnostics are often much more sensitive in detecting nucleic acid than is culture for viable organisms, leading to difficulties in interpreting the clinical relevance of false-positive results. The challenges of assessing diagnostic tests have been increasingly recognized in recent years. For instance, the Standards for Reporting of Diagnostic Accuracy initiative [47, 48] offers guidelines on the reporting of diagnostic studies, whereas the Quality Assessment tool for Diagnostic Accuracy Studies provides corresponding guidance on their evaluation [49].

Nevertheless, respiratory diagnostics are particularly limited by the inability to determine whether the detection of a particular pathogen in a symptomatic patient indicates that it is causative of the illness or results from contamination, colonization, or prolonged shedding from a prior unrelated infection, particularly when testing specimens from the upper respiratory tract. This issue is not typically addressed in diagnostic evaluation studies, but it has become more relevant as molecular diagnostics have expanded the lower limits of pathogen detection by several orders of magnitude. Attempts to answer this question have suggested an additional category of test performance, the “epidemiological” specificity of a test, to describe the ability of an assay to assign true etiologic status to a pathogen for a specific illness. Ultimately, determination of the epidemiologic specificity of a respiratory diagnostic would require the interpretation of microbiologic results in conjunction with all other clinical and laboratory data, perhaps in the form of a predictive model. Such analyses are uncommon but will be a main focus of the PERCH study.

Respiratory diagnostics are further complicated by the absence of a perfect gold standard. Culture is difficult or insensitive for some pathogens and unavailable for others (eg, human metapneumovirus, parainfluenzavirus type 4, rhinovirus group C, or Pneumocystis jiroveci). Serologic tests are often not available and usually require paired serum specimens for accurate results. Statistical methods to adjust for such alloyed gold standards, such as discrepant analysis, have been frequently employed, but they can be susceptible to significant bias [50]. Comparative evaluations of respiratory diagnostic assays must also take into account variations in which panel of pathogens is selected, which genetic sequences are targeted, what specimen sources are used [3, 51], and even what methods are used for nucleic acid extraction [52]. The US Food and Drug Administration has recently published industry guidance that may encourage additional work in this area [53].

As the PERCH evaluation progressed, the concepts derived from our deliberations were distilled into a list of desirable and essential attributes summarizing our strategy for evaluation (Table 2). This list addressed issues such as assay performance (range of targets, acceptable specimen sources, sensitivity, and specificity), operational concerns (space requirements, assay throughput, quality assurance programs, maintenance requirements, and reagent availability), and strategic issues (capacity for automation, versatility and future utility, start-up and maintenance costs, and developer engagement). For additional input, we presented our summary to the Pneumonia Methods Working Group, an expert committee formed to advise PERCH. Ultimately, this outline of key qualities and data allowed us to articulate our thoughts and communicate our strategy more effectively to collaborators, advisors, and assay developers.

Table 2.

Desirable and Essential Attributes of Molecular Diagnostic Platforms for the Pneumonia Etiology Research for Child Health Project

Desirable	Essential
Flexibility to modify existing targets or incorporate new ones	Ability to detect target pathogens
Ease of use, workflow, and space	Demonstrated high analytical sensitivity and specificity
Rapid turnaround times	Demonstrated high clinical sensitivity and specificity
Nucleic acid extraction procedure included in overall process (and automated)	Ability to process a variety of respiratory tract specimens
Small specimen volume requirements	Specimen collection requirements well-characterized and suitable for field studies
Readily available reagents with long expiry dates and room-temperature storage requirements	Inclusion of control specimens and quality control procedures
Capacity to provide quantitative or semiquantitative results	Available maintenance and support
Licensed with an accreditation authority	Acceptable time frame for development
Comprehensive cost information available

We applied our list of attributes to more than a dozen candidate diagnostic systems that met our initial criteria, and developed a short list of candidate platforms. We then tested these final assays in our PERCH-affiliated laboratories, using a standardized set of mock specimens. This process allowed us to engage with the assay manufacturers and their academic partners, directly compare the performance characteristics of the platforms, and gain essential information that could only be acquired through hands-on experience, such as capabilities for technology transfer, ease of use, and workflow. Details of this evaluation will be the subject of a separate article.

CONCLUSIONS

By including a phase for protocol development, the PERCH investigators were able to perform an extensive literature review of respiratory diagnostics, clearly outline the major theoretical and practical concerns, and engage a group of experts for critical input. Through this process, we confirmed the suitability of molecular diagnostics for our needs and identified critical information gaps. Our evaluation highlighted numerous advantages of this technology, including excellent sensitivity and adaptability for a full range of respiratory pathogens and specimen sources, as well as clear capabilities for multiplexing and automation. We nevertheless realized that our conclusions represent but a snapshot in time, and the field of molecular diagnostics is rapidly evolving, with constant improvements in accuracy, speed, automation, and cost. Yet it can also be expected that methods for evaluating respiratory diagnostics will continue to evolve in parallel, providing new answers to the practical and conceptual challenges that shaped the development of a diagnostic testing strategy for PERCH.