Accuracy of LightCycler(R) SeptiFast for the detection and identification of pathogens in the blood of patients with suspected sepsis: a systematic review protocol.

UNLABELLED: Background There is growing interest in the potential utility of molecular diagnostics in improving the detection of life-threatening infection (sepsis). LightCycler® SeptiFast is a multipathogen probe-based real-time PCR system targeting DNA sequences of bacteria and fungi present in blood samples within a few hours. We report here the protocol of the first systematic review of published clinical diagnostic accuracy studies of this technology when compared with blood culture in the setting of suspected sepsis. Methods/design Data sources: the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects (DARE), the Health Technology Assessment Database (HTA), the NHS Economic Evaluation Database (NHSEED), The Cochrane Library, MEDLINE, EMBASE, ISI Web of Science, BIOSIS Previews, MEDION and the Aggressive Research Intelligence Facility Database (ARIF). STUDY SELECTION: diagnostic accuracy studies that compare the real-time PCR technology with standard culture results performed on a patient's blood sample during the management of sepsis. DATA EXTRACTION: three reviewers, working independently, will determine the level of evidence, methodological quality and a standard data set relating to demographics and diagnostic accuracy metrics for each study. Statistical analysis/data synthesis: heterogeneity of studies will be investigated using a coupled forest plot of sensitivity and specificity and a scatter plot in Receiver Operator Characteristic (ROC) space. Bivariate model method will be used to estimate summary sensitivity and specificity. The authors will investigate reporting biases using funnel plots based on effective sample size and regression tests of asymmetry. Subgroup analyses are planned for adults, children and infection setting (hospital vs community) if sufficient data are uncovered. Dissemination Recommendations will be made to the Department of Health (as part of an open-access HTA report) as to whether the real-time PCR technology has sufficient clinical diagnostic accuracy potential to move forward to efficacy testing during the provision of routine clinical care. Registration PROSPERO-NIHR Prospective Register of Systematic Reviews (CRD42011001289).

Introduction

Sepsis is the clinical syndrome resulting from a host's systemic inflammatory response to infection.1 When severe, it is associated with considerable mortality and is a major international healthcare problem.2 Confirmation of sepsis requires objective evidence for infection,3 which should always include an attempt at microbiological identification of live pathogens from blood samples by culture techniques.3 4 However, culture routinely takes several days before a positive result is available and at least 5 days to determine that a specimen is culture negative.5 This temporal separation between initial clinical suspicion and confirmation of infection routinely results in the early and sustained delivery of potent broad spectrum antibiotics aimed at covering the most likely pathogens as a ‘safety first’ strategy because delay in appropriate antimicrobial therapy is associated with increased mortality.6 7 The inevitable consequence is unnecessary antibiotic prescription, which is associated with the development of antimicrobial resistance (eg, MRSA), Clostridium difficile infection as well as a range of avoidable adverse effects, and acquisition costs, of antimicrobial drug use.8 This is driven by a lack of access to time-critical high-specificity biomarkers of infection in critical care9 where overwhelming systemic inflammation of the body is a common occurrence and is often not caused by infection (it may, eg, be caused by trauma, blood transfusion, pancreatitis).10 There is therefore an urgent need to develop techniques that can provide accurate diagnostic information within hours of clinical signs appearing and so allow more informed use of antibiotic therapy at an early stage.

There is growing interest in the potential of real-time PCR technology to address this problem based on its ability to detect minute amounts of pathogen DNA in patient blood samples with results available within 4–6 h.11 Proof of concept studies have focused on two approaches using PCR for genomic amplification with either (1) broad range detection of bacterial or fungal DNA with universal primers, followed by species identification using a post-PCR technique such as gene sequencing or electrospray mass spectrometry or (2) using species-specific hybridisation probes that provide direct confirmation of the species present.12 Intuitively the latter approach would seem to have the greatest clinical utility assuming an appropriate pathogen panel can be established.

While the laboratory analytical sensitivity and specificity of these techniques for the detection of pathogen DNA in blood has been evaluated,12 there remains an acknowledged lack of clinical trial data to define the diagnostic reliability of such tests in patients who develop a systemic inflammatory response due to suspected infection. This has been due in part to the lack of standardised technology platforms that meet accepted regulatory standards for clinical diagnosis.

SeptiFast, manufactured by Roche Diagnostics, GmbH, Mannheim, Germany and run on their real-time PCR instrument (the LightCycler®), is the first real-time PCR-based system to be awarded a Conformité Européenne (CE) mark for pathogen detection and identification in suspected bloodstream infection12 13 and, to date, the most intensively investigated in clinical cohort studies. The system uses a multiplex approach, which allows detection of 25 of the most common pathogen species causing bloodstream infection in a single blood sample (table 1). Identification of the pathogens is based on the use of species-specific probes targeting the internal transcribed spacer region between the 16S and 23S areas of ribosomal DNA of bacteria and between the 18S and 5.8S ribosomal regions of the fungal genome. This real-time PCR technology has been extensively assessed at the laboratory level on clinical isolates and shown to have excellent analytical specificity and exclusivity, confirming its analytical validity.13 An EU registration study (unpublished), undertaken as part of the CE-marking process, investigated 278 critically ill patients with suspected sepsis from Denmark, Germany and Italy. Roche Diagnostics reported that the molecular test conferred a high diagnostic specificity and a 3- to 10-fold higher sensitivity when compared with conventional blood culture technology. Since this study, numerous commercial clinical diagnostic studies have been reported, predominantly focused on suspected sepsis. In addition, a large, independent, multicentre, level III clinical diagnostic accuracy study of this real-time PCR technology is currently recruiting14 as part of our detailed National Institute of Health Research (NIHR) Health Technology Assessment (HTA)-funded programme to assess real-time PCR technologies in sepsis diagnosis, treatment and outcome. A small number of newer emerging CE-marked PCR technologies are also currently available that can detect pathogen DNA rapidly in blood samples without the need for pre-culture, either using an all-pathogen approach without simultaneous speciation (eg, SepsiTest; Molzym GmbH, Bremen, Germany) or by using a multiplex approach based on a wide pathogen panel (eg, VYOO; SIRS-LAB GmbH, Jena, Germany). Compared with SeptiFast, the current number of clinical diagnostic studies using these technologies is very limited. Therefore, at this time, and as part of our independent HTA clinical studies,14 we describe here the protocol of a systematic review focused on the diagnostic test accuracy of LightCycler® SeptiFast for pathogen detection and identification in the blood of patients with suspected sepsis. This systematic review has been registered with PROSPERO—the NIHR International Prospective Register of Systematic Reviews (CRD42011001289).15

Table 1

Pathogens detectable using the LightCycler® SeptiFast test

Gram-negative bacteria	Gram-positive bacteria	Fungi
Escherichia coli	Staphylococcus aureus	Candida albicans
Klebsiella (pneumoniae/oxytoca)	Coagulase-negative staphylococci*	Candida tropicalis
Serratia marcescens	Streptococcus pneumonia	Candida parapsilosis
Enterobacter (cloacae/aerogenes)	Streptococcus spp.†	Candida glabrata
Proteus mirabilis	Enterococcus faecium	Candida krusei
Acinetobacter baumannii	Enterococcus faecalis	Aspergillus fumigatus
Pseudomonas aeruginosa
Stenotrophomonas maltophilia

Single probe detects a group of staphylococcal pathogens including S epidermidis, S haemolyticus.

Single probe detects a group of streptococcal pathogens including S pyogenes, S agalacticae, S mitis.

Methods

Inclusion criteria of studies

Participants

Patients suspected of developing sepsis, including adults and children, who require blood cultures irrespective of where their care is being delivered, and including suspected community or hospital acquired infection.

Target conditions

Sepsis, including severe sepsis and septic shock.16

Index test

LightCycler® SeptiFast as the index test on blood for the detection and identification of bacterial and fungal pathogens.13

Comparator test (reference standard)

Blood cultures used as the reference test and underpinning routine clinical practice.5 All diagnostic metrics will be reported using this reference standard. Clearly, there could be limitations to this standard, particularly in the setting of intercurrent antimicrobial therapy. However, we do not know the full extent of this problem or indeed whether studies have deliberately included or excluded such patients. In the absence of a straightforward and internationally agreed approach to an alternative reference standard at present, we believe that this is the most robust approach and is consistent with methods currently used in independent HTA clinical studies in this setting.14

Types of studies

We will include any clinical diagnostic accuracy study that compares the index real-time PCR test with standard culture results performed on a patient's blood sample during the management of sepsis.

Search methods for identifying studies

Electronic searches

We will search the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects (DARE), the Health Technology Assessment Database (HTA), the NHS Economic Evaluation Database (NHSEED), The Cochrane Library, MEDLINE, EMBASE, ISI Web of Science, BIOSIS Previews, MEDION and the Aggressive Research Intelligence Facility database (ARIF). The CE mark for the index test was announced in January 2006; therefore, this systematic review will only consider publications from this date in humans. There will be no language restrictions in the electronic search for trials.

Search terms/search strategy

Specific search strategies will be developed for each electronic database, commencing with MEDLINE (table 2). The MEDLINE strategy will be adapted for each subsequent database and search yields reported and compared between databases.

Table 2

MEDLINE search strategy

#1	sepsis.mp. or exp Sepsis/
#2	septic shock.mp. or Shock, Septic/
#3	fung?emia.mp. or Fungemia/
#4	bacter?emia.mp. or Bacteremia/
#5	blood?stream infection$.mp.
#6	blood poison$.mp.
#7	Systemic Inflammatory Response Syndrome/ or SIRS.mp.
#8	septic?emia.mp.
#9	"severe sepsis".mp.
#10	(presumed adj4 sepsis).mp.
#11	(suspected adj4 sepsis).mp.
#12	#1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 or #10 or #11
#13	PCR.mp. or Polymerase Chain Reaction/
#14	SeptiFast.mp.
#15	LightCycler.mp.
#16	multiplex PCR.mp.
#17	real time PCR.mp.
#18	real?time PCR.mp.
#19	Molecular Diagnostic Techniques/ or molecular diagnosis.mp.
#20	molecular identification.mp.
#21	#13 or #14 or #15 or #16 or #17 or #18 or #19 or #20
#22	blood cultur$.mp.
#23	Bacteriological Techniques/mt [Methods]
#24	Blood/mi [Microbiology]
#25	#22 or #23 or #24
#26	#12 and #21 and #25
#27	Animals/
#28	#26 not #27
#29	Viruses/
#30	#28 not #29
#31	limit #30 to (humans and yr=“2006 -Current”)

mp=protocol supplementary concept, rare disease supplementary concept, title, original title, abstract, name of substance word, subject heading word, unique identifier.

Other resources

Backward tracking will be performed by hand-searching the reference lists of all relevant articles uncovered from the electronic search and forward tracking using the keyword ‘SepiFast’ with ISI Citation Indices and Google Scholar and with a conference proceedings search using the Web of Science ISI Proceedings (2006 to present). We will request reference lists held by the only manufacturer of the index test (Roche Diagnostics) and include public-domain clinical diagnostic accuracy data collected by Roche Diagnostics to file for the CE mark. In addition, we will search for unpublished studies and ongoing trials involving the SeptiFast platform in the following online registers: http://www.nlm.nih.gov/hsrproj, http://www.controlled-trials.com/mrct/, http://portal.nihr.ac.uk/Pages/Portfolio.aspx and http://www.who.int/trialsearch, with identified corresponding authors of eligible trials and content experts contacted to identify potentially relevant studies and associated data.

Data collection and analysis

Selection of studies (Salford, UK)

The initial selection of titles and abstracts will be conduct by two review authors (CW and PD) independently using the inclusion criteria detailed above. The full papers of all abstracts deemed eligible (by any reviewer) will be obtained and read to determine their inclusion in the review. Disagreement at each step will be resolved with discussion between the two review authors (PD and CW) and a third author (GW).

Data extraction and management (Belfast and Warwick, UK)

A standard set of data will be extracted for each study using a tailored data extraction form which will include information regarding the inclusion criteria detailed above, an assessment of the level of evidence using the Oxford Centre for Evidence-based Medicine Levels of Evidence17 and additional information including: Study design; Clinical setting (ie, community, emergency department, in-hospital, critical care and general/specialist); Participant demographics; Clinical features of included population (illness aetiology); Intercurrent treatment (antimicrobial therapy); Reference standard methodology, including contamination rates; Supporting test results (culture of samples other than blood); Index test setting (point-of-care, near-patient, clinical or research laboratory, batched or individual analysis); Reported index test laboratory failures; Missing participant data; 2 by 2 table of results for primary outcome and reported diagnostic accuracy metrics; Follow-up (eg, survival and length of intensive care and hospital stay).

Three review authors (DFM, RM and GDP) will independently extract data, and any discrepancies will be resolved by discussion or if necessary by consultation with a fourth author (BB).

Assessment of methodological quality

Three independent authors (DFM, RM and GDP) will assess the quality of each individual study using the checklist (table 3) adapted from the QUADAS tool.18 Each question on the checklist will be answered with a yes/no response or noted as unclear if insufficient information is reported to enable a judgement to be made, and the reasons for the judgement made will be documented. Published standard operating procedures5 and interpretation of the reference standard (blood culture),5 including definitions of blood culture contamination, will be made available to the independent reviewers for reference. In addition, the 2006 CE-marked index test protocol will be provided to each reviewer as provided by Roche Diagnostics to purchasers.

Table 3

Format of assessment of methodological quality adapted from QUADAS tool17

Quality Indicator	Notes
1. Was the spectrum of patients representative of the spectrum of patients who will receive the test in practice?	‘Yes’ if the characteristics of the participants are well described and probably typical of patients with suspected sepsis. ‘No’ if the sample is unrepresentative of people with suspected sepsis. ‘Unclear’ if the source or characteristics of participants is not adequately described
2. Were the selection criteria described?	2a ‘Yes’ by international sepsis definitions,15 ‘No’ otherwise: 2b ‘Yes’ by some other specified sepsis definition ‘No’ otherwise or 2c ‘Unclear’ if insufficient information provided.
3. Is the time period between reference standard and index test short enough to be reasonably sure the target condition did not change between the two tests	‘Yes’ if reference and index tests performed on blood samples drawn at the same time. ‘No’ if tests were performed on blood samples taken at different times. ‘Unclear’ if insufficient information is provided
4. Is partial verification avoided?	‘Yes’ if all participants who received the index text also underwent the reference test. ‘No’ if not all the participants who received the index test also underwent the reference test. ‘Unclear’ if insufficient information is provided. If not all participants received the reference tests, how many did not (of the total)?
5. Is differential verification avoided?	‘Yes’ if the same reference test was used regardless of the index test results. ‘No’ if different reference tests are used depending on the results of the index test. ‘Unclear’ if insufficient information is provided.If any participants received a different reference test, what were the reasons stated for this, and how many participants were involved?
6. Was the execution of the index test done in accordance with the CE-mark protocol?	‘Yes’ as per CE-marked protocol described by manufacturer (Roche Diagnostics) from January 2006. ‘No’ if CE-mark protocol breached. ‘Unclear’ if insufficient information provided. (CE-marked protocol will be provided to the independent reviewers).
7. Was the execution of the reference standard described in sufficient detail to permit its replication?	‘Yes’ if clinical standard described and is consistent with published standard operating procedures.5 ‘No’ if reference standard falls short of standard operating procedures.5 ‘Unclear’ if insufficient information provided. Also comment on how culture contaminations were defined and reported?
8. Are the reference standard test results blinded?	‘Yes’ if the report stated that the person undertaking the reference test did not know the results of the index tests, or if the two tests were carried out in different places. ‘No’ if the report stated that the same person performed both tests, or that the results of the index tests were known to the person undertaking the reference tests. ‘Unclear’ if insufficient information provided.
9. Are the index test results blinded?	‘Yes’ if the report stated that the person undertaking the index test did not know the results of the reference tests, or if the two tests were carried out in different places. ‘No’ if the report stated that the same person performed both tests, or that the results of the index tests were known to the person undertaking the reference tests. 'Unclear' if insufficient information provided.
10. Were uninterpretable results reported?	‘Yes’ if the number of participants in the two-by-two table matches the number of participants recruited into the study, or if sufficient explanation is provided for any discrepancy. ‘No’ if the number of participants in the two-by-two table does not match the number of participants recruited into the study, and insufficient explanation is provided for any discrepancy. ‘Unclear’ if insufficient information is given to permit judgement. Report how many results were uninterpretable (of the total).
11. Were any withdrawals explained?	‘Yes’ if there are no participants excluded from the analysis, or if exclusions are adequately described. ‘No’ if there are participants excluded from the analysis and there is no explanation given. ‘Unclear’ if not enough information is given to assess whether any participants were excluded from the analysis. Report how many participants were excluded from the analysis, for reasons other than uninterpretable results.

Statistical analysis and data synthesis

Data for 2×2 tables of index test against reference standard will be extracted from each study. Initially, these will be plotted as a coupled forest plot of sensitivity and specificity, and a scatter plot in ROC space (plotting sensitivity against 1 − specficity for each study). This will identify any issues of heterogeneity. Summary sensitivity and specificity will be estimated using the bivariate model method19 because the CE-marked index test is a semi-quantitative real-time PCR technique and reports results for the same threshold for positivity.

Investigations of heterogeneity

We will investigate the effects of patient characteristics (eg, aetiology of sepsis) and infection acquisition (community vs hospital) on test performance by incorporating covariates into the fitted models if sufficient individual studies are identified.

Sensitivity analysis

We will explore the potential effects of missing data using a range of assumptions. This will include the effects of both missing participant data from included studies and missing data from studies that did not provide data in a form that could be extracted and included in analysis.

Subgroup analysis

If adequate data are available, we will plan two subgroup analyses of adult versus paediatric populations and hospital versus community acquired infection.

Assessment of reporting bias

If there are sufficient studies included in analyses, we will investigate reporting biases using funnel plots based on effective sample size and regression tests of asymmetry, as recommended by Deeks and colleagues.20

Discussion

Blood culture technology is at the centre of evidence-based guidelines for the investigation and treatment of patients with sepsis. While culture has been refined over the last century, it remains insufficiently time critical and cannot assist with early management decisions, inevitably resulting in wasteful and potentially dangerous overtreatment with antimicrobial chemotherapy. PCR-based technologies have become standard laboratory techniques over the last two decades and could deliver real opportunity in terms of sensitivity and speed of pathogen detection in the clinical setting of life-threatening infection. LightCycler® SeptiFast is the first PCR-based system to be awarded a CE-mark for pathogen detection and identification in blood samples and, to date, is the most intensively investigated multiplex real-time PCR assay in the clinical setting of sepsis. The purpose of our planned systematic review is to determine, for the first time, the clinical diagnostic accuracy of this real-time PCR technology as part of an NIHR-funded HTA of this technology in the setting of potentially life-threatening infections. Based on the results of this non-commercial systematic review, independent recommendations will be made to the National Health Service providers as to whether LightCycler® SeptiFast has sufficient clinical diagnostic accuracy to move forward to efficacy and effectiveness testing during the provision of routine patient care.