Diagnostic accuracy of loop-mediated isothermal amplification in detection of Clostridium difficile in stool samples: a meta-analysis.

INTRODUCTION: Clostridium difficile infection (CDI) remains a diagnostic challenge for clinicians. More recently, loop-mediated isothermal amplification (LAMP) has become readily available for the diagnosis of CDI, and many studies have investigated the usefulness of LAMP for rapid and accurate diagnosis of CDI. However, the overall diagnostic accuracy of LAMP for CDI remains unclear. In this meta-analysis, our aim was to establish the overall diagnostic accuracy of LAMP in detection of Clostridium difficile (CD) in stool samples.
MATERIAL AND METHODS: A search was done in PubMed, MEDLINE, EMBASE and Cochrane Library databases up to February 2014 to identify published studies that evaluated the diagnostic role of LAMP for CD. Methodological quality was assessed according to the quality assessment for studies of diagnostic accuracy (QUADAS) instrument. The sensitivities (SEN), specificities (SPE), positive likelihood ratio (PLR), negative likelihood ratio (NLR) and diagnostic odds ratio (DOR) were pooled statistically using random effects models. Statistical analysis was performed by employing Meta-Disc 1.4 software. Summary receiver operating characteristic (SROC) curves were used to summarize overall test performance. Funnel plots were used to test the potential publication bias. RESULT: A total of 9 studies met inclusion criteria for the present meta-analysis. The pooled SEN and SPE for diagnosing CD were 0.93 (95% CI: 0.91-0.95) and 0.98 (95% CI: 0.98-0.99), respectively. The PLR was 47.72 (95% CI: 15.10-150.82), NLR was 0.07 (95% CI: 0.04-0.14) and DOR was 745.19 (95% CI: 229.30-2421.72). The area under the ROC was 0.98. Meta-regression indicated that the total number of samples was a source of heterogeneity for LAMP in detection of CD. The funnel plots suggested no publication bias.
CONCLUSIONS: The LAMP meets the minimum desirable characteristics of a diagnostic test of SEN, SPE and other measures of accuracy in the diagnosis of CD, and it is suitable as a rapid, effective and reliable stand-alone diagnostic test for diagnosis of CDI, potentially decreasing morbidity and nosocomial spread of CD.

Introduction

Clostridium difficile (CD) is a Gram-positive sporogenic anaerobic bacterium; it accounts for 15% to 30% of all episodes of antibiotic-associated diarrhea (AAD) and 95% to 100% of pseudomembranous colitis (PMC) [1, 2], with 6% mortality overall, rising to 13.5% in older patients [3]. The major CD virulence factors are enterotoxin TcdA (toxin A, 308 kDa) and cytotoxin TcdB (toxin B, 270 kDa), encoded along with three other genes (TcdC, TcdD, TcdE) in the pathogenicity locus (PaLoc) [4]. In addition to toxins A and B, some strains also produce a third toxin known as binary toxin, encoded by ctdA and ctdB, located outside the PaLoc. The role of binary toxin in the pathogenesis of C. difficile remains unclear [5, 6]. The morbidity, mortality and relapse rates of disease caused by CD have markedly increased in many parts of the world in the past decade [7, 8], and CD is now recognized not only as one of the major causes of serious healthcare-associated infections [9], but also as a community acquired infection [10, 11]. Increased Clostridium difficile infection (CDI) incidence and severity have been attributed largely to the emergence of a new strain of CD (BI/NAP1/027) [12], which is characterized by its expression of an ADP-ribosylating binary toxin [13]. The main risk factors for CDI are exposure to specific antibiotics, hospitalization and advanced age (age > 65 years) and so on [14, 15]. Therefore rapid and accurate diagnosis of CDI is essential both for improving outcomes of patients with CDI and for reducing horizontal transmission in health care facilities.

The laboratory diagnosis for CD consists of the detection of toxigenic CD and/or its toxins A or B in stool, for which a variety of methods are available, each with its own qualities and limitations. Currently, the gold standards for the diagnosis of CDI are the cell culture cytotoxicity neutralization assay (CCCNA) and anaerobic toxigenic culture (TC) [1, 16]. The CCCNA is sensitive but is extremely time-consuming, and its requirement for cell culture prohibits its application for near-patient rapid testing. Toxigenic culture is slow and laborious, often requiring 48 to 72 h to complete, and therefore is unlikely to be adopted by a clinical laboratory as the standard method for CD testing. The enzyme immunoassay (EIA) for detection of toxins A and B has been the most widely used diagnostic test for CDI because of its rapid turnaround, low cost and simplicity, and toxin gene expression is known to be repressed [16]. However, EIA for toxins A and B is known to have low sensitivity [1, 16–18]. With the development of an EIA for glutamate dehydrogenase (GDH), which is a C. difficile cell wall common antigen, the sensitivity for the detection of C. difficile approaches 100%; however, because GDH is ubiquitous for both toxigenic and non-toxigenic strains, specificity of the GDH assay is poor. The polymerase chain reaction (PCR) assay for detecting the toxin genes has been widely used for identification of types of toxin produced by recovered isolates. Detection of tcdA and/or tcdB in stool specimens by PCR, nested PCR, and real-time PCR has also been developed and evaluated. Although reported to be rapid and sensitive diagnostic methods, they are not necessarily of practical use in clinical laboratories, where special equipment such as a thermal cycler and detection systems are not available. Therefore CDI remains a diagnostic challenge for clinicians.

More recently, loop-mediated isothermal amplification (LAMP) [19-24] has become readily available for the diagnosis of CDI, and compared to other non-culture-based methods, LAMP is a sensitive and specific method [25], although more expensive than traditional assays [26]. This test has potential for rapid and accurate diagnosis and was released by the FDA in July 2010 for US laboratory use.

We performed a meta-analysis to investigate the performance of LAMP assays for diagnosis of CDI when compared with reference standards of CCCNA or TC.

Material and methods

Publication search

This meta-analysis was conducted according to guidelines for diagnostic meta-analysis [27, 28]. Web of Science, EMBASE, Cochrane Database, MEDLINE, PubMed and CBM were all searched (the last search was updated on February, 2014) using the strategy of (Clostridium difficile OR C. difficile OR C. diff OR CD AND loop-mediated isothermal amplification OR LAMP). The search was limited to clinical studies involving human patients, with a diagnosis or suspected diagnosis of CDI. No language or publication date restrictions were applied to the search. All the searched studies were retrieved, and their references were also checked for other relevant publications. We also reviewed articles to find additional eligible studies.

Inclusion and exclusion criteria

Two investigators independently reviewed the titles and abstracts of all the records searched above. For records which could not be evaluated through the titles and abstracts, full texts were retrieved for detailed evaluation according to the inclusion and exclusion criteria. Disagreements about particular studies were discussed and resolved. The reasons why studies were excluded were listed. Authors of studies were contacted when the information was not available in the published study. Studies meeting the following selection criteria were included in this meta-analysis: (1) studies about LAMP in detection of CD; (2) performed stool specimen analyses from inpatients or outpatients; (3) compared LAMP to a reference method, either CCCNA or TC; (4) reported total number of patients tested and positive/negative results that allow calculation of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN); (5) studies with more than 20 patients because a small sample size may be vulnerable to selection bias. We excluded studies if (1) all samples were not tested by at least 1 reference test, that is, CCCNA or TC; (2) the reference test was performed only on a subset of samples, that is, only positives, negatives, or those that were discordant; (3) the reference test was a combination of > 1 diagnostic test; (4) they involved animal studies or laboratory cultures of CD; (5) studies that focused on detecting virulence of different CD ribotypes; (6) LAMP tests were confirmed using other diagnostic tests but not CCCNA or TC; (7) studies with duplicate data reported in other studies; (8) studies that were conference abstracts, letters, editorials, case reports or case series, basic research studies and review articles. To assess trial methodology, included publications were reviewed independently by two authors and given a quality score by using the QUADAS (quality assessment for studies of diagnostic accuracy, an evidence-based quality assessment tool to be used in systematic reviews of diagnostic accuracy studies, maximum score 14) tools [29].

Data synthesis and meta-analysis

Data were analyzed using the freeware program Meta-DiSC (version 1.4) [30]. We used standard methods recommended for meta-analysis of diagnostic test evaluations [31]. The statistical analysis was based on the following steps:

Searching for the presence of heterogeneity: heterogeneity was explored by the chi-square (χ2) test and assessed using I2 analysis, where 0% indicates low heterogeneity and 100% indicates high discordance between studies [32]. Statistical tests were two sided and significance was set at p < 0.05.

Testing of the presence of cut-off threshold effects: estimates of diagnostic accuracy differ if not all studies use the same cut-off point for a positive test result or for the reference standard. In the case of diagnostic tests with a continuous or ordinal outcome, the receiver operating characteristic (ROC) curve presents pairs of sensitivity and specificity for different values of the cut-off point of a test. One test for the presence of a cut-off point effect between studies by calculating a Spearman correlation coefficient between sensitivity and specificity of all included studies should be taken [31].

Dealing with heterogeneity: a random effects model was used for the meta-analysis to obtain a summary accuracy parameter if heterogeneity was identified; if heterogeneity due to a threshold effect was present, the accuracy data should be pooled by fitting a summary receiver operating characteristic (SROC) curve and calculating the area under the curve (AUC). If there was no threshold effect but significant heterogeneity, subgroup analyses were conducted using meta-regression to determine the contribution of individual factors on heterogeneity, where p < 0.05 indicates a contribution to heterogeneity.

Statistical pooling: we calculated several diagnostic accuracy measures by pooling data from all data series using a fixed effects model according to the Mantel-Haenszel method and random effects model based on the work of Der Simonian and Laird [33]. The following measures of test accuracy were computed for each study: sensitivities (SEN), specificities (SPE), positive likelihood ratio (PLR), negative likelihood ratio (NLR) and diagnostic odds ratio (DOR). Analysis was based on a SROC curve.

Publication bias: we tested for the potential presence of this bias by using funnel plots [34]. Publication bias is assessed visually by using a scatter plot of the inverse of the square root of the effective sample size versus the diagnostic log odds ratio, which should have a symmetrical funnel shape when publication bias is absent [35].

Results

Literature search

The results of the literature research are presented in Figure 1. The initial search yielded a total of 548 potential relevant studies. After the review of titles and abstracts, 534 articles were excluded, and then 14 full manuscripts were retrieved for detailed evaluation of the overall diagnostic accuracy of LAMP in detection of CD in the stool samples. Of these articles, 1 article was excluded because the LAMP test was performed only for the samples that had discordant EIA results (GDH+/toxin–) [36]; 1 article was excluded because the LAMP test was performed as part of a 2-step glutamate dehydrogenase assay [20]; 3 articles were excluded because samples were not tested by at least 1 reference test [19, 20, 37]. As a consequence, only 9 articles were considered to be eligible for inclusion in the analysis according to the inclusion and exclusion criteria [22, 23, 38–44].

Figure 1

Flow diagram of study selection process

Baseline characteristics

The main characteristics of the studies included in the meta-analysis are shown in Table I. The studies included were conducted in different countries, and the publication years ranged from 2005 to 2014. The total number of stool samples from patients at risk of CDI enrolled in the studies was 3621, ranging from 74 to 986 per study. The prevalence of CDI across all studies ranged from 7.5% to 62.7%.

Table I

Characteristics of included studies

First author	Year of publication	Country of origin	Total no. of samples	Reference test	Prev. CDI (%)	TP	FN	FP	TN
Ylisiurua [38]	2013	Finland	430	TC	40.9	172	4	1	253
Pancholi [39]	2012	USA	200	CCNA	17.5	20	15	3	161
Bamber [40]	2012	USA	810	TC	10.2	76	7	14	713
Lalande [23]	2011	French	472	TC	10.4	45	4	4	419
Noren [41]	2013	Sweden	302	TC	29.1	84	4	2	212
Bruins [42]	2012	England	986	TC	7.5	68	5	3	906
Norén [43]	2011	Sweden	272	TC + CCNA	13.2	36	0	14	222
Kato [22]	2005	Japan	74	TC	54.1	38	2	10	24
McElgunn [44]	2014	USA	75	TC + CCNA	62.7	36	11	0	27

CCCNA – Cell culture cytotoxicity neutralization assay, TC – toxigenic culture.

Assessment of study quality

Methodological quality of all 9 studies included was evaluated according to the QUADAS guidelines. In the total included studies, more than 89% of the publications had high quality on the representative spectrum, and more than 67% had high quality in the items of uninterpretable results reported. In addition, nine items (selective criteria clearly described, acceptable reference standard, acceptable delay between tests, partial verification avoided, differential verification avoided, incorporation avoided, adequate reference standard description, adequate index test description, reference standard interpretation bias) had 100% high quality. However, only one of the 9 eligible studies showed the item of the index test results blinded [23], and all of the 9 eligible studies showed the item of the relevant clinical information and withdrawals explained unclearly.

Threshold effect

Computation of the Spearman correction coefficient between the logic of SEN and logic of 1-SPE of LAMP was –0.283 (p = 0.460), indicating no threshold effect that could cause variations in accuracy estimates among the individual studies.

Diagnostic accuracy

Figure 2 shows the forest plot of the SEN of the included studies. The range for SEN was 0.77–0.98 (mean: 0.93, 95% CI: 0.91–0.95). Figure 3 shows the forest plot of the SPE; the range for SPE was 0.71–1.00 (mean: 0.98, 95% CI: 0.98–0.99). Figure 4 shows the forest plot of the PLR; the range for PLR was 3.23–282.25 (mean: 47.72, 95% CI: 15.10–150.82). Figure 5 shows the forest plot of the NLR; the range for NLR was 0.02–0.24 (mean: 0.07, 95% CI: 0.04–0.14). Figure 6 shows the forest plot of the DOR; the range for DOR was 45.60–10879.00 (mean: 745.19, 95% CI: 229.30–2421.72). Q values of SEN, SPE, PLR, NLR and DOR were 25.62, 88.98, 131.39, 33.77 and 39.33 respectively, with significant heterogeneity (I2 = 68.8%, 91.0%, 93.9%, 77.6% and 79.7%), with the values of the χ2 test < 0.01, indicating significant heterogeneity among the included studies. Figure 7 presents the SROC curve for the included studies; the AUC and the Q* were 0.98 and 0.94, which demonstrate excellent accuracy.

Figure 2

Forest plot of sensitivity for LAMP in the detection of Clostridium difficile. The point estimates of sensitivity from each study are shown as solid circles. Error bars indicate 95% confidence intervals

Figure 3

Forest plot of specificity for LAMP in the detection of Clostridium difficile. The point estimates of specificity from each study are shown as solid circles. Error bars indicate 95% confidence intervals

Figure 4

Forest plot of PLR for LAMP in the detection of Clostridium difficile. The point estimates of PLR from each study are shown as solid circles. Error bars indicate 95% confidence intervals

Figure 5

Forest plot of NLR for LAMP in the detection of Clostridium difficile. The point estimates of NLR from each study are shown as solid circles. Error bars indicate 95% confidence intervals

Figure 6

Forest plot of DOR for LAMP in the detection of Clostridium difficile. The point estimates of DOR from each study are shown as solid circles. Error bars indicate 95% confidence intervals

Figure 7

The SROC curve for LAMP in the detection of Clostridium difficile

Meta-regression and sub-group analyses

The meta-regression and sub-group analyses were used to explore the overall heterogeneity and the possible sources of heterogeneity, which include type of reference test (TC and other), estimated prevalence of CDI (< 20% and ≥ 20%), and the total number of samples (< 200 and ≥ 200). Meta-regression indicated that the total number of samples was the source of heterogeneity for LAMP in detection of CD (p = 0.0356) (Table II). Subgroup analyses were conducted based on the total number of samples (< 200 and ≥ 200) (Table III), which showed that the diagnostic accuracy of the total number of samples more than 200 was better than that less than 200. In addition, when we excluded studies in which the total number of samples was less than 200, the I2 for heterogeneity of SEN decreased from 68.8% to 23.4%, NLR decreased from 77.6% to 13.8%, while SPE (90.0% to 91.8%), PLR (93.9% to 95.4%) and DOR (79.7% to 80.2%) had minimal change.

Table II

Meta-regression (Inverse Variance Weights)

Var	RDOR	Coeff	Value of p
Total number of samples	0.03	–3.501	0.0356
Reference test	2.78	1.021	0.3539
Prev-CDI	0.29	–1.244	0.1848

Table III

Diagnostic accuracy by the total number of samples

Samples	SEN (95% CI)	SPE (95% CI)	PLR (95% CI)	NLR (95% CI)	DOR (95% CI)
< 200	0.88 (0.80–0.93)	0.96 (0.93–0.97)	25.81 (4.6–144.72)	0.11 (0.02–0.55)	297.53 (28.28–3129.77)
≥ 200	0.95 (0.93–0.97)	0.99 (0.98–0.99)	64.33 (12.40–333.64)	0.06 (0.04–0.09)	1106.67 (285.86–4284.33)

Publication bias

The funnel plot for publication bias was symmetric (Figure 8), indicating no potential publication bias.

Figure 8

The funnel plot for publication bias

Discussion

Loop-mediated isothermal amplification is a novel nucleic acid amplification method using DNA polymerase with strand displacement activity and six primers that recognize eight regions on the target nucleic acid, leading to extremely high SPE [45, 46]. The Illumigene test targets within the PaLoc, a conserved 204-bp region of the conserved 5’ sequence of the CD tcdA gene based on the LAMP technology [47], which is different from those of other real-time PCR-based methods that mostly focus on the tcdB gene. Variability in genes coding for toxins is not a rare phenomenon. Stamper et al. found that 21.5% of CD strains are variant for the toxin A and B genes [48] and that tcdA was more conserved than tcdB. One potential concern regards the detection of A B+ variant strains, which belong to toxin type VIII or X, and they represent 3.9% of CD isolates in a recent pan-European survey of CD infection [8]. Actually, the target 5’ region of tcdA within PaLoc is intact in all strains, including those with a large deletion in the tcdA gene. Coyle et al. recently reported that Illumigene CD was positive in stools spiked with A B+ strains from toxinotypes VIII and X [49].

A variety of methods are available to detect CDI. The Illumigene CD assays showed greater SEN and quicker TATs (45 min and 1 h, respectively) compared to the CCCNA (the median TAT for the positive specimens was 24 h (range: 6–72 h)). The combination of a quick turnaround time with high performance might result in better management of CDI and timely implementation of infection control measures. The duration of hospital stay for patients infected with CD is believed to be the most influential contributor to increased hospital costs [50]. Early and accurate detection of CDI is important to ensure that the patient receives appropriate therapy and spends less time in the hospital [51]. With accurate diagnosis, infection control measures can be initiated to interrupt CD transmission to other patients and may result in additional healthcare cost savings. What is more, LAMP can be performed in any laboratory without special requirements such as separate pre- and post-PCR rooms, which are necessary for real-time PCR or other PCR-based techniques, and LAMP cost-efficiency ($26) compared to the Xpert C. difficile assay ($46).

Our findings indicate that LAMP is a useful diagnostic test with a high degree of accuracy on the basis of SEN (0.93) and SPE (0.98) statistics. SEN and SPE are true performance statistics for a test independent of disease prevalence in a population, and the major determinant for their values is that the cutoff differentiates positive from negative test results. Meanwhile, PLR (SEN/1-SPE) and NLR (1-SEN/SPE) were also calculated. PLR and NLR are used to evaluate how a study measure influences posttest probability using the Bayes theorem. For a positive test result, Pretest probability ×PLR = Posttest probability, and for a negative test result, Pretest probability ×NLR = Posttest probability. In our study the value of pooled PLR (47.72) is higher than 10, indicating that the positive result of the given test is useful for the confirmation of CDI, and the value of pooled NLR (0.07) is lower than 0.1, indicating that the negative result is useful for exclusion of CDI. As a single indicator measure of the diagnostic test accuracy that comprises a combination of SEN and SPE, the DOR describes the odds of positive test results in patients with CDI compared with the odds of positive results in those without the disease. It is calculated as DOR = PLR/NLR. In our study, the value of DOR was 745.19, which indicates better discriminatory test performance (higher accuracy). The SROC has been recommended to represent the performance of a diagnostic test, and the AUC is an alternative global measure of test performance, which is not only useful to summarize the curve, but also quite robust to heterogeneity [52, 53]. A prior study [54] showed that to demonstrate excellent accuracy, the AUC should be in the region of 0.97 or above. The AUC of our study was 0.98, which also demonstrates excellent accuracy. A summary measure of accuracy (Q*) was also calculated, which corresponds to the upper left-most point on the SROC curve, where SEN equals SPE. This value can be between 0 and 1, with 1 indicating the highest SEN/SPE. This value has been recommended over the AUC of greatest interest [53, 55]. So the SEN, SPE, DOR, LR and AUC data all support the use of LAMP for diagnosis of CDI as a highly discriminatory test when the test results were compared with CCCNA and TC. We also anticipated some degree of heterogeneity of diagnostic measures across studies, because of differences in the total number of samples, type of reference test and prevalence of CDI. We found high heterogeneity among studies (as defined by the I2 statistic) for all measures when the studies were pooled together. In this meta-analysis, the diagnostic threshold effect and publication bias did not introduce significant heterogeneity. We therefore performed meta-regression to investigate potential sources of the observed between-study heterogeneity. Then we found that the total number of samples was the source of heterogeneity for LAMP in detection of CD. SEN and SPE do not take into account the total number of samples in the tested population. Therefore, we sub-grouped the studies based on their total number of samples. When we excluded studies in which the total number of samples was less than 200, we found that the I2 for heterogeneity of SEN decreased from 68.8% to 23.4%, and NLR decreased from 77.6% to 13.8%, though SPE (90.0% to 91.8%), PLR (93.9% to 95.4%) and DOR (79.7% to 80.2%) had minimal change.

Furthermore, the limitations of this meta-analysis cannot be ignored. First, significant heterogeneity of diagnostic accuracy measures was expected and was found among studies, and the random effects model partially accounted for the between-study heterogeneity. However, we performed a limited number of subgroup analyses to reduce the degree of study heterogeneity. Second, we evaluated the publication bias by using a scatter plot of the inverse of the square root of the effective sample size versus the diagnostic log odds ratio in our study, and it has a symmetrical funnel shape showing that publication bias is absent, but it cannot evaluate the level of publication bias and the result varies when different people interpret it. Third, our meta-analysis did not adjust for differences in study variables, physician experience and training, institutional characteristics and so on.

Despite these limitations, LAMP seems to be a promising test according to current data. LAMP meets the minimum desirable characteristics of a diagnostic test of SEN, SPE, cost-efficiency, rapid results, ease of use preferably by non-expert users, and it is suitable as a rapid, effective and reliable stand-alone diagnostic test to be of practical use in many clinical laboratories for diagnosis of CDI, potentially decreasing morbidity and nosocomial spread of CD.