Comparison of two commercial and one in-house real-time PCR assays for the diagnosis of bacterial gastroenteritis.

INTRODUCTION: The aim of the study was a comparative evaluation of in-house real-time PCR and commercial real-time PCR (Fast Track Diagnostics (FTD), ampliCube/Mikrogen) targeting enteropathogenic bacteria from stool in preparation of Regulation (EU) 2017/746 on in vitro diagnostic medical devices.
METHODS: Both 241 stool samples from patients and 100 samples from German laboratory control schemes ("Ringversuche") were used to comparatively assess in-house real-time PCR, the FTD bacterial gastroenteritis kit, and the ampliCube gastrointestinal bacterial panels 1&2 either with the in-house PCRs as gold standard and as a test comparison without gold standard applying latent class analysis. Sensitivity, specificity, intra- and inter-assay variation and Cohen's kappa were assessed.
RESULTS: In comparison with the gold standard, sensitivity was 75-100% for strongly positive samples, 20-100% for weakly positive samples, and specificity ranged from 96 to 100%. Latent class analysis suggested that sensitivity ranges from 81.2 to 100% and specificity from 58.5 to 100%. Cohen's kappa varied between moderate and nearly perfect agreement, intra- and inter-assay variation was 1-3 to 1-4 Ct values.
CONCLUSION: Acceptable agreement and performance characteristics suggested replaceability of the in-house PCR assays by the commercial approaches.

BACKGROUND

With impeding enactment of Regulation (EU) 2017/746 on in vitro diagnostic medical devices, in-house diagnostic assays will have to be replaced by commercially available assays unless an appropriate performance level cannot be met by assays available on the market. At present, well-established molecular diagnostic assays are available for many target organisms with relevance for the microbiological laboratory. However, not all assays are accredited for all diagnostic platforms and nucleic acid extraction strategies, making customized diagnostic solutions challenging if stockpiling of a broad variety of different diagnostic platforms within the diagnostic laboratory shall be avoided.

For the diagnosis of infectious enteritis, broad-spectrum multiplex real-time PCR assays have been developed and evaluated [1]. While such assays were formerly applied for screenings or outbreak investigations in settings, where traditional culture-based diagnostic approaches were unfeasible due to logistic reasons [2], they have nowadays been established in the routine diagnostic laboratory for cost saving purposes. If only PCR-positive samples are subjected to labour-intensive diagnostic culture in hospital laboratories, the workload of skilled and thus expensive laboratory personnel can be reduced. This strategy is not unusual in hospital laboratories.

For example, at the Tropical Microbiology Subdepartment of the Bundeswehr Hospital Hamburg, Germany, an in-house real-time PCR targeting Salmonella spp., Shigella spp./enteroinvasive Escherichia coli (EIEC), Campylobacter jejuni, and Yersinia spp. [3] next to a Phocid Herpes virus DNA-base internal control [4] on RotorGene 6000/Rotor-Gene Q cyclers (Qiagen, Hilden, Germany) after standardized nucleic acid extraction from stool samples applying the QIAamp DNA Stool Mini Kit (Qiagen) has been used for more than a decade for the screening of stool samples provided by military returnees from the tropics. This in-house assay has a number of disadvantages. Associated with low melting temperatures of the oligonucleotides, high limits of detection of 104 to 105 bacterial copies have been recorded [3]. In comparison, commercial assays targeting Shigella spp./EIEC have shown considerably lower cycle threshold (Ct) values when applied with identical samples [2]. However, a previous evaluation under diagnostic real-life conditions had indicated acceptable sensitivity and specificity when applied with stool from patients with gastroenteritis [3].

However, with the enactment of Regulation (EU) 2017/746, which is expected for the middle of 2022, diagnostic use of in-house diagnostic approaches will be unfeasible if commercial CE-IVD-(Conformité Européenne – in vitro diagnostics-)accredited test assays with comparable performance characteristics are available. Accordingly, studies comparing commercially available test assays with the presently applied in-house diagnostic approaches are desirable and the results of respective studies may guide decisions regarding the future use of commercial test platforms.

In the study presented here, we have assessed two commercial real-time multiplex PCR assays targeting enteropathogenic bacteria with a well-established in-house real-time PCR protocol. Aim of the assessment was the proof of at least comparable performance characteristics to allow a switch to the commercial approaches in line with Regulation (EU) 2017/746 on in vitro diagnostic medical devices.

METHODS

Samples

Anonymized nucleic acid extraction residuals from stool samples assessed at the diagnostic laboratory of the Tropical Microbiology Subdepartment of the Bundeswehr Hospital Hamburg, Germany, located at the German National Reference Centre for Tropical Pathogens Bernhard Nocht Institute for Tropical Medicine Hamburg, were included in the test comparison. Those residual sample materials were either from screening assessments of German soldiers and policemen returning from tropical deployments [5, 6], from migrants travelling under poor hygiene conditions [7] or from studies conducted in resource-limited tropical settings (unpublished data) to ensure high proportions of positive samples for test comparison purposes. In addition, well-characterized sample materials from German external laboratory control schemes (“Ringversuche” by INSTAND e.V., Düsseldorf, Germany) were included. In total, 341 samples were assessed, comprising 241 patient samples and 100 “Ringversuch” samples. Details regarding the case definitions for the gold standard-based test comparison approach are presented in the “Case definitions” sub-heading below.

Nucleic acid extraction and sample storage

All diagnostic stool samples and “Ringversuch” materials had been subjected to standardized nucleic acid extraction applying the QIAamp DNA Stool Mini Kit-based nucleic acid extraction scheme as described by the manufacturer. The nucleic acid extractions had been stored frozen at –80 °C prior to the analyses.

Applied PCR schemes

The included residual materials from diagnostically assessed stool samples were assessed by the abovementioned in-house multiplex real-time PCR targeting Salmonella spp., Shigella spp./EIEC, C. jejuni, and Yersinia spp. as well as by an in-house real-time PCR targeting enterotoxigenic Escherichia coli (ETEC) on RotorGene Q cyclers (Qiagen, Hilden, Germany) as previously described [3, 8]. Both in-house PCR assays were applied with a Phocid Herpes virus DNA-based internal control [4]. In house testing was done in parallel with the commercial PCR runs to exclude effects of discrepant sample age.

Due to similar spectra of target pathogens as well as compatibility with the RotorGene cyclers and the QIAamp DNA Stool Mini Kit-based nucleic acid extraction scheme as confirmed by the manufacturers, comparatively assessed commercial real-time PCR assays comprised:

The Fast Track Diagnostics (FTD) bacterial gastroenteritis kit (Siemens, Berlin & Munich, Germany) targeting Campylobacter coli/jejuni/lari, Clostridioides difficile, enterohemorrhagic E. coli (EHEC), Salmonella spp., Shigella spp./EIEC, Yersinia enterocolitica, and an internal control.

The ampliCube gastrointestinal bacterial panel 1 (Mikrogen, Neuried, Germany) targeting Campylobacter spp., Salmonella spp., Y. enterocolitica as well as the ampliCube gastrointestinal bacterial panel 2 targeting EHEC, enterotoxigenic E. coli (ETEC), and Shigella spp./EIEC, both assays combined with an internal control.

Case definitions

For the purpose of a gold standard-based test comparison, samples from the diagnostic routine that had either been positive for any of the targets from the in-house multiplex real-time PCR targeting enteroinvasive pathogens described above [3] or for ETEC in another in-house real-time PCR [8] were included as positive controls. For the evaluation of the PCR targets C. difficile and EHEC, which had not been assessed in the diagnostic routine at our subdepartment so far, well-characterized samples from German external laboratory control schemes (“Ringversuche”) were used as positive controls.

Those well-characterized “Ringversuch” materials were also considered as negative control materials for PCR assays targeting infectious agents not abundant in the “Ringversuch” sample materials, while samples from the routine diagnostics were not considered as confirmed negative. Accordingly, only “Ringversuch” sample materials were used as negative controls in the gold standard-based test comparison.

Test comparison strategy with and without gold standard as well as statistics

Based on the abovementioned assumptions, a gold standard-based test comparison assessing diagnostic sensitivity, specificity, intra- and inter-assay variation as described previously [9] was performed. Thereby, strongly and weakly positive samples in the gold standard assessments were used for sensitivity assessments with cycle threshold (Ct) value 34 in the gold standard PCRs as the cut-off to discriminate between each other. In analogy to the detection thresholds of the in-house PCRs, copy numbers smaller than 105 were considered as weakly positive for the “Ringversuch” samples.

In addition to the gold standard-based assessment, the results from all samples were subjected to latent class analysis (LCA) [10, 11] as a test comparison without a gold standard next to calculation of Cohen’s kappa as described [12].

Ethics

As clarified previously (WF-011/19) by the Ethics Committee of the Medical Association of Hamburg, Germany, in line with German National Laws, use of anonymized residual sample materials for test comparison purposes in diagnostic laboratories is neither a “research project involving human beings” in line with § 9 chapter 2 of the “Hamburgisches Kammergesetz für Heilberufe” (Hamburg’s Association of Health Care Professions Act) nor a research project requiring ethical advice according to §15 chapter 1 of the “Berufsordnung für Hamburger Ärzte und Ärztinnen” (Professional Regulations for Physicians in Hamburg). Therefore, ethical clearance was not required for this assessment.

RESULTS

Results of the gold standard-based test comparison

In the gold standard-based assessment, the FTD assays’ sensitivity ranged from 75 to 100% for both the strongly and weakly positive samples, while the ampliCube assays showed 87.3–100% sensitivity for strongly positive and 20–100% sensitivity for weakly positive ones. Thereby, sensitivity of weakly positive samples could not be calculated for all parameters due to lacking appropriate sample materials. Gold standard-based specificity using the “Ringversuch” samples ranged from 98.6 to 100% for the FTD assays as well as 96.0 to 100% for the ampliCube assays. Inter- and intra-assay variation for the FTD and ampliCube assays were quite similar, ranging from 1 to 3 Ct values for in-house and FTD assays and from 1 to 4 Ct values for ampliCube assays. Again, samples with Ct values >34 were quite rare, so samples slightly lower but close to this value were accepted as weakly positive samples for this assessment as well.

Focussing on the FTD platform, sensitivity of 100% was recorded for strongly and weakly positive samples with C. jejuni and Shigella spp./EIEC, respectively. Sensitivity >90% was seen for strongly positive samples with Yersinia spp. and Salmonella spp., respectively, sensitivity >80% for both strongly and weakly positive samples with EHEC and sensitivity >70% for both strongly and weakly positive samples with C. difficile. Specificity was 100% with the exception of the assays for Y. enterocolitica (99.0%) and C. difficile (98.6%).

Focussing on the ampliCube platform, sensitivity of 100% was detected for weakly positive samples with C. jejuni as well as for strongly positive samples with ETEC. Sensitivity >90% was seen for strongly positive samples with C. jejuni, Yersinia spp., and Shigella spp./EIEC, respectively, sensitivity >80% for strongly positive samples with Salmonella spp. and EHEC, sensitivity ≥50% for weakly positive samples with Shigella spp./EIEC and EHEC, and sensitivity ≥20% for weakly positive samples with ETEC. Recorded specificity was 100% in all instances with the exception of the Campylobacter spp.-specific assay (96.0%).

Details of the gold standard-based assessment are provided in Tables 1 and 2.

Table 1.

Sensitivity and specificity compared with the diagnostic gold standard (in-house PCR or materials from laboratory control schemes)

Assay/target organism	Sensitivity with strongly positive samples in% (n/n)	Sensitivity with weakly positive samples in% (n/n)	Specificity in% (n/n)
FTD/Campylobacter coli/jejuni/lari	100.0 (97/97)	100 (2/2)	100 (100/100)
FTD/Clostridioides difficile	75.0 (6/8)	75.0 (12/16)	98.6 (73/74)
FTD/enterohemorrhagic E. coli (EHEC)	88.2 (15/17)	82.4 (14/17)	100 (72/72)
FTD/Salmonella spp.	94.9 (75/79)	n.a.	100 (97/97)
FTD/Shigella spp./enteroinvasive E. coli (EIEC)	100 (18/18)	100 (2/2)	100 (99/99)
FTD/Yersinia enterocolitica	91.7 (11/12)	n.a.	99.0 (99/100)
ampliCube/Campylobacter spp.	97.9 (95/97)	100 (2/2)	96.0 (96/100)
ampliCube/Salmonella spp.	87.3 (69/79)	n.a.	100 (97/97)
ampliCube/Yersinia enterocolitica	91.7 (11/12)	n.a.	100 (100/100)
ampliCube/enterohemorrhagic E. coli (EHEC)	88.2 (15/17)	58.8 (10/17)	100 (71/71)
ampliCube/enterotoxigenic E. coli (ETEC)	100 (21/21)	20.0 (1/5)	100 (97/97)
ampliCube/Shigella spp./enteroinvasive E. coli (EIEC)	94.4 (17/18)	50 (1/2)	100 (98/98)

Table 2.

Intra- and inter-assay-variation. Ct = cycle threshold

Assay/target organism	Assessed sample category	Intra-assay variation in total Ct-values	Inter-assay variation in total Ct-values
In-house enterotoxigenic E. coli (ETEC)	Strongly positive samplesWeakly positive samplesNegative samples	110	120
In-house Salmonella spp.	Strongly positive samplesWeakly positive samplesNegative samples	010	120
In-house Shigella spp./enteroinvasive E. coli (EIEC)	Strongly positive samplesWeakly positive samplesNegative samples	030	100
In-house Campylobacter jejuni	Strongly positive samplesWeakly positive samplesNegative samples	110	030
In-house Yersinia spp.	Strongly positive samplesWeakly positive samplesNegative samples	120	220
FTD/Campylobacter coli/jejuni/lari	Strongly positive samplesWeakly positive samplesNegative samples	020	120
FTD/Clostridioides difficile	Strongly positive samplesWeakly positive samplesNegative samples	030	120
FTD/enterohemorrhagic E. coli (EHEC)	Strongly positive samplesWeakly positive samplesNegative samples	020	220
FTD/Salmonella spp.	Strongly positive samplesWeakly positive samplesNegative samples	010	030
FTD/Shigella spp./enteroinvasive E. coli (EIEC)	Strongly positive samplesWeakly positive samplesNegative samples	010	120
FTD/Yersinia enterocolitica	Strongly positive samplesWeakly positive samplesNegative samples	110	100
ampliCube/Campylobacter spp.	Strongly positive samplesWeakly positive samplesNegative samples	100	120
ampliCube/Salmonella spp.	Strongly positive samplesWeakly positive samplesNegative samples	210	220
ampliCube/Yersinia enterocolitica	Strongly positive samplesWeakly positive samplesNegative samples	100	030
ampliCube/enterohemorrhagic E. coli (EHEC)	Strongly positive samplesWeakly positive samplesNegative samples	020	100
ampliCube/enterotoxigenic E. coli (ETEC)	Strongly positive samplesWeakly positive samplesNegative samples	140	120
ampliCube/Shigella spp./enteroinvasive E. coli (EIEC)	Strongly positive samplesWeakly positive samplesNegative samples	010	120

Results of the latent class analysis-based test comparison

As negative results in the gold standard PCRs, however, do not necessarily exclude the abundance of target DNA, the test results were subjected to a LCA-based test comparison approach without a gold standard in a next step for the parameters salmonellae, shigellae/EIEC, campylobacter, yersiniae, ETEC, and EHEC, respectively. Concerning C. difficile, such an approach was unfeasible, because this parameter was included in the FTD assay only. As shown in Table 3, calculated sensitivities were slightly higher, ranging from 81.2 to 100% with calculated specificities ranging from 58.5 to 100%.

Table 3.

Sensitivity and specificity as calculated by latent class analysis, mean as well as median cycle threshold values and agreement kappa. N = number. Ct = cycle threshold. SD = standard deviation. CI = confidence interval

PCR with target organism	N	Positives n (%)	Sensitivity (0.95 CI)	Specificity (0.95 CI)	Positives Ct mean (SD), median of the positives	Cohen’s kappa (0.95 CI)
Salmonella in-house	341	79(23)	0.945(0.861, 0.979)	0.988(0.963, 0.996)	22.36(5.09)22	0.839(0.771, 0.883)
Salmonella FTD	341	93(27)	0.985(0.903, 0.998)	0.947(0.911, 0.968)	22.24(7.20)22
Salmonella ampliCube	341	74(22)	0.908(0.817, 0.956)	0.996(0.966, 0.999)	21.00(6.52)20
Yersinia in-house	341	12(4)	1(0, 1)	0.997(0.979, 0.999)	21.83(3.43)22	0.888(0.744, 0.959)
Yersinia FTD	341	14(4)	1(0, 1)	0.990(0.972, 0.997)	23.78 (5.91)22.50
Yersinia ampliCube	341	11(3)	1(0, 1)	1(n.e.)	20.27(3.28)20
Campylobacter in-house	341	99(29)	0.832(0.753, 0.889)	1(0, 1)	21.75(4.76)21	0.542(0.468, 0.623)
Campylobacter FTD	341	119(35)	1(0, 1)	1(0, 1)	20.78(5.82)19
Campylobacter ampliCube	341	209(61)	0.983(0.935, 0.995)	0.585(0.519, 0.648)	23.38(6.62)23
Shigella/EIEC in-house	341	20(6)	0.902(0.673, 0.976)	1(n.e.)	21.40(7.23)19	0.803(0.669, 0.890)
Shigella/EIEC FTD	341	27(8)	1(0, 1)	0.985(0.962, 0.993)	22.25(8.15)21
Shigella/EIEC ampliCube	341	24(7)	0.899(0.676, 0.974)	0.987(0.966, 0.995)	21.83(8.56)19
ETEC in-house	341	29(9)	0.949(n.e.)	0.987(0, 1)	25.96(6.14)25	0.828(0.718, 0.938)
ETEC ampliCube	341	28(8)	0.956(0, 1)	0.991(0.963, 0.997)	18.82(4.87)17
EHEC FTD	341	36(11)	1(0, 1)	0.995(n.e.)	28.63(3.44)30	0.862(0.769, 0.956)
EHEC ampliCube	341	28(8)	0.812(0.630, 0.916)	1(0, 1)	30.46(3.78)31

Focussing on the FTD platform, calculated sensitivity of 100% was observed for all assays with the exception of the Salmonella spp.-specific assay (98.5%) and C. difficile-specific assay, for which this assessment was unfeasible. Calculated specificity was 100% for C. coli/jejuni/lari and >90% for all other parameters.

Focussing on the ampliCube platform, sensitivity of 100% was calculated for the Y. enterocolitica-specific assay, sensitivity >90% for the assays targeting Salmonella spp., Campylobacter spp., and ETEC, sensitivity >80% for the assays targeting EHEC and Shigella spp./EIEC. Calculated specificity was 100% for the assays targeting EHEC and Y. enterocolitica, >90% for the assays targeting ETEC, Shigella spp./EIEC and Salmonella spp., and >50% for the Campylobacter spp.-specific assay.

In line with this latter result, almost perfect agreement as defined by Landis and Koch [12] (Cohen’s kappa 0.81–1.00) could be shown for all assessed parameters with the exception of campylobacter, for which only moderate agreement (0.41–0.60 according to [12]) was recorded. Details including mean Ct-values are shown in Table 3.

DISCUSSION

As shown in this study, similar performances were observed for the commercial assays and the in-house approaches, making the commercial platforms suitable as likely replacements in response to Regulation (EU) 2017/746. Thereby, the overall performance of the FTD platform was slightly more promising than the performance of the ampliCube kits, but the differences were marginal.

The reduced sensitivity of the C. difficile assay was the major weakness of the FTD platform, while the low calculated specificity and the thus difficult interpretability of positive results of the Campylobacter spp. assay was the major weakness of the ampliCube platform. However, the single outlier with only 58.5% calculated specificity for the ampliCube Campylobacter spp. PCR most likely resulted from the fact that the ampliCube assay was genus-specific while species-specific sequences were the PCR targets of the chosen competitor assays. For the other PCR targets of the ampliCube platform, quite acceptable specificities ranging from 94.7 to 100% were observed. For both assessed commercial platforms, the observed phenomena should be considered when their diagnostic results are interpreted in the clinical setting.

While PCR was formerly primarily used for the detection of difficult to grow or culturally poorly discriminable pathogens like C. difficile [13-25], Camplycobacter spp. [26-29], or diarrheagenic E. coli [4, 30, 31], diagnostic real-time PCR as an initial screening tool for pathogen diagnostics from stool samples is increasingly used with broad detection panels [1, 32–36]. The here presented study is just a piece in the puzzle of the ongoing assessments of performance characteristics of those molecular tools and is meant to help estimating their usefulness but also the limits of their usefulness for the diagnostic routine.

Nevertheless, the study has a number of limitations. First of all, the study included a limited number of compared test assays adapted to the PCR platforms as available in our laboratory, while other and also broader syndromic PCR panels for stool samples are available as described [1, 32–36]. Secondly, only limited numbers of sample materials were available, limiting the interpretability of the results. Thirdly, due to the preselection of samples for the assessments, the methodological prerequisites for the application of LCA testing [10, 11] cannot be considered as completely fulfilled, so the results of those calculations have to be regarded as approximations only. In spite of those limitations, the data provided by the assessment can help to assess the performance characteristics of the compared assays in a diagnostic real-life setting.

CONCLUSION

As demonstrated by the test comparison, both commercial diagnostic approaches are suitable for a replacement of the previously applied in-house assay.