Comparative performance evaluation of Wako β-glucan test and Fungitell assay for the diagnosis of invasive fungal diseases.

The Fungitell assay (FA) and the Wako β-glucan test (GT) are employed to measure the serum/plasma 1,3-β-D-glucan (BDG), a well-known invasive fungal disease biomarker. Data to convincingly and/or sufficiently support the GT as a valuable alternative to the FA are yet limited. In this study, we evaluated the FA and the GT to diagnose invasive aspergillosis (IA), invasive candidiasis (IC), and Pneumocystis jirovecii pneumonia (PJP). The FA and GT performances were compared in sera of patients with IA (n = 40), IC (n = 78), and PJP (n = 17) with respect to sera of control patients (n = 187). Using the manufacturer's cutoff values of 80 pg/mL and 11 pg/mL, the sensitivity and specificity for IA diagnosis were 92.5% and 99.5% for the FA and 60.0% and 99.5% for the GT, respectively; for IC diagnosis were 100.0% and 97.3% for the FA and 91.0% and 99.5% for the GT, respectively; for PJP diagnosis were 100.0% and 97.3% for the FA and 88.2% and 99.5% for the GT, respectively. When an optimized cutoff value of 7.0 pg/mL for the GT was used, the sensitivity and specificity were 80.0% and 97.3% for IA diagnosis, 98.7% and 97.3% for IC diagnosis, and 94.1% and 97.3% for PJP diagnosis, respectively. At the 7.0-pg/mL GT cutoff, the agreement between the assays remained and/or became excellent for IA (95.1%), IC (97.3%), and PJP (96.5%), respectively. In conclusion, we show that the GT performed as well as the FA only with a lowered cutoff value for positivity. Further studies are expected to establish the equivalence of the two BDG assays.

Introduction

Invasive aspergillosis (IA), invasive candidiasis (IC), and Pneumocystis pneumonia (PJP, previously known as PCP) represent the most prevalent invasive fungal diseases (IFDs) worldwide [1]. These diseases mainly affect immunocompromised (or immunosuppressed) hosts, causing estimated over 1.6 million deaths annually [2]. Causative agents of IC include different Candida species [3], whereas the main cause of IA remains Aspergillus fumigatus [4] and PJP is uniquely caused by Pneumocystis jirovecii, formerly Pneumocystis carinii [5]. As IFD symptoms can be subtle and/or nonspecific, it is difficult to identify and treat the cause of disease, especially in patients with hematological malignancies [6]. Furthermore, microbiological confirmation of IFD with conventional, culture-dependent methods may yield false-negative results [7], hence, molecular, culture-independent methods to enhance the diagnostic sensitivity need to be developed [8]. Thus, pending the microbiological diagnosis, an empirical treatment targeting infectious and non-infectious causes may be necessary [6].

As broad fungal biomarker for IFD (the only notable exception are mucormycosis, cryptococcosis, and blastomycosis) [9-12], serum 1,3-β-D-glucan (BDG) has shown wide utility in specific clinical settings [13], including IA, IC, and PJP [14]. In a meta-analysis, He et al. focused on the accuracy of cutoff values to diagnose IFD obtained with BDG detection assays [15]. One was the Fungitell assay (FA; Associates of Cape Cod, East Falmouth, MA), FDA cleared and Conformité Européenne (CE) marked, which has been most used in the Western Hemisphere (Europe and United States), and another was the Wako β-glucan test (GT; FUJIFILM Wako Pure Chemical Corporation, Osaka, Japan), CE marked, which has recently been introduced in Europe.

Both assays rely on the BDG ability to activate factor G, a serine protease zymogen, in the Limulus (horseshoe crab) coagulation cascade. Activated factor G converts the inactive pro-clotting enzyme to the active form, which in turn cleaves an artificial substrate used for colorimetric (FA) or turbidimetric (GT) detection. Although BDG concentrations are measured through spectrophotometric readings, dissimilarity of cutoff values between the assays may be related to differences in the standards and/or affinity/reactivity of reagents in each assay [16]. Using the proposed 80 pg/mL (FA) and 11 pg/mL (GT) cutoff values [15], a recent comparison of the two assays for PJP diagnosis showed GT to be more specific and FA to be more sensitive, at a statistically significant level [17]. Interestingly, the sensitivity of GT equaled that of FA (at a cutoff of ≥60 pg/mL) and the specificity was significantly better than that of the FA, when the GT cutoff value lowered from 11 pg/mL to 3.616 pg/mL [17]. Consistently, previous work showed sensitivities of the FA for both IC (i.e., candidemia) and PJP diagnoses to be superior to those of the GT [18]. Again, lowering the GT cutoff value to ≥3.8 pg/mL resulted in sensitivities of the GT that became acceptable for candidemia (with a decreasing specificity from 98.0% to 91.0%) and excellent for PJP, respectively [18]. However, while more data are required to support the GT as a valuable alternative to the FA (especially for patients with candidemia), not enough data exist about the GT to diagnose IA.

Therefore, we compared the performance of the GT with that of the FA in well-characterized groups of patients with IA, IC, and PJP, with reference to appropriate control patients. Similar to previous studies [17, 18], we also tried to define the optimal GT cutoff values which could allow to reliably exclude IFD (mainly due to Aspergillus, Candida, or P. jirovecii).

Materials & methods

Ethics statement

This study was conducted at the Fondazione Policlinico Universitario A. Gemelli (FPG) IRCCS of Rome, Italy, and was approved by the Ethics Committee of the FPG (application number 38367/19) and a waiver of informed consent was granted. Sample processing and data analysis were performed anonymously.

Study design

We conducted a retrospective performance assessment study on archived patients’ serum samples, which had been collected as part of routine clinical care at the FPG IRCCS, a large tertiary care hospital in Rome, Italy. We used serum instead of plasma samples because of our previous findings showing that the two sample types may be considered equivalent [19]. Samples were from adult patients (mainly oncology/hematology patients) who were classified as having proven or probable IFD according to the 2008 European Organization for the Research and Treatment of Cancer/Mycoses Study Group (EORTC/MSG) criteria [20]. Samples were also from patients who were classified as having PJP if they had clinical signs and symptoms of respiratory infection (i.e., dyspnea, cough, or hypoxemia) supported with radiological findings (e.g., diffuse ground glass opacities on chest radiograph) and with positive Pneumocystis-specific PCR and/or immunofluorescence staining results of respiratory samples [21]. To enhance the robustness of the analysis, only patients with proven or probable IFDs were evaluated as IFD cases. Serum samples from patients who had major risk factors for IFD but who did not meet the criteria for proven or probable disease (i.e., who had no evidence of IFD) were included as controls. Furthermore, no diagnoses were based on detection of BDG (see below), which is part of the mycological criteria for the 2008 EORTC/MSG classification of patients with proven/probable/possible or no IFD [20]. At the time of writing this article—and shortly after completion the study, the EORTC/MSG education and research consortium updated definitions of IFDs [22]. However, according to the 2019 EORTC/MSG criteria, no reclassification was required for either IA or IC, while PJP cases were all classifiable as proven IFD, in our study [22].

We initially considered all the patients with a first serum BDG sample collected for routine mycological testing eligible. Then, we included only the patients for whom a classification as proven or probable IA (or IC) according to the 2008 EORTC/MSG definitions was available. All patients’ samples were from the closest time (± 2 days) when evidence or no evidence of IFD was obtained for cases and controls, respectively, and were stored at ‒80°C until the study time.

Serum BDG measurement

Before BDG testing, frozen serum samples were thawed at room temperature and briefly vortexed. The same investigator tested each sample’s aliquots with the FA and the GT in parallel and in a blinded manner for each patient’s disease classification at the time of testing. The two assays were performed in accordance with the manufacturer’s instructions. In both assays, the BDG measurement relies on a modification of the Limulus amebocyte lysate (LAL), which results in colorimetric (FA) or turbidimetric (GT) reaction changes. Briefly, for the FA 5 μL of serum (in duplicate) was used and the LAL reaction was monitored at 37°C for 40 min in an ELx808 microplate reader (BioTek Instruments, Winooski, VT). By comparing with a standard curve, the mean optical density change over time was calculated to determine the sample’s BDG concentration. A positivity threshold of 80 pg/mL was used throughout the study. For the GT, 100 μL of serum was used and the LAL reaction (i.e., gelation) was monitored at 37°C for a maximum of 90 min in a MT-6500 toxinometer (FUJIFILM Wako Pure Chemical Corporation). By comparing with a calibration curve (supplied with each lot by the manufacturer), the gelation time was calculated to determine the sample’s BDG concentration. A positivity threshold of 11 pg/mL was initially used in the study (see below for positivity threshold optimization). Samples with positive results of > 500 pg/mL (FA) or > 600 pg/mL (GT) were diluted and retested.

Data collection and statistical analysis

Statistical analysis was performed using GraphPad Prism version 8.2 (GraphPad Software, La Jolla, CA) and MedCalc Statistical Software version 19.0.7 (MedCalc Software bvba, Belgium). Patients’ demographics and BDG data were expressed as percentages or medians with interquartile ranges (IQR), respectively. To determine sensitivity and specificity of BDG assays, together with their respective 95% confidence intervals (CIs), we constructed 2 × 2 tables using IFD (IA, IC, or PJP) patients as true cases and non-IFD patients as controls. For both assays, receiver operating characteristic (ROC) curves were generated and cutoff values were derived to assess the diagnostic accuracy of serum BDG for case (IFD) versus control (non-IFD) patients. The highest Youden index indicated the optimal BDG cutoff. Variables were compared using chi-square test and Mann-Whitney U test, as appropriate, whereas differences in performance parameters between BDG assays were assessed using McNemar’s test. Agreement between BDG assays was determined by Spearman’s correlation, whereas the strength of agreement was determined by Cohen’s kappa statistic. Thus, values higher than 0.80 represented excellent agreement, values between 0.80 and 0.4 represented substantial to moderate agreement, and values lesser than 0.4 represented fair to slight agreement. Statistical significance was set at a < .05 P-value.

Results

Study samples

Table 1 shows demographics for the 322 patients from whom 322 serum samples were tested with two BDG assays (FA and GT), of which 78 samples were from proven IC (75 candidemia and 3 intraabdominal candidiasis) cases, 40 samples from probable IA (38 pulmonary and 2 disseminated aspergillosis) cases, and 17 samples from PJP cases. For all 135 cases, fungal disease classification (proven or probable) was obtained as previously described [20, 21]. We also included 187 samples from 187 patients, for whom clinical, radiographic, and microbiological findings indicated no evidence of IFD (controls). Table 2 shows details about the mycological evidence used to classify the 135 cases as IFD.

Table 1

Characteristics of 322 patients from whom serum samples were tested for BDG measurement.

	Value for patients with:
	IA (n = 40)	IC (n = 78)	PJP (n = 17)	No IFD (n = 187)
Median (interquartile range) age, years	53.0 (38.7–60.2)	52.5 (40.2–60.0)	52.0 (33.0–64.0)	54.0 (42.0–68.0)
Sex, male/female	18/22	34/44	7/10	90/97
Underlying condition, no. of patients (%)
Abdominal surgery	0 (0.0)	6 (7.7)	0 (0.0)	15 (8.0)
Solid tumor	9 (22.5)	22 (28.2)	2 (11.8)	23 (12.3)
Haematologic malignancy/HSCT	23 (57.5)	2 (2.6)	7 (41.2)	92 (49.2)
Otherb	8 (20.0)	48 (61.5)	8 (47.0)	57 (30.5)

BDG, 1,3-β-D-glucan; IA, invasive aspergillosis; IC, invasive candidiasis; PJP, Pneumocystis jirovecii pneumonia; IFD, invasive fungal disease; HSCT, hematopoietic stem cell transplantation.

a BDG testing was performed in parallel with the Fungitell assay (FA) and the Wako β-glucan test (GT).

b Includes patients in intensive care and infectious disease units.

Table 2

Mycological characteristics of IFD cases.

Characteristic	IA cases (N = 40)		IC cases (N = 78)		PJP cases (N = 17)
	n	%	n	%	n	%
Proven IFD
By culture	‒	‒	75	96.1	NA	‒
By histology	‒	‒	3	3.8	NA	‒
By immunofluorescence	NA	‒	NA	‒	17b	100
Probable IFD
By galactomannan antigen tested in:
Serum	37c	92.5	NA	‒	NA	‒
BALF	5d	12.5	NA	‒	NA	‒
CSF	2e	5.0	NA	‒	NA	‒

NA, Not applicable; BALF, bronchoalveolar lavage fluid; CSF, cerebrospinal fluid.

a All IFD cases were classified as described in the text.

b 15 cases were also PJP-specific PCR positive.

c The median (interquartile range) value of galactomannan index (optical density [OD] of sample/OD of cutoff control) was 1.6 (1.3–2.1). Indices of ≥0.5 were considered positive.

d Two cases had positive galactomannan detection results also in serum samples. Indices were 0.8 (serum) and 1.5 (BALF) in one case and 0.9 (serum) and 2.2 (BALF) in the other case.

e Two cases had positive galactomannan detection results also in CSF samples.

Performances of the FA and the GT

The FA (cutoff for positivity, ≥80 pg/mL) and the GT (cutoff for positivity, ≥11 pg/mL) gave positive results in 132 (97.8%) and 110 (81.5%) of 135 IFD samples, respectively. Of 187 control samples, 182 (97.3%) and 186 (99.5%) had negative results with the FA and the GT, respectively. The one patient with a false-positive result by the GT was highly positive by the FA. Overall, the median BDG concentration was 398 pg/mL with the FA and 31.63 pg/mL with the GT. In IA cases, the median (IQR) BDG concentrations were 201 (119–377) pg/mL and 14.07 (7.45–41.46) pg/mL for the FA and the GT, respectively. These levels were significantly higher than the levels in controls (0 (0–11) and 0 (0–0), respectively; P < .0001). Using the aforementioned cutoff values, the percentages of sensitivity and specificity were 92.5 and 99.5 for the FA, and 60.0 and 99.5 for the GT, respectively (Table 3). The sensitivity of FA was significantly greater than the sensitivity of GT (92.5% versus 60.0%, P < .001).

Table 3

Performance of the Fungitell assay and the Wako β-glucan test using indicated cutoffs to distinguish between IFD and non-IFD patients.

Parameter	Fungitell assay result (cutoff of ≥80 pg/mL)	Wako β-glucan test result (cutoff of ≥11 pg/mL)	Wako β-glucan test result (cutoff of ≥7 pg/mL)
Invasive candidiasis, n = 78
True positives	78	71	77
False negatives	0	7	1
True negatives	182	186	182
False positives	5	1	5
Sensitivity, % (95% CIb)	100.0 (95.3–100.0)	91.0 (82.6–95.5)	98.7 (93.0–99.9)
Specificity, % (95% CI)	97.3 (93.8–98.8)	99.5 (97.0–99.9)	97.3 (93.8–98.8)
Invasive aspergillosis, n = 40
True positives	37	24	32
False negatives	3	16	8
True negatives	182	186	182
False positives	5	1	5
Sensitivity, % (95% CI)	92.5 (80.1–97.4)	60.0 (44.6–73.6)	80.0 (65.2–89.5)
Specificity, % (95% CI)	97.3 (93.8–98.8)	99.5 (97.0–99.9)	97.3 (93.8–98.8)
Pneumocystis jirovecii pneumonia, n = 17
True positives	17	15	16
False negatives	0	2	1
True negatives	182	186	182
False positives	5	1	5
Sensitivity, % (95% CI)	100.0 (81.5–100)	88.2 (65.6–97.9)	94.1 (73.0–99.7)
Specificity, % (95% CI)	97.3 (93.8–98.8)	99.5 (97.0–99.9)	97.3 (93.8–98.8)

a Serum samples from 135 patients with invasive fungal disease (IFD) and from 187 patients without evidence of IFD (non-IFD controls) were tested with the Fungitell assay (FA) and the Wako β-glucan test (GT). The manufacturers’ cutoff values (FA, ≥80 pg/mL, and GT, ≥11 pg/mL) and, only for the GT, the optimized cutoff value (≥7 pg/mL) were used as sample’s positivity thresholds.

b CI, confidence interval.

In IC cases, the median (IQR) BDG concentrations were 516 (250–837) pg/mL and 45.57 (16.10–109.9) pg/mL for the FA and the GT, respectively. These levels were significantly higher than the levels in controls (P < .0001). Using the aforementioned cutoff values, the percentages of sensitivity and specificity were 100.0 and 97.3 for the FA, and 91.0 and 99.5 for the GT, respectively (Table 3). The sensitivity of FA was significantly greater than the sensitivity of GT (100.0% versus 91.0%, P < .001).

In PJP cases, the median (IQR) BDG concentrations were 512 (404–639) pg/mL and 46.88 (26.43–166.3) pg/mL for the FA and the GT, respectively. These levels were significantly higher than the levels in controls (P < .0001). Using the aforementioned cutoff values, the percentages of sensitivity and specificity were 100.0 and 97.3 for the FA, and 88.2 and 99.5 for the GT, respectively (Table 3). The sensitivity of FA was significantly greater than the sensitivity of GT (100.0% versus 88.2%, P = .01).

To improve the GT diagnostic performance, we determined the optimal positivity threshold for the GT (7.0 pg/mL) with the highest Youden index, which corresponds to the maximal sensitivity and specificity combination. As depicted in Fig 1, ROC analysis for proven/probable IFD versus no IFD at the optimized cutoff value generated an area under the curve (AUC) of 0.992 (95% CI, 0.975–0.999). Using the 7.0-pg/mL cutoff, the percentages of sensitivity and specificity for the diagnosis of proven/probable IFD were 80.0 and 97.3 for IA, 98.7 and 97.3 for IC, and 94.1 and 97.3 for PJP, respectively (Table 3). Only for IA, the sensitivity of FA remained significantly greater than the sensitivity of GT (92.5% versus 80.0%, P < .03). Similarly, we performed a ROC analysis for the FA using an optimized cutoff of 79.0 pg/mL, and this resulted in an AUC of 0.990 (95% CI, 0.972–0.998). Additionally, we tried to define whether a different cutoff value would be required for IA compared to IC or PJP. The values for GT determined with the highest Youden index were 2.6 pg/mL for IA and 7.0 pg/mL for both IC and PJP, whereas those for FA were 79 pg/mL for IA, 105 pg/mL for IC, and 88 pg/mL for PJP.

Fig 1

Receiver operating characteristic (ROC) curves of the GT (A) and the FA (B).

The optimized thresholds for positivity in both assays (7.0 pg/mL and 79 pg/mL, respectively) are marked with a dot.

Comparability of the FA and GT results

Assessing the quantitative agreement between the assays showed that a significant correlation between the BDG levels measured by the FA and the GT for IFD and non-IFD patient samples (Fig 2). In particular, these levels correlated significantly—although differently—for the IA (r = 0.670 and P < .0001; n = 40), IC (r = 0.703 and P < .0001; n = 78), and PJP (r = 0.667 and P = .004; n = 17) patient samples. Assessing the qualitative agreement between the assays showed that an excellent (or substantial) concordance regarding the results (positive/negative) obtained with the FA (cutoff for positivity, ≥80 pg/mL) and GT (cutoff for positivity, ≥11 pg/mL). This occurred with the samples from patients with IFD (91.8% agreement; Cohen’s kappa statistic, 0.82), IA (91.6% agreement; Cohen’s kappa statistic, 0.67), IC (95.0% agreement; Cohen’s kappa statistic, 0.88), and PJP (96.0% agreement; Cohen’s kappa statistic, 0.76), respectively. Using the optimized GT cutoff value (≥7.0 pg/mL), the concordance between the assays remained and/or became perfect for the samples from patients with IFD (95.6% agreement; Cohen’s kappa statistic, 0.91), IA (95.1% agreement; Cohen’s kappa statistic, 0.83), IC (97.3% agreement; Cohen’s kappa statistic, 0.93), and PJP (96.5% agreement; Cohen’s kappa statistic, 0.81), respectively.

Fig 2

Correlation between the BDG concentrations (pg/mL) determined by the FA and GT assays.

Discussion

To the best of our knowledge, two retrospective case-control studies have carefully evaluated the GT—the last approved BDG assay on the European market of s—in comparison with the FA in serum samples [17, 18]. Using a similar study design, we compared the GT with the FA to diagnose IFD in 322 patients categorized into IC (n = 78), IA (n = 40), PJP (n = 17), and non-IFD (n = 187) groups. Indeed, we established the overall GT and FA performances in two patient groups (135 with IFD and 187 without IFD), as well as the specific GT and FA performances in the IA, IC, and PJP groups with respect to the non-IFD group. Importantly, to compare the GT with the FA results, we followed the strategy of testing all archived patient samples in parallel, which reduces the bias due to probability of BDG degradation during sample storage, as successfully shown [17]. In contrast, Friedrich et al. performed FA measurement at the time of sampling and GT measurement on long-term stored samples [18].

By applying the manufacturer’s recommended cutoffs for positivity, we showed that the sensitivity of the FA was higher than the sensitivity of the GT in IFD patients overall (97.8% versus 81.5%, respectively) and, in particular, IA (92.5% versus 60.0%, respectively), IC (100% versus 91.0%, respectively), or PJP (100% versus 88.2%, respectively) patients. While the PJP values in our study are very similar to those reported by Friedrich et al. in PJP patients (sensitivity of FA and GT were 100% and 88.9%, respectively) [18], our IC values differ substantially from those reported by Friedrich et al. in candidemia patients (sensitivity of FA and GT were 86.7% and 42.5%, respectively) [18]. However, our findings are surprising, particularly if compared to the results of several meta-analyses published over recent years [9-11]. The pooled sensitivity and specificity of BDG for invasive fungal infections (excluding P. jirovecii infections) varied from 76.0% and 85.0% [10], 76.8% and 85.3% [9] to 80.0% and 82.0% [11], respectively. Conversely, in one meta-analysis [11], the pooled sensitivity and specificity for PJP were 96.0% and 84.0%, respectively.

In agreement with the results by meta-analyses [9-11] but in marked contrast with those by Friedrich et al. [18], one study [23] showed a sensitivity of 67.0%—neighbor, albeit still distant, to ours—and a specificity of 93.0% for the GT in a similar-sized cohort of patients with candidemia. Like in the Friedrich et al.’s study [18], applying a decreased cutoff of 7 pg/mL allowed Dichtl et al. to obtain a sensitivity of 73.0%, while specificities in both the studies remained as high as 91.0% [18] and 93.0% [23]. Another study by Dichtl et al. [24] conducted in the setting of PJP, showed that the specificity of GT—using an 11-pg/mL cutoff—was 100% in 25 control individuals who tested negative for P. jirovecii DNA by quantitative real-time PCR from respiratory samples. In the same study [24], the GT sensitivity increased from 86.0% to 91.0% after exclusion of the cases with slightly positive PCR results—that are usually negative in microscopy. Thus, our data unravel unexpected features of both GT and FA, which seem to subvert the paradigms of specificity as a “FA weakness” and of sensitivity as a “GA weakness” [25].

The reasons for conflicting results among studies are unknown, but we cannot exclude the influence of factors as a possible explanation for the observed differences in BDG assays’ sensitivity and specificity. For example, findings from stratified (subgroup) analyses in two independent meta-analysis studies led to observe a lower EORTC/MSG diagnostic accuracy attributable to lower sensitivity compared to similar criteria (e.g., histopathological examination and/or microbiological culture from blood or sterile material) [11, 15]. In the present study, we used the 2008 EORTC/MSG criteria for both IA and IC [20], and 3 (1.3%) of 78 patients had an IC diagnosis based on histology only. Unlike us, Friedrich et al. [18] exclusively used mycological culture from blood as a reference standard. Although blood culture serves to document the proven presence of invasive fungal infection [20], three patients with false-positive BDG results in the study by Friedrich et al. had negative blood cultures but were positive for the mannan antigen [18], which is a highly specific serum biomarker for IC [26]. Furthermore, fungal antigen levels in the blood are generally higher in culture-positive cases, so the inclusion of patients based on a blood culture positive may not allow a correct appraisal of the BDG performance [23]. In our study, the median (IQR) value of galactomannan OD index in 40 patients diagnosed with probable IA was 1.6 (1.3–2.1), giving a measure of moderately high fungal burdens in these patients. This could indicate later infection stages in these patients—that in turn could result in higher BDG values—unless the exposure to mold-active antifungal prophylaxis in many patients could have masked their actual galactomannan levels [8]. Ultimately, for all analyses, we used a mixed control group of patients with different (albeit partially overlapping) risk factors for IFD [8, 27], which could skew the BDG diagnostic parameters. To exclude this possibility, we performed a rough analysis using control subgroups. Interestingly, we found no one or slight differences in specificities when they were calculated with patients only at risk of IC (97.6% [FA] and 99.4% [GT]), IA (100% for both FA and GT), or PJP (99.1% [FA] and 100% [GT]) (data not shown).

Unlike IA, where the low sensitivity (60.0%) makes the GT not suitable for diagnosis using the manufacturer’s recommended cutoff value of 11 pg/mL, the sensitivities of the GT in IC (91.0%) and PJP (88.2%) were already good without lowering the cutoff value. However, the GT sensitivity in IA increased to 80.0% with a positivity threshold of 7.0 pg/mL and without compromising specificity (97.3%). Applying the 7.0-pg/mL cutoff value, the GT sensitivity also increased for both IC (from 91.0% to 98.7%) and PJP (from 88.2% to 94.1%), whereas at this cutoff the specificity for both IC and PJP slightly decreased (from 99.5% to 97.3%). Interestingly, the GT cutoff value proposed in this study was almost twice the value proposed elsewhere, such as 3.8 in candidemia [18] and 3.616 in PJP [17]. Of note, our value also encompasses IA, and this is not surprising because BDG testing for yeast or yeast-like fungi such as Candida species and P. jirovecii may require a cutoff value different from that required for molds such as Aspergillus species. Accordingly, cutoffs to optimize the performance of the BDG assay may be different depending on the pathogen and host [14]. Therefore, it should be mandatory to interpret BDG results in relation to not only the specific pathogen, but also the specific patient population [28].

Our comparison of two BDG assays showed that both quantitative correlation and qualitative agreement between the FA and the GT were very good. However, one of the assays failed to categorize patients correctly. In particular, 13 IA, 7 IC, and 2 PJP samples provided false negative results only with the GT, because the BDG content in these samples did not reach the positivity threshold recommended by the manufacturer. Lowering the GT cutoff value allowed us to recover 5 IA, 6 IC, and 1 PJP samples as true positive samples, thereby indicating that the manufacturer’s cutoff value might not be appropriate for patients with IA, IC, and PJP. This was in agreement with the observations reported elsewhere [17, 18].

With the present study, we aimed to provide further data about the role of serum BDG in the diagnosis of IFDs by means of a comparative assessment of both assays in terms of sensitivity and specificity using a large number of patient samples. However, we acknowledge the unavoidable limitations of the study. First, we did not explore whether lower sensitivity of the GT, particularly in IA samples, could be due to the use of serum instead of plasma, which the GT manual has listed as a principal specimen for clinical investigation. Nevertheless, we previously showed that serum samples effectively equate plasma samples for BDG measured with GT in patients with probable or proven fungal diseases [19]. Second, we did not assess the predictive value of both assays because of the artificially high prevalence of IFD in our test population influencing the value. However, we well know that, at least for candidemia, high sensitivity and negative predictive value for serum BDG are both helpful to exclude disease in order to withhold treatment [29]. Third, we waived to compare both assays for important features such as their layout and workflow. However, we experimented that GT is technically less complex than the FA regarding the possibility either of testing samples individually or up to 16 samples in parallel, as well as the use of a standard curve provided by the manufacturer and, above all, more simplicity to execute the assay. Fourth, unlike previous studies [17, 18], the number of PJP case tested by us was very small.

In conclusion, while confirming the good diagnostic accuracy of serum BDG assay [30], our findings support and/or extend the diagnostic value of both FA and GT into clinical settings such as IA, IC, and PJP. We show that the GT performed almost as the FA after optimizing the GT cutoff value for positivity. Further studies are yet necessary to establish the equivalence of both assays in order to provide equally reliable BDG results that can ultimately help clinicians with hard-to-diagnose fungal diseases.

(CSV)

Click here for additional data file.

(CSV)

Click here for additional data file.

(CSV)

Click here for additional data file.

(CSV)

Click here for additional data file.

1 Apr 2020

PONE-D-19-33241

Comparative performance evaluation of Wako β-glucan test and Fungitell assay for the diagnosis of invasive fungal diseases including Pneumocystis pneumonia

PLOS ONE

Dear Dr. Sanguinetti,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I would ask that you address the comments raised by the reviewers, in particular I would ask that you fully address the comments and questions raised regarding the significant improvements of the Glucan Test and Fungitell Assay compared to the published literature.

We would appreciate receiving your revised manuscript by May 14 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Jeffrey Chalmers, Ph.D.

Academic Editor

PLOS ONE

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Financial Disclosure section:

"M.S. received funding from FUJIFILM Wako Pure Chemical Corporation to perform this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

We note that you received funding from a commercial source: FUJIFILM Wako Pure Chemical Corporation which produces the Wako Beta-Glucan test.

Please provide an amended Competing Interests Statement that explicitly states this commercial funder, along with any other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc.

Within this Competing Interests Statement, please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your amended Competing Interests Statement within your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

Reviewer #3: N/A

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This article addresses the diagnostic cutoffs for the available fungal diagnostic tests. It suggests new cutoffs that improve detection for those tests, which is in agreement with previous data. The change in cutoff is very small however. This article may be relevant for the field.

Reviewer #2: Thank you for the opportunity to review this manuscript.

Carolis and colleagues present a well- performed study demonstrating the performance of the Wako beta glucan assay in comparison to the Fungitell assay. Overall, the study design is robust and performance of the study is well explained.

English language editing can be sought to improve the readability of the manuscript.

Specific comments:

- Line 45: suggest reword the opening sentence to: "Invasive Aspergillosis, IC and PJP represent the most prevalent invasive fungal infections worldwide".

- Line 46: reword "previously known as" to "also known as". PCP is still being used as a term for the disease

- Line 51: Suggest remove "e.g. respiratory"

- Line 51: non-specific, not unspecific

- Line 54: Suggest reword- instead of unsuccessful, rather mention that culture methods have poor sensitivity or yield false negative results

- Line 56: reword as empiric treatment may be necessary. (the reference used here relates specifically to respiratory disease in patients with haematological disease). In many other cases of invasive fungal disease, it may not be appropriate to initiate empiric therapy

- Line 57: Reword as a "broad fungal biomarker", rather than almost universal

- Study design: additional detail about the patient population at risk of IFD at this hospital. From the data, it appears that most cases are seen in Oncology. Were these only adult patients, or were paediatric patients included as well?

- Line 97- 98: elaborate on the negative samples chosen, since the EORTC/MSG guideline does not have a specific definition for what is considered "no evidence of IFD".

- Methods section: Serum BDG measurement- the fact that plasma is the samples of choice for the Wako assay should be mentioned to the reader in the methods, as well as the fact that a serum had not been validated on the assay before performance of this study. The only place this is mentioned is in the limitations section of the discussion.

- Line 263: hard to diagnose, not hard to diagnosis

- Additional limitations that require mentioning: the very small number of PJP cases.

- Table 1: Interquartile range should be added for patient age

- Line 341- patients "in" intensive care

- Table 2: It is not clear how the Wako median and IQR BDG value can be 0 (0- 0) for the group "No evidence of IFD", particularly if there were cases of false positives detecting value above 7 and 11pg/ml. This aspect requires re- analysis.

- Figure 1: requires axis titles on both x and y axes

Reviewer #3: De Carolis and colleagues present a comprehensive comparison of the Fungitell assay (FA) and the Glucan test (GT) representing the two CE certified beta-1,3-D-glucan assays (BDG). Four study cohorts are included in this study (patients suffering from invasive aspergillosis, invasive candidiasis, PJP, and a control group at risk for fungal infections but without evidence for fungal infections). The tests were performed in parallel and revealed astonishing high sensitivities and specificities. The authors demonstrate that sensitivities of GT for separate subgroups can be significantly further increased (without a major loss of specificity) by lowering the cut off.

This study enhances our knowledge about the performance of BDG diagnostics. The scientific community and all healthcare professionals benefit from this comparison of the two assays that are currently available in Europe.

However, there are some major concerns about this manuscript.

Major comments

1. The performance of BDG testing in this study is astonishing. The FA is commonly known to have the superior sensitivity but a lower specificity in comparison with GT (“FA weakness: specificity”). Contrarily, the GT is characterized by high specificity but inferior sensitivity compared to FA (“GT weakness: sensitivity”). However, in this study both assays excel not only in their strength but also in their putative weakness:

FA: The presented sensitivities (IA: 93 %, IC: 100 %, PJP: 100 %) are astonishing and (to my knowledge) the highest sensitivities published so far. Also the specificities (the “FA weakness”) are the best I did encounter until today (IA: 98 %, IC: 97 %; PJP: 97 %).

For comparison, I would like to refer to the results of different meta-analyses over the recent years, e.g., Onishi et al., 2011 (IFI sensitivity and specificity: 80 % and 82 %), Lu et al., 2011 (IFI sensitivity and specificity: 76 % and 85 %), Karageorgopoulos et al., 2011 (IFI sensitivity and specificity: 77 % and 85 %),…

GT: High specificity is a well known feature of the GT. However, the results of this study are still excelling: 100 % for IA, 100 % for IC, 100 % for PJP. Also the sensitivity is notable (the “GT weakness”), particularly in the setting of IC / candidemia: While other recent studies (Friedrich et al., 2018, Dichtl et al., 2018) found only sensitivities of 43 – 67 %, this study peaked with a never seen sensitivity of 91 %.

This absolutely raises the questions why De Carolis and colleagues experienced so much better results than virtually all other groups. The “discrepant” results of virtually all previous studies must be addressed in the manuscript and the reasons for this difference must be discussed.

2. There are several conclusions that are not speculative or euphemistic but wrong. The authors state that “GT performed as wells as the FA” (l38). This statement is based on results like sensitivities in the setting of IA of 92.5 % (FA) and 60 % (GT). 60 % is far from 92.5 %. L39: “Further studies are expected to definitely establish the equivalence of the two BDG assays.” This prognosis lacks a scientific basis. Concerning the comparability of the assays, the authors rely on the expression “perfect concordance regarding the results (positive / negative) obtained with the FA […] and the GT […]” (ll180-181). Then they demonstrate that this “perfect concordance” is characterized by the following results in the different subgroups: “IFD (91.8 % agreement)”, “IA (91.6 % agreement)”, “IC (95 % agreement)”, and “PJP (96 % agreement)”. The agreement is high, but perfect agreement means 100 %.

3. This study tends to favor the GT. Why was a ROC curve analysis only performed for the GT? Why were the results only reevaluated with an optimized cut off of the GT? What would happen to the results of FA testing when an optimized cut off was used?

This manuscript should be carefully revised in order to provide a neutral comparison of both tests that meets all scientific requirements.

Minor comments

The title suggests that PJP is not an invasive fungal infection. Why?

L24: The assays are not restricted to serum (plasma!).

L71: In which specificity did decreasing the cut off of the GT result?

L74: not

Ll139-141: This sentence (x samples of x cases, y samples of y cases, z samples of z cases) should be reworded.

Ll141-143: There is no EORTC/MSG definition for PJP.

Ll149-150: This sentence just reflects the results of the previous sentence.

Ll150-151: The first part of the sentence just reflects the results of the two previous sentences.

Ll209-211: There is no EORTC/MSG case definition for PJP. However, there is an EORTC/MSG case definition for proven invasive Candida infections: Cultivation of Candida from blood culture (= sterile body fluid) makes this infection a proven IC according to the criteria. Hence, the study of Friedrich et al. also relied on a large cohort of proven IC.

Ll211-215: This statement should be clarified.

Ll215-217: Particularly in combination with the previous statement concerning the study of Friedrich et al., this statement is very confusing. Why do the authors draw the conclusion that positive blood cultures might be a suboptimal gold standard?

L243: about the role of serum BDG (assay)

L247: could be the use of serum instead of plasma; ?

Reviewer #4: This manuscript studied the performance of the Wako Glucan test for the detection of beta-D-glucan (BDG) in patients with invasive fungal infections, and compared it to the Fungitel assay, which is currently the most widely used assay. The study was well designed and is appropriate for this scientific question. The sample size is sufficiently large with regards to IA and IC (40 and 78 cases respectively), but rather small for the PJP subgroup (only 17 cases). Overall, the manuscript is concise, well written and well structured.

We found only some minor issues:

1. English editing for spelling and grammar is required. For example, sentences like lines 255-257 are hard to read and understand, and use non-standard idioms.

2. How were patients selected? The authors mention that they selected patients with proven or probable IFD, which suggests a pre-existing register. The selection process can be biased depending on how it is performed (eg selection based on positive cultures invariably leads to higher biomarker levels, compared to selection based on PCR or other antigen tests). Please add this information to the M&M section.

3. The authors used the 2008 EORTC/MSG definition, likely because this study was conceived and performed prior to the release of the current 2019 revisions, which is entirely understandable. However, for clarity purposes, it seems best to at least acknowledge these new definitions. Furthermore, would it be possible to use the new classification for this manuscript instead? I believe this would require only a minor effort: for IC, no reclassification should be required as the criteria have been extended on top of the 2008 definitions. The classification used for PJP in this manuscript matches with EORTC/MSG proven or probable PJP (depending on positive IF or PCR). Only for IA, reclassification could be required depending on the result of serum or BALf GM, which can likely be done easily. Radiologic features and host features were extended, and therefore do not need revision for the 2019 definitions.

4. Please provide information on serum GM values in IA as a measure of fungal burden (see also comment 6)

5. The authors use a mixed control group of patients with different underlying diseases for all analyses. However, patients at risk for IC do not have the same risk factors as those at risk for IA (eg abdominal surgery is not a major risk factor for IA or PJP), which could skew the test parameters (eg use of surgical gauze during surgery is a known source of false positive BDG, which would not be present for patients at risk for IA or PJP). Was there a large difference in specificity in these control subgroups?

6. The sensitivity for BDG in IC and IA in this study is significantly higher than reported (see the meta-analyses by Onishi et al, 2011; Lu et al, 2011). What could be the reason for this according to the authors? Were patients diagnosed only in the later stages of infection (which leads to higher BDG levels)?

7. Table 1 or in text: please provide more info on the mycological features of the cases. How many were culture positive, how many GM, how many had positive IF or PCR in PJP? How many IC cases were blood culture positive, and how many histopathology, or both? This is relevant as BDG levels are generally higher in culture positive cases.

8. Table 2 appears to be a duplicate of what is already given in text. Please remove either one.

9. Line 141: please remove “proven” PJP when using your own definitions, as this could be confused with the 2019 defined proven PJP.

10. Section on performance of FA and GT (lines 145 and following): please provide p-values for all comparisons of diagnostic parameters between these two assays, using appropriate statistical methods. The authors compare sensitivity and specificity regularly throughout the manuscript and make claims about superiority of one over the other, without evaluating if this difference is statistically significant.

11. As the levels of BDG depend on the disease (higher in IC/PJP than in IA), it seems appropriate to define a cutoff for each. The authors even mention this themselves in the discussion, yet did not perform this relatively simple data exercise. Could they therefore verify of a different cutoff would be required for IA, compared to PJP/IC?

12. Lines 177-179: Is the difference between correlation coefficient statistically significantly different between the three diseases? They appear to be very close to each other.

13. Lines 209-2011: What do the authors mean? A positive blood culture can be considered as proven IC, so there should not be any difference there. Do the authors imply that, as they also included proven cases based on positive histopathology without positive blood culture in addition to only a positive blood culture, that this could be an explanation for a higher sensitivity in their study? This seems counter-intuitive. Comment 7 would also help in clarifying this issue.

14. Could the authors add a small explanation to the discussion or intro on why the numeric values are different between the FA and GT, while still testing for the same component (BDG) due to the use of different standards? This could otherwise be confusing for non-expert readers.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

28 May 2020

PONE-D-19-33241

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please include your amended Competing Interests Statement within your cover letter.

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters).

Reviewer #1: This article addresses the diagnostic cutoffs for the available fungal diagnostic tests. It suggests new cutoffs that improve detection for those tests, which is in agreement with previous data. The change in cutoff is very small however. This article may be relevant for the field.

Answer: We thank the reviewer for his/her appreciation of our study. We agree that the change in the assay’s cutoff we proposed is small. However, we believe that this information may be relevant in the fungal diagnostics field.

Reviewer #2: Thank you for the opportunity to review this manuscript.

De Carolis and colleagues present a well-performed study demonstrating the performance of the Wako beta glucan assay in comparison to the Fungitell assay. Overall, the study design is robust and performance of the study is well explained.

English language editing can be sought to improve the readability of the manuscript.

Answer: We thank the reviewer for his/her appreciation of our study. As requested, the English language editing benefited from reviewing by an expert colleague.

Specific comments:

- Line 45: suggest reword the opening sentence to: "Invasive Aspergillosis, IC and PJP represent the most prevalent invasive fungal infections worldwide".

Answer: We reworded the opening sentence as requested.

- Line 46: reword "previously known as" to "also known as". PCP is still being used as a term for the disease.

Answer: We modified as suggested.

- Line 51: Suggest remove "e.g. respiratory"

Answer: The specification was removed.

- Line 51: non-specific, not unspecific

Answer: The term “unspecific” was changed to “nonspecific”.

- Line 54: Suggest reword- instead of unsuccessful, rather mention that culture methods have poor sensitivity or yield false negative results.

Answer: The sentence was modified as suggested.

- Line 56: reword as empiric treatment may be necessary. (the reference used here relates specifically to respiratory disease in patients with hematological disease). In many other cases of invasive fungal disease, it may not be appropriate to initiate empiric therapy.

Answer: We modified “is necessary” as “may be necessary”.

- Line 57: Reword as a "broad fungal biomarker", rather than almost universal.

Answer: We modified the term as suggested.

- Study design: additional detail about the patient population at risk of IFD at this hospital. From the data, it appears that most cases are seen in Oncology. Were these only adult patients, or were pediatric patients included as well?

Answer: Many cases were from oncology/hematology patients, and all were from adult patients. See the text and revised Table 1.

- Line 97-98: elaborate on the negative samples chosen, since the EORTC/MSG guideline does not have a specific definition for what is considered "no evidence of IFD".

Answer: We clarified that our cases did not meet the criteria for proven or probable disease and thus were considered as not having evidence of IFD.

- Methods section: Serum BDG measurement- the fact that plasma is the sample of choice for the Wako assay should be mentioned to the reader in the methods, as well as the fact that a serum had not been validated on the assay before performance of this study. The only place this is mentioned is in the limitations section of the discussion.

Answer: According to this comment, we added a sentence clarifying this issue in the M&M section.

- Line 263: hard to diagnose, not hard to diagnosis

Answer: The term “hard-to-diagnosis” was changed to “hard-to-diagnose”.

- Additional limitations that require mentioning: the very small number of PJP cases.

Answer: We added the additional limitation concerning the number of PJP cases.

- Table 1: Interquartile range should be added for patient age.

Answer: The interquartile range for patient age was added. See revised Table 1.

- Line 341- patients "in" intensive care.

Answer: “in” was corrected in the Table 1 footnote.

- Table 2: It is not clear how the Wako median and IQR BDG value can be 0 (0-0) for the group "No evidence of IFD", particularly if there were cases of false positives detecting value above 7 and 11 pg/ml. This aspect requires re-analysis.

Answer: We checked for correctness all the values previously presented in Table 2 (these values now appear in the text, because we deleted the original Table 2 as suggested by reviewer 4).

- Figure 1: requires axis titles on both x and y-axes.

Answer: Figure 1 was emended to show axis title on both x and y-axes. In addition, Figure 1 was modified, following the suggestion of reviewer 3, to show the results of the ROC analysis with an optimized cutoff also for the FA.

Reviewer #3: De Carolis and colleagues present a comprehensive comparison of the Fungitell assay (FA) and the Glucan test (GT) representing the two CE certified beta-1,3-D-glucan assays (BDG). Four study cohorts are included in this study (patients suffering from invasive aspergillosis, invasive candidiasis, PJP, and a control group at risk for fungal infections but without evidence for fungal infections). The tests were performed in parallel and revealed astonishing high sensitivities and specificities. The authors demonstrate that sensitivities of GT for separate subgroups can be significantly further increased (without a major loss of specificity) by lowering the cutoff.

This study enhances our knowledge about the performance of BDG diagnostics. The scientific community and all healthcare professionals benefit from this comparison of the two assays that are currently available in Europe.

However, there are some major concerns about this manuscript.

Major comments

1. The performance of BDG testing in this study is astonishing. The FA is commonly known to have the superior sensitivity but a lower specificity in comparison with GT (“FA weakness: specificity”). Contrarily, the GT is characterized by high specificity but inferior sensitivity compared to FA (“GT weakness: sensitivity”). However, in this study both assays excel not only in their strength but also in their putative weakness:

FA: The presented sensitivities (IA: 93 %, IC: 100 %, PJP: 100 %) are astonishing and (to my knowledge) the highest sensitivities published so far. Also the specificities (the “FA weakness”) are the best I did encounter until today (IA: 98 %, IC: 97 %; PJP: 97 %).

For comparison, I would like to refer to the results of different meta-analyses over the recent years, e.g., Onishi et al., 2011 (IFI sensitivity and specificity: 80 % and 82 %), Lu et al., 2011 (IFI sensitivity and specificity: 76 % and 85 %), Karageorgopoulos et al., 2011 (IFI sensitivity and specificity: 77 % and 85 %),..

Answer: We expanded the Discussion to contextualize our findings by referring to the results of several meta-analyses published over the recent years.

GT: High specificity is a well-known feature of the GT. However, the results of this study are still excelling 100 % for IA, 100 % for IC, 100 % for PJP. Also the sensitivity is notable (the “GT weakness”), particularly in the setting of IC / candidemia: While other recent studies (Friedrich et al., 2018, Dichtl et al., 2018) found only sensitivities of 43 – 67 %, this study peaked with a never seen sensitivity of 91 %.

This absolutely raises the questions why De Carolis and colleagues experienced so much better results than virtually all other groups. The “discrepant” results of virtually all previous studies must be addressed in the manuscript and the reasons for this difference must be discussed.

Answer: We expanded the Discussion to address the “discrepant” results of relevant previous studies in order to explain the reasons for this difference.

2. There are several conclusions that are not speculative or euphemistic but wrong. The authors state that “GT performed as well as the FA” (L38). This statement is based on results like sensitivities in the setting of IA of 92.5% (FA) and 60% (GT). 60% is far from 92.5 %. L39: “Further studies are expected to definitely establish the equivalence of the two BDG assays.” This prognosis lacks a scientific basis. Concerning the comparability of the assays, the authors rely on the expression “perfect concordance regarding the results (positive / negative) obtained with the FA […] and the GT […]” (L180-181). Then they demonstrate that this “perfect concordance” is characterized by the following results in the different subgroups: “IFD (91.8 % agreement)”, “IA (91.6 % agreement)”, “IC (95 % agreement)”, and “PJP (96 % agreement)”. The agreement is high, but perfect agreement means 100 %.

Answer: All the statements emphasizing the GT performance were mitigated as requested. Additionally, the term “perfect concordance” was changed to “excellent concordance” according to the definitions specified in M&M.

3. This study tends to favor the GT. Why was a ROC curve analysis only performed for the GT? Why were the results only reevaluated with an optimized cut off of the GT? What would happen to the results of FA testing when an optimized cut off was used?

Answer: To be absolutely impartial (see comment below), we performed a ROC curve also for the FA. See Results of the revised manuscript.

This manuscript should be carefully revised in order to provide a neutral comparison of both tests that meets all scientific requirements.

Answer: We revised the manuscript throughout to be somewhat neutral into comparing both FA and GT tests in order to satisfy all scientific requirements.

Minor comments

The title suggests that PJP is not an invasive fungal infection. Why?

Answer: The title was modified to avoid misunderstanding about PJP.

L24: The assays are not restricted to serum (plasma!).

Answer: The sentence was modified to also mention the plasma.

L71: In which specificity did decreasing the cutoff of the GT result?

Answer: This important detail was added.

L74: not

Answer: “not” was used instead of “no”.

L139-141: This sentence (x samples of x cases, y samples of y cases, z samples of z cases) should be reworded.

Answer: The sentence was amended.

L141-143: There is no EORTC/MSG definition for PJP.

Answer: According to this observation, we added the appropriate reference (Alanio et al. JAC 2016) for PJP.

L149-150: This sentence just reflects the results of the previous sentence.

Answer: The sentence was deleted.

L150-151: The first part of the sentence just reflects the results of the two previous sentences.

Answer: The first part of the sentence was deleted.

L209-211: There is no EORTC/MSG case definition for PJP. However, there is an EORTC/MSG case definition for proven invasive Candida infections: Cultivation of Candida from blood culture (= sterile body fluid) makes this infection a proven IC according to the criteria. Hence, the study of Friedrich et al. also relied on a large cohort of proven IC.

Answer: The sentence was modified to clarify that 2008 EORTC/MSG definitions do not include PJP as well as to precise that blood culture is really a proven IC criterion.

Ll211-215: This statement should be clarified.

Answer: The statement was modified for clarity.

Ll215-217: Particularly in combination with the previous statement concerning the study of Friedrich et al., this statement is very confusing. Why do the authors draw the conclusion that positive blood cultures might be a suboptimal gold standard?

Answer: The statement was modified to improve the clarity.

L243: about the role of serum BDG (assay)

Answer: The term assay was deleted.

L247: could be the use of serum instead of plasma?

Answer: The sentence was amended.

Reviewer #4: This manuscript studied the performance of the Wako Glucan test for the detection of beta-D-glucan (BDG) in patients with invasive fungal infections, and compared it to the Fungitell assay, which is currently the most widely used assay. The study was well designed and is appropriate for this scientific question. The sample size is sufficiently large with regards to IA and IC (40 and 78 cases respectively), but rather small for the PJP subgroup (only 17 cases). Overall, the manuscript is concise, well written and well structured.

We found only some minor issues:

1. English editing for spelling and grammar is required. For example, sentences like lines 255-257 are hard to read and understand, and use non-standard idioms.

Answer: As requested, the English language editing benefited from reviewing by an expert colleague.

2. How were patients selected? The authors mention that they selected patients with proven or probable IFD, which suggests a pre-existing register. The selection process can be biased depending on how it is performed (e.g. selection based on positive cultures invariably leads to higher biomarker levels, compared to selection based on PCR or other antigen tests). Please add this information to the M&M section.

Answer: Details about the selection of patients with proven or probable IFD were provided in the M&M section.

3. The authors used the 2008 EORTC/MSG definition, likely because this study was conceived and performed prior to the release of the current 2019 revisions, which is entirely understandable. However, for clarity purposes, it seems best to at least acknowledge these new definitions. Furthermore, would it be possible to use the new classification for this manuscript instead? I believe this would require only a minor effort: for IC, no reclassification should be required as the criteria have been extended on top of the 2008 definitions. The classification used for PJP in this manuscript matches with EORTC/MSG proven or probable PJP (depending on positive IF or PCR). Only for IA, reclassification could be required depending on the result of serum or BALF GM, which can likely be done easily. Radiologic features and host features were extended, and therefore do not need revision for the 2019 definitions.

Answer: We added a sentence stating the status of IFD cases according to the 2019 EORTC/MSG definition criteria. See the M&M section of the revised manuscript.

4. Please provide information on serum GM values in IA as a measure of fungal burden (see also comment 6)

Answer: Information about the serum GM values in IA was provided. See Discussion and the new Table 2.

5. The authors use a mixed control group of patients with different underlying diseases for all analyses. However, patients at risk for IC do not have the same risk factors as those at risk for IA (e.g. abdominal surgery is not a major risk factor for IA or PJP), which could skew the test parameters (e.g. use of surgical gauze during surgery is a known source of false positive BDG, which would not be present for patients at risk for IA or PJP). Was there a large difference in specificity in these control subgroups?

Answer: As suggested, we performed a rough analysis using control subgroups to assess potential differences in specificity among the patients at risk for a different IFD.

6. The sensitivity for BDG in IC and IA in this study is significantly higher than reported (see the meta-analyses by Onishi et al, 2011; Lu et al, 2011). What could be the reason for this according to the authors? Were patients diagnosed only in the later stages of infection (which leads to higher BDG levels)?

Answer: A comment about the possible reasons for the high sensitivity seen in our study was added.

7. Table 1 or in text: please provide more info on the mycological features of the cases. How many were culture positive, how many GM, how many had positive IF or PCR in PJP? How many IC cases were blood culture positive, and how many histopathology, or both? This is relevant as BDG levels are generally higher in culture positive cases.

Answer: We added a new Table 2 (in substitution of the original Table 2, which was deleted; see comment below) to include all the information about mycological features of the cases.

8. Table 2 appears to be a duplicate of what is already given in text. Please remove either one.

Answer: Table 2 was deleted.

9. Line 141: please remove “proven” PJP when using your own definitions, as this could be confused with the 2019 defined proven PJP.

Answer: “proven” was removed when referring to as PJP.

10. Section on performance of FA and GT (lines 145 and following): please provide p-values for all comparisons of diagnostic parameters between these two assays, using appropriate statistical methods. The authors compare sensitivity and specificity regularly throughout the manuscript and make claims about superiority of one over the other, without evaluating if this difference is statistically significant.

Answer: Our study aimed to evaluate two BDG assays in parallel. According to the main comments of reviewer 3, we omitted claims about the superiority of one over the other. Therefore, a statistical comparison of these differences was waived.

11. As the levels of BDG depend on the disease (higher in IC/PJP than in IA), it seems appropriate to define a cutoff for each. The authors even mention this themselves in the discussion, yet did not perform this relatively simple data exercise. Could they therefore verify of a different cutoff would be required for IA, compared to PJP/IC?

Answer: As suggested, we performed additional analyses to verify the potentiality of a different cutoff for IA compared to PJP/IC. See Results of the revised manuscript.

12. Lines 177-179: Is the difference between correlation coefficient statistically significantly different between the three diseases? They appear to be very close to each other.

Answer: We checked that the differences were statistically significant.

13. Lines 209-211: What do the authors mean? A positive blood culture can be considered as proven IC, so there should not be any difference there. Do the authors imply that, as they also included proven cases based on positive histopathology without positive blood culture in addition to only a positive blood culture, that this could be an explanation for a higher sensitivity in their study? This seems counter-intuitive. Comment 7 would also help in clarifying this issue.

Answer: The sentence was modified to improve clarity and to add relevant information, also in accordance with what suggested in comment 7.

14. Could the authors add a small explanation to the discussion or intro on why the numeric values are different between the FA and GT, while still testing for the same component (BDG) due to the use of different standards? This could otherwise be confusing for non-expert readers.

Answer: A possible explanation in the Introduction section was added.

Submitted filename: PONE-D-19-33241 Responses to Reviewers Comments.doc

Click here for additional data file.

15 Jun 2020

PONE-D-19-33241R1

Comparative performance evaluation of Wako β-glucan test and Fungitell assay for the diagnosis of invasive fungal diseases

PLOS ONE

Dear Dr. Sanguinetti,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please address the final reviewer concerns.

Please submit your revised manuscript by Jul 30 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Jeffrey Chalmers, Ph.D.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

Reviewer #4: No

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: (No Response)

Reviewer #4: Most of my comments have been addressed. However, I disagree with the reply to comment 10 regarding statistical analysis. Overall, Reviewer #3's comments are largely the same as mine. We both state that you cannot make any proclamation about superiority without doing due analysis. However, this does not mean this is not possible, or indeed, that this can be waived!

The fact that this is a head-to-head analysis means a pairwise comparison is indicated, such as McNemar's test. Using this test, you can confidently state for example, that the sensitivity of FA > 80 is significantly greater than the sensitivity of GT > 7 (92.5% vs 80%, p=0.025). Most statistical packages have tools specifically for comparing 2 diagnostic tests.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 Jun 2020

PONE-D-19-33241R1

Reviewer #4: Most of my comments have been addressed. However, I disagree with the reply to comment 10 regarding statistical analysis. Overall, Reviewer #3's comments are largely the same as mine. We both state that you cannot make any proclamation about superiority without doing due analysis. However, this does not mean this is not possible, or indeed, that this can be waived!

The fact that this is a head-to-head analysis means a pairwise comparison is indicated, such as McNemar's test. Using this test, you can confidently state for example, that the sensitivity of FA > 80 is significantly greater than the sensitivity of GT > 7 (92.5% vs 80%, p=0.025). Most statistical packages have tools specifically for comparing two diagnostic tests.

Answer: We are delighted in noticing that we were able to satisfy most of the reviewer’s comments. Regarding the unsatisfied remaining comment raised by the reviewer (also in line with the reviewer #3), we treasured the suggestion of the reviewer. Thus, we used the McNemar's test to assess statistically the differences in performance parameters between FA and GT. Accordingly, we added five sentences, one of which in the Materials & Methods section (Statistical Analysis paragraph) and other four in the Results section (Performances of the FA and the GT). See the revised version of the manuscript (PONE-D-19-33241R1 with Track Changes).

Submitted filename: PONE-D-19-33241R1 Response to Reviewers.doc

Click here for additional data file.

30 Jun 2020

Comparative performance evaluation of Wako β-glucan test and Fungitell assay for the diagnosis of invasive fungal diseases

PONE-D-19-33241R2

Dear Dr. Sanguinetti,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jeffrey Chalmers, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #4: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #4: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #4: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #4: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #4: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #4: No

17 Jul 2020

PONE-D-19-33241R2

Comparative performance evaluation of Wako β-glucan test and Fungitell assay for the diagnosis of invasive fungal diseases

Dear Dr. Sanguinetti:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jeffrey Chalmers

Academic Editor

PLOS ONE