Repeatability and reproducibility of a handheld quantitative G6PD diagnostic.

BACKGROUND: The introduction of novel short course treatment regimens for the radical cure of Plasmodium vivax requires reliable point-of-care diagnosis that can identify glucose-6-phosphate dehydrogenase (G6PD) deficient individuals. While deficient males can be identified using a qualitative diagnostic test, the genetic make-up of females requires a quantitative measurement. SD Biosensor (Republic of Korea) has developed a handheld quantitative G6PD diagnostic (STANDARD G6PD test), that has approximately 90% accuracy in field studies for identifying individuals with intermediate or severe deficiency. The device can only be considered for routine care if precision of the assay is high. METHODS AND
FINDINGS: Commercial lyophilised controls (ACS Analytics, USA) with high, intermediate, and low G6PD activities were assessed 20 times on 10 Biosensor devices and compared to spectrophotometry (Pointe Scientific, USA). Each device was then dispatched to one of 10 different laboratories with a standard set of the controls. Each control was tested 40 times at each laboratory by a single user and compared to spectrophotometry results. When tested at one site, the mean coefficient of variation (CV) was 0.111, 0.172 and 0.260 for high, intermediate, and low controls across all devices respectively; combined G6PD Biosensor readings correlated well with spectrophotometry (rs = 0.859, p<0.001). When tested in different laboratories, correlation was lower (rs = 0.604, p<0.001) and G6PD activity determined by Biosensor for the low and intermediate controls overlapped. The use of lyophilised human blood samples rather than fresh blood may have affected these findings. Biosensor G6PD readings between sites did not differ significantly (p = 0.436), whereas spectrophotometry readings differed markedly between sites (p<0.001).
CONCLUSIONS: Repeatability and inter-laboratory reproducibility of the Biosensor were good; though the device did not reliably discriminate between intermediate and low G6PD activities of the lyophilized specimens. Clinical studies are now required to assess the devices performance in practice.

Introduction

The 8-aminoquinolines primaquine and tafenoquine are the only drugs currently on the market with hypnozoitocidal properties, important for the clearance of Plasmodium vivax and P. ovale from the human host [1-3]. Well tolerated in the majority of recipients, 8-aminoquinolines are strong oxidants that can cause hemolysis in individuals with low activity levels of the glucose-6-phosphate dehydrogenase enzyme (G6PD), known as G6PD deficiency (G6PDd) [4, 5]. The G6PD gene is located on the X-chromosome, males are either hemizygous deficient or normal, whereas females are homozygous deficient, normal, or heterozygous for the gene. Heterozygous females have two distinct red blood cell (RBC) populations, G6PD normal and G6PD deficient, that circulate in a ratio determined through the random process of lyonization [6]. Therefore, the G6PD activity levels of heterozygous females–and their associated hemolytic risk—is dependent on the proportion of deficient cells, those cells at greatest risk of drug induced hemolysis. Approximately 400 million people worldwide are affected by G6PDd, with allele frequencies reaching up to 35% in malaria endemic areas [7, 8]. Accordingly the WHO recommends routine G6PD testing prior to radical cure (schizontocidal and hypnozoitocidal treatment) with primaquine, whenever possible [9]. A 14-day course of primaquine is prescribed to patients with more than 30% G6PD enzyme activity, while eight weekly doses are recommended in patients with less than 30% activity [9]. These long treatment courses affect treatment adherence, and lead to lower effectiveness [10, 11]. Short course, high dose primaquine treatment regimens, as well as a single dose tafenoquine treatment regimen, are likely to improve effectiveness, however these will require more stringent criteria to protect those at risk of hemolysis [12, 13]. While qualitative G6PD diagnostics have good discriminatory power at the 30% activity threshold, they cannot discriminate patients at higher G6PD activity levels [14]; to date this can only be done by quantitative spectrophotometry [15, 16]. Not only is spectrophotometry logistically unsuitable for supporting case management of P. vivax patients in remote areas where most patients live, spectrophotometry has also been shown to exhibit significant variability in its measurements [17]. For example, definitions of 100% G6PD enzyme activity in U/gHb differ significantly between studies [15]. Given the current definitions of “G6PD deficient” (<30% of normal G6PD levels) and “intermediate” (30–70% or 30%-80% of normal levels), this leads to different diagnostic cut-offs between areas [18, 19]. Comparisons of G6PD activity of standardized quality control samples show significant variation between laboratories, suggesting that at least some of the variability observed in population-level G6PD readings may be due to the spectrophotometric assay itself [15]. This diagnostic variability confounds the definition of global absolute cut-offs for case management. The consequence of this assay-derived variability is that site and assay specific G6PD baseline (100% activity) levels need to be established before local deficient and intermediate thresholds can be set, adding significant complexity to the roll out of G6PD testing in P. vivax endemic settings [18]. G6PD levels are affected by RBC density [20], any G6PD measurement therefore needs to be normalized by an Hb reading and this may also contribute to the observed variability in spectrophotometry reading.

A hand-held quantitative G6PD diagnostic has been developed by SD Biosensor (STANDARD G6PD test, Suwon-si, ROK), hereafter referred to as the “Biosensor”. The device consists of the Biosensor and a single use test strip that is inserted into the Biosensor. To generate a reading, 10μl of blood are added to a lysis buffer, 10μl of the blood buffer solution are then added to the single use test strip inserted into Biosensor. The test strip contains 5-bromo-4-chloro-3-indolyl-phosphate (BCIP) that is reduced to violet nitro blue tetrazolium (NBT) in the presence of the G6PD enzyme, the color intensity is directly proportional to G6PD activity and is measured through reflectance photometry. The Biosensor device quantifies Hb concentration using a photo-reflectance based algorithm informed by the sample’s color intensity. This is measured on a separate spot to that for the G6PD activity. The handheld device displays G6PD activity (in U/gHb) and hemoglobin (Hb) levels (in g/dL) two minutes after applying the blood buffer solution, however the manufacturer indicates that results cannot be considered if Hb readings are equal to or below 7g/dL. Field evaluation studies showed the Biosensor to have an accuracy of approximately 90% in identifying intermediate and deficient individuals when compared to spectrophotometry [21-23]. Since ease of use, time to diagnosis, and logistics and operational feasibility are preferable to spectrophotometry, the Biosensor has the potential to provide a quantitative G6PD measurement at the bed side and support point-of-care diagnosis and treatment decisions [17]. The aim of this study was not to assess accuracy, but to determine the Biosensor’s repeatability (assay precision when repeated under constant conditions) and reproducibility (assay precision under different conditions, such as across devices, operators and sites), since robust performance of these characteristics is necessary for rolling-out universal Biosensor thresholds for clinical decisions [24].

Methods

Ethics statement

IRB approvals or waivers were obtained from each participating institution prior to conducting the laboratory study. Risks to the technicians of using reconstituted human blood controls were discussed during the training and minimized by involving only technicians experienced in Good Laboratory Practice. Since no participants were enrolled, no informed consent was collected (S1 Table).

Overview

Despite the Biosensor and reference method spectrophotometry being developed for fresh blood we had to use lyophilized and reconstituted standardized controls instead in order to ensure identical samples were used throughout the study period [21]. The study comprised of two phases (Fig 1). In the first phase (Phase A) baseline repeatability and inter-device reproducibility were defined under identical conditions. Ten Biosensors were tested repeatedly in parallel in a single laboratory by a single technician using commercial controls with a range of G6PD enzyme activity levels classified by the manufacturer as “High”, “Intermediate”, and “Low”. Each control was tested 20 times over the course of five days with each Biosensor device and was tested in parallel by spectrophotometry and Hemocue. The study would only proceed to the second phase if mean repeatability of all Biosensors met or exceeded minimal requirements (see “statistical analysis” below). In the second phase (Phase B) reproducibility was assessed by shipping each device to a different, well-established laboratory. At each site, an identical set of controls was tested 40 times over the course of 10 days (120 measurements in total / site) by Biosensor and each reference assay, spectrophotometry and Hemocue. Reference methods were standardized as detailed below, and standard operating procedures were followed across all sites.

Fig 1

Schematic overview of the study design.

CV: coefficient of variation. (Images reproduced with permission from SD Biosensor and Hemocue).

Control samples

To ensure that identical samples were tested across all sites, commercial controls were used, with all controls within one phase being from the same lot (Analytical Control Systems, Inc., Indiana, USA; S2 Table). ACS controls are routinely used to monitor quality of reference G6PD testing by spectrophotometry. They are derived from whole blood obtained from human donors in FDA licensed centers and pooled to represent high, intermediate, or low G6PD activity (Cat. Nos.: HC-108, HC-108IN, and HC-108DE respectively). ACS provide lot specific G6PD activity range guides and Hb estimates, these are based on automated spectrophotometry estimates conducted by Pointe-Scientific on the reconstituted controls (32 vials in total). ACS recommend that laboratories develop their own in-house ranges but provides guideline ranges per control category as well. We were unable to establish spectrophotometry-based ranges applicable to all sites due to the inherent site-specific variability of spectrophotometry (15), we therefore considered the manufacturer recommended ranges. ACS controls were provided in lyophilized form and reconstituted in each laboratory at standardized intervals. All reconstituted controls were stored at 4°C and used within two days.

Biosensor

The SD Biosensor STANDARD G6PD test was performed following the manufacturer instructions. The assay uses single-use test strips (Cat. No. 02G6S10) that are inserted into the Biosensor (Cat. No. 02GA10). All test strips used in this study were from the same manufacturing lot (S2 Table). In brief, 10μl of the reconstituted control sample were mixed with the assay extraction buffer, then 10μl of the control-buffer solution were added to a single use test device that had already been inserted into the Biosensor. Sample transfer devices supplied with the Biosensor test, known as “Ezi tubes”, were used at each step. The displayed normalized G6PD activity and Hb readings were recorded once the measurement was completed after the 2 mins running period.

Each Biosensor’s functionality was checked daily with a multi-use STANDARD G6PD check strip, and sample testing only commenced if the quality control check was completed successfully. Every five days each Biosensor was also quality controlled with reconstituted control samples provided by SD Biosensor (“level 1” and “level 2”; Cat. No. 02G6C10). If results were outside of the recommended ranges for G6PD activity or Hb reading, the quality control testing was repeated. If three consecutive quality control readings were outside the recommended ranges, testing with the respective Biosensor device was aborted.

Spectrophotometry and Hemocue

Spectrophotometry was performed using kits from Pointe Scientific (Michigan, USA; Cat. No. G7583) according to the manufacturer’s recommendations. The brand of spectrophotometer varied between laboratories, but all instruments were temperature-controlled, cuvette based, and measured absorption at 340 nm. Sample absorbance was measured at 0 and 5 minutes at 37°C and the difference in absorbance was used to calculate G6PD activity in U/dL following a standard formula provided by the Pointe-Scientific assay manufacturer. Measurements were run in duplicate (i.e. the sample reaction was divided into 2 separate cuvettes and measured independently) and the mean of the two G6PD results was recorded. If the coefficient of variation of the two measures exceeded 15% (CV>0.15), a third measurement was required. G6PD activity was then normalized by a Hb reading (Hemocue 201, 301 or 801, Angelholm, Sweden) to generate G6PD activity in U/gHb. Hemocue devices were used following manufacturer instructions, and the Hb reading was recorded separately.

Training

All technicians conducting the experiments had at least a bachelor’s degree or higher, several years’ experience of working in a laboratory and familiarity with spectrophotometry, however not necessarily with the G6PD assay used in this study. Each technician received standardized training in an online session and had to pass a Biosensor proficiency test prior to conducting the experiments. In each laboratory, a single technician performed the analyses with each diagnostic across all study testing days.

Statistical analysis

Data were recorded on standardized forms and then transferred to an Excel database (Microsoft Corp, Washington, USA), with standard data entry cross-checks. Analysis was undertaken using Stata version 15 (Stata Corp, College Station, Texas, USA).

Depending on data distribution, summary findings were displayed as mean or median with 95% confidence intervals or interquartile range (IQR) respectively. Repeatability and reproducibility were assessed by linear, random effects, and mixed effects regression models as appropriate, and by calculating coefficients of variation (CV). Spearman’s Rank coefficient (rs) was calculated to determine the correlation between Biosensor and reference method. Absolute differences between experimental (Biosensor) and reference assays (spectrophotometer and Hemocue) were assessed by Bland Altman plots and the Wilcoxon matched pairs signed rank test. The study only progressed to Phase B if the mean CV for the High control sample across all devices was less than 0.150 [25]. Combined mean difference and correlation coefficients were calculated for Phase A where one spectrophotometry reading served as reference for all ten devices. In Phase B findings were not combined since each site performed their own reference measurement and the reference method for G6PD activity is known to show significant variation, not allowing for a direct comparison [15].

Following Bonferroni correction, the level of significance was set at p<0.005 whenever multiple comparisons were done.

Results

Phase A

Results from the ten Biosensor devices tested with High and Intermediate controls were available for five consecutive days, and Low controls for four days due to limited stocks of same-lot Low controls.

Hb-normalized G6PD activities did not differ significantly between Biosensor devices (Fig 2 and S3 Table, p = 1.000). However, Hb readings differed significantly (p<0.001, adjusted R2 = 0.121). Compared to device 1 (baseline), devices 3 and 9 had significantly lower Hb readings with a difference of 0.477 g/dL (p = 0.007) and 0.623 g/dL (p<0.001) respectively, while device 5 had a significantly higher Hb result with a difference of 0.665 g/dL (p<0.001) when comparing readings from all three controls (Fig 3 and S3 Table).

Fig 2

G6PD activity measured by each Biosensor device and spectrophotometry.

Red circled bars = reference method (Hb normalized spectrophotometry result), Blue shade = recommended range for ACS Low controls, Red shade = recommended range for ACS Intermediate controls, Green shade = recommended range for ACS High controls, Blue dotted line = median activity of ACS Low controls across all devices excluding spectrophotometry, Red dotted line = median activity of ACS Intermediate controls across all devices excluding spectrophotometry, Green dotted line = median activity of ACS High controls across all devices excluding spectrophotometry, dots represent outliers.

Fig 3

Hb levels measured by each Biosensor device and Hemocue.

Red circled bars = reference method (Hemocue), Blue line = recommended point estimate for ACS Low controls, Red line = recommended point estimate for ACS Intermediate controls, Green line = recommended point estimate for ACS High controls, Blue dotted line = median Hb reading for Low controls, Red dotted line = median Hb reading for Intermediate controls, Green dotted line = median Hb reading for High controls, dots represent outliers.

Each run included a spectrophotometer measurement which was matched with a result from each of the 10 Biosensor devices (Fig 1). The Hb-normalized G6PD activity readings of the Biosensor and spectrophotometry were positively correlated across all three control categories (rs = 0.859, p<0.001), however median readings differed significantly for Low (mean difference: -0.1 U/gHb, 95% limit of agreement [95%LoA]: -0.8 to 0.5, p<0.001) and Intermediate controls (mean difference: -1.2 U/gHb, 95%LoA: -2.3 to -0.1, p<0.001) while median activities did not differ significantly for High controls (mean difference: -0.1 U/gHb, 95%LoA: -2.3 to 2.1, p = 0.554). Hb readings from the Biosensor and Hemocue showed a significant correlation in five out of 10 devices (p<0.005; Table 1).

Table 1

Correlation of Biosensor and reference method, Phase A.

Device	G6PD (rs, p*)	Hb (rs, p*)
1	0.871, <0.001	0.427, <0.001
2	0.884, <0.001	0.278, 0.031
3	0.861, <0.001	0.424, <0.001
4	0.839, <0.001	0.288, 0.025
5	0.874, <0.001	0.416, 0.001
6	0.868, <0.001	0.314, 0.015
7	0.874, <0.001	0.183, 0.163
8	0.856, <0.001	0.394, 0.002
9	0.885, <0.001	0.384, 0.003
10	0.871, <0.001	0.322, 0.012
Pooled	0.868, <0.001	0.318, <0.001

Highlighted correlations are not significant

*level of significance set at p<0.005 following Bonferroni correction

Since normalized G6PD activity did not differ significantly between Biosensor devices, results were pooled and compared to spectrophotometry by Bland-Altman plot. The G6PD readings of the Biosensor were significantly lower (mean difference: 0.5U/gHb, 95%LoA: -2.2 to 1.3; p<0.001) compared to spectrophotometry (Fig 4). Mean Hb readings using the Biosensor were 1.8g/dL (95%LoA: -4.7 to 1.1) lower than those of the Hemocue (p<0.001) (S1 Fig).

Fig 4

Phase A: G6PD activity measured by Biosensor and spectrophotometry. Red circled results are influential points generated by spectrophotometry; Left: Red horizontal lines indicate upper and lower end of 95% limit of agreement, black dotted line indicates mean difference. Right: horizontal dotted lines indicate the overlap between Low and Intermediate readings by Biosensor.

Median G6PD readings from the Low control were below the manufacturer recommended range (0.8U/gHb– 3.8U/gHb) when measured by Biosensor (0.7U/gHb, interquartile range (IQR): 0.6–0.8) but not by spectrophotometry (0.9U/gHb, IQR: 0.8–1.0), while median G6PD readings for Intermediate (Biosensor: 1.7U/gHb, IQR: 1.5–1.8; spectrophotometry: 2.7, IQR: 2.5–2.9) and High controls (Biosensor: 7.6U/gHb, IQR: 7.5–8.2; spectrophotometry: 7.6U/gHb, IQR: 7.0–8.1) were below the recommended ranges (Intermediate: 3.7U/gHb– 6.8U/gHb; High 10.3U/gHb– 19.1U/gHb respectively) for both assays. Neither the median Hb readings of the Biosensor nor Hemocue were equal to the point estimate provided by the control sample manufacturer (Tables 2 and S3 and Figs 2 and 3).

Table 2

Phase A results: coefficient of variation and median value measured for all Biosensors combined, spectrophotometry and Hemocue.

Summaries are for all Phase A results combined.

	G6PD activity (in U/gHb)			Hb (in g/dL)
Assay	Low	Intermediate	High	Low	Intermediate	High
Biosensors (n = 600)
CV	0.260	0.172	0.111	0.061	0.068	0.069
Median	0.7	1.7	7.6	14.8	14.9	14.1
IQR	0.6–0.8	1.5–1.8	7.2–8.2	14.3–15.6	14.2–15.5	13.5–15.0
SD	0.17	0.28	0.85	0.91	1.01	0.98
Range	0.1–1.0	0.6–2. 4	5.7–10.5	12.9–17.5	12.3–17.5	12.0–17.7
Reference (n = 60)
CV	0.141	0.165	0.124	0.009	0.007	0.022
Median	0.9	2.7	7.6	18.4	16.5	14.9
IQR	0.8–1.0	2.5–2.9	7.0–8.1	18.2–18.5	16.4–16.6	14.7–15.0
SD	0.12	0.47	0.96	0.01	0.01	0.02
Range	0.7–1.0	2.4–4.6	6.4–11.0	18.0–18.6	16.3–16.7	13.9–15.1
	Recommended range G6PD (U/gHb)			Recommended value Hb (g/dL)
Manufacturer recommendation	0.8–3.8	3.7–6.8	10.3–19.1	13.0	13.2	13.1

CV = coefficient of variation, IQR = interquartile range, SD = standard deviation

Median readings of High, Intermediate, and Low controls were distinct by spectrophotometry (High vs. Intermediate: p<0.001 and Intermediate vs. Low: p<0.001), but while median readings by Biosensor also differed significantly between all three control categories (all p<0.001) six Intermediate readings overlapped with 122 Low results. All six Intermediate readings were generated by different devices and during different testing runs. Influential outliers generated by two spectrophotometry readings were identified visually (Fig 4).

The median CV across all Biosensor G6PD measurements for High controls was 0.111, below the pre-defined acceptability threshold of 0.150, while the CV for Hb measurement was below 0.070 for all controls (Table 2). The study therefore proceeded to Phase B.

Phase B

Biosensors and controls were shipped to one laboratory in Bangladesh, Brazil, France and Indonesia respectively, two laboratories in Thailand, and four in the USA for further testing. Staff at six laboratories had previous experience with the Biosensor. Pooled readings from Low controls were within the recommended range (0.8U/gHb– 3.8U/gHb) by Biosensor (median: 2.2U/gHb, IQR: 2.0–2.4) and spectrophotometry (1.9U/gHb, IQR: 1.6–2.2). However Intermediate pooled readings by Biosensor (1.9U/gHb, IQR: 1.6–2.1) and spectrophotometry (3.1U/gHb, IQR: 2.7–3.5) were below the manufacturer recommended range (3.7U/gHb– 6.8U/gHB), as were the readings of High controls (Biosensor: 8.0U/gHb, IQR: 7.4–8.7; spectrophotometry: 8.2U/gHb, IQR: 7.4–9.2; recommended range 10.3U/gHb– 19.1 U/gHb). Observed ranges across all categories were greater for spectrophotometry than for the Biosensor (S4 Table).

The correlation between Biosensor and reference method was significant and positive for G6PD readings (in U/gHb), while Hb readings of five Biosensor devices did not correlate significantly with the Hemocue at the 0.5% (p<0.005) significance level (Table 3).

Table 3

Mean difference and correlation of Biosensor and reference method by site in phase B.

Site	G6PD (U/gHb): Mean Difference (95% LoA, p)	G6PD (rs, p)	Hb (g/dL): Mean Difference (95% LoA, p)	Hb (rs, p)
1	-0.1 (-1.8 to 1.7, 0.245)	0.548, <0.001	-1.9 (-4.0 to 0.1, <0.001)	0.254, 0.005 *
2	-0.2 (-1.8 to 1.4, 0.004)	0.564, <0.001	-1.9 (-4.1 to 0.2, <0.001)	-0.023, 0.806 *
3	0.2 (-1.7 to 2.0, 0.088)	0.659, <0.001	-3.0 (-4.9 to -1.2, <0.001)	0.350, <0.001
4	0.1 (-2.0 to 2.2, 0.504)	0.536, <0.001	-2.5 (-4.7 to -0.3, <0.001)	0.312, 0.001
5	1.1 (-1.9 to 4.2, <0.001)	0.575, <0.001	0.6 (-0.8 to 2.0, <0.001)	0.416, <0.001
6	-0.3 (-3.2 to 2.7, 0.055)	0.509, <0.001	-2.6 (-5.9 to 0.6, <0.001)	0.095, 0.303 *
7	-1.1 (-3.1 to 0.8, <0.001)	0.514, <0.001	-0.1 (-1.7 to 1.5, 0.133)	0.239, 0.009 *
8	-0.9 (-2.8 to 1.0, <0.001)	0.609, <0.001	-1.5 (-3.4 to 0.4, <0.001)	0.500, <0.001
9	-2.6 (-7.6 to 2.3, <0.001)	0.569, <0.001	-0.6 (-3.0 to 4.3, 0.009)	0.391, <0.001
10	-1.0 (-3.1 to 1.1, <0.001)	0.563, <0.001	-0.9 (-3.0 to 1.1, <0.001)	0.211, 0.021 **

95% LoA = 95% limit of agreement

* Correlation was non-significant across all control categories (all p>0.005)

**Only correlation for Intermediate controls was significant: rs = 0.580, p = 0.001

Mean G6PD readings by Biosensor and spectrophotometry differed significantly in five of 10 sites, while Hb readings showed a significant difference between Biosensor and Hemocue in eight of 10 sites. Observed mean differences between Biosensor and spectrophotometry ranged from -2.6U/gHb to +1.1U/gHb across the ten devices and from -3.0 g/dL to 0.6g/dL between Biosensor and Hemocue (Table 3).

Low and Intermediate controls could not be differentiated by Biosensor, in fact median readings of Intermediate controls (1.9U/gHb, IQR: 1.6–2.1) were significantly lower than median Low readings (2.2U/gHb, IQR: 2.0 to 2.4, p<0.001), a trend that was seen across all sites. In contrast High controls (8.0U/gHb, IQR 7.4–8.7) were clearly distinct from the other controls across all sites (p<0.001) (Figs 5 and S2–S5, and S4 Table).

Fig 5

Box and whisker plot of G6PD activity / control by Biosensor (A) and spectrophotometry (B) across Phase B sites. Blue shade = recommended range for ACS Low controls, Red shade = recommended range for ACS Intermediate controls, Green shade = recommended range for ACS High controls, Blue dotted line = median activity of ACS Low controls across all devices, Red dotted line = median activity of ACS Intermediate controls across all devices, Green dotted line = median activity of ACS High controls across all devices, dots represent outliers.

Median spectrophotometry measures for Low (1.9U/gHb, IQR: 1.6–2.2) and Intermediate (3.1U/gHb, IQR: 2.7–3.5) readings differed significantly (p<0.001), as did Intermediate and High (8.3U/gHb, IQR: 7.4–9.3) controls (p<0.001). This trend and level of significance was consistent within all sites but not between sites (Figs 5 and S3–S6, and S4 Table).

Repeatability (within-site assay precision, as measured by CV) varied more between sites for spectrophotometry than the Biosensor (Fig 5). Site-level CVs of the Biosensor for the High control ranged from 0.103 to 0.125 (SD: 0.009), while spectrophotometry results ranged from 0.050 to 0.137 (SD: 0.043) (S5 Table).

Normalized G6PD activities did not differ significantly between Biosensors (p = 0.436), however spectrophotometry readings showed significant variation by site (p<0.001) (Figs 5, S2, S3, S4, S5 and S6). Hb readings differed significantly by Biosensor across all sites (p<0.001) and variation was even greater for the Hemocue (p<0.001) (S5 Table).

Comparing Phases A and B

The CVs between Phase A and Phase B did not differ significantly (p = 0.201). The Intermediate and High controls in Phase A and B were from the same manufacturing lots so Biosensor readings could be compared directly (S2 Table). G6PD activities differed significantly between Phase A and B (p<0.001) and while Intermediate control readings in Phase A were significantly higher for eight of the 10 devices, the difference did not exceed 0.4U/gHb. For High controls, a significant difference was observed in three devices with a maximum difference of 1.5U/gHb (Table 4).

Table 4

Mean difference in G6PD activity for Intermediate and High controls by each Biosensor device between Phases A and B.

Low controls were not directly comparable between Phases as these were from different lots.

Device	Intermediate: mean difference in U/gHb*, (95% CI, p)	High: mean difference in U/gHb*, (95% CI, p)
1	-0.1 (-0.1 to 0.2, 0.481)	0.0 (-0.5 to 0.5, 0.993)
2	-0.3 (-0.4 to -0.1, <0.001)	0.1 (-0.3 to 0.5, 0.657)
3	-0.3 (-0.5 to -0.1, 0.016)	-0.5 (-1.1 to 0.0, 0.050)
4	-0.3 (-0.5 to -0.2, <0.001)	-1.5 (-2.1 to -1.0, <0.001)
5	-0.1 (-0.6 to 0.3, 0.559)	-0.2 (-0.6 to 0.2, 0.399)
6	-0.3 (-0.4 to -0.2, <0.001)	-0.2 (-0.7 to 0.2, 0.326)
7	0.1 (-0.1 to 0.3, 0.184)	0.0 (-0.4 to 0.4, 0.906)
8	-0.4 (-0.6 to -0.2, <0.001)	-0.8 (-1.3 to -0.3, 0.003)
9	-0.2 (-0.3 to 0.0, 0.032)	-0.1 (-0.7 to 0.5, 0.765)
10	-0.2 (-0.4 to -0.04, 0.018)	-0.3 (-0.8 to 0.2, 0.224)
Pooled	-0.2 (-0.3 to -0.1, <0.001)	-0.4 (-0.5 to -0.2, <0.001)

*Phase A–Phase B

Discussion

The reproducibility (inter-device precision) of the Biosensor did not differ significantly between devices, either when handled by the same technician (Phase A), when operated in different settings by different end users (Phase B), or when the same device was handled by different operators (Phase A vs B). In contrast there was significant variation when G6PD activity was measured by spectrophotometry between sites despite standardized controls and procedures, a phenomenon that has been reported previously [15].

Four out of the ten participating sites had not used the Biosensor previously. Following standardized online training, all sites were able to generate G6PD measurements with good precision that did not differ significantly between sites. Precision of the spectrophotometry results was more variable between sites, with some sites exceeding Biosensor repeatability while others had lower precision. There was a good correlation between the Biosensor and spectrophotometry results when these were assessed in a single lab in Phase A. However, while spectrophotometry could discriminate reliably between the three control types, the Biosensor results less clearly distinguished Low from Intermediate controls, with six of the 200 repeat measurements overlapping. Correlation between Biosensor and spectrophotometry was lower in Phase B when devices were assessed in different laboratories due to the variability of the spectrophotometry. In Phase B the Biosensor did not distinguish between Low and Intermediate controls. In fact, the results were significantly lower for Intermediate compared to Low controls and this was consistent across all sites; in contrast spectrophotometry in Phase B was able to distinguish between all three control categories.

Besides G6PD activity, the Biosensor also measures and displays Hb concentration. The repeatability of Hb measurements by Biosensor was better than by Hemocue, with best inter-device repeatability observed when either device was operated by a single user. Hb readings of both devices correlated poorly, not least since the recommended Hb point estimates for all three controls were very similar. Absolute pooled readings for the Biosensor were 1.8g/dl lower in Phase A compared to paired readings of the Hemocue, however readings from the Biosensor were closer to the recommended point estimate suggested by the ACS manufacturer. Determining accuracy of Biosensor Hb readings against the Hemocue reference assay with reconstituted lyophilised controls is of limited clinical relevance. A study from the US compared paired Hb measurements from fresh, venous, samples by Biosensor and Hemocue (model 201+) and found the mean difference to be 1.0g/dL [22]; a study comparing Biosensor Hb readings from venous blood samples to the results of a complete blood count (CBC), found readings to differ by 0.4g/dL [21], and a recent study from Brazil found the mean difference again to be less than 1 g/dl [23].

Our findings have several limitations. G6PD activity and Hb levels were measured in commercial lyophilised controls to ensure cross-laboratory standardisation, but the Biosensor and reference assays are developed for testing fresh venous or capillary blood. Stabilizing agents contained in the controls may have affected the Biosensor and/or reference method and this effect may differ by assay. Repeatability and reproducibility of each assay should not have been impacted by this difference, but it may have affected accuracy of either device which is best assessed using fresh blood samples [21, 22]. While providing reference ranges, the manufacturer ACS suggests developing in house reference ranges for all controls. We were unable to establish spectrophotometry-based ranges applicable to all sites due to the inherent site-specific variability of spectrophotometry [15] and instead considered the ranges provided. This approach likely explains why G6PD readings generated by Biosensor and spectrophotometry were below the ACS manufacturer’s recommended range and Hb readings by either assay did not match the guideline point estimates. This was consistent across different devices when assessed by a single user and by different laboratories. Unfortunately, the supplier was unable to provide additional controls from the same lots for further testing to clarify this issue. The observed narrow activity ranges meant that there was little difference in activity levels between the Intermediate and Low controls, limiting the activity range that was assessed. Two spectrophotometry readings for Intermediate and High controls appeared to be outliers in Phase A, both of which were included since readings were within the recommended range and this may also have reduced the correlation between assays and the derived absolute difference. Although sites used different Hemocue models (Hemocue 201, 301 and 801), results from all devices were pooled which may have resulted in an increase in variability for the Hb reference. Finally, all measurements were done by highly qualified technicians in a research setting, not reflective of a real-world scenario, accordingly reproducibility of the Biosensor may be lower when used in a clinical setting.

The precision of the Biosensor demonstrated in this study, and the good accuracy reported from field and other evaluation studies [21–23, 26], indicate that the Biosensor could be a valuable quantitative point-of-care diagnostic; however, we found that spectrophotometry, when performed well, remains the gold standard with precision superior to the Biosensor. The reproducibility observed in this study indicates that the technology is likely to permit direct comparison of results generated by different Biosensor devices and trained users [15]. If confirmed in clinical settings, the Biosensor has the potential to be an important tool to facilitate the broader roll out of 8-aminoquinoline radical cure. Clinical data will be important to further investigate the poor discriminatory power of the Biosensor at low and intermediate G6PD activities observed with the lyophilised samples, however given that the Biosensors’ most probable designation will be to distinguish G6PD normal individuals from those with less than normal activity (at a cut-off of 70% activity for Tafenoquine) the observed poor discriminatory power at lower activities is unlikely to be of significant practical relevance. Finally, it will be important to verify whether the observed precision demonstrated here is maintained when the device is operated under routine conditions and in anaemic patients, as well as to define training requirements for intended users at the point-of-care. In conclusion, our findings suggest that the Biosensor offers reproducible quantitative diagnosis of G6PD status at the point-of-care in the hands of well-trained technicians. If repeatability and reproducibility as well as the previously reported accuracy are confirmed under real life conditions, the Biosensor has the potential to simplify access to effective radical cure of P. vivax malaria.

Underlying database.

(XLSX)

Click here for additional data file.

Bland Altman Plot: Hb measured by Biosensor and Hemocue. Red horizontal lines indicate upper and lower end of 95% limit of agreement, green line indicates mean difference.

(TIF)

Click here for additional data file.

G6PD activity/ site / control by Biosensor.

(TIF)

Click here for additional data file.

Low Controls: G6PD activity / site / assay.

(TIF)

Click here for additional data file.

Intermediate Controls: G6PD activity / site / assay.

(TIF)

Click here for additional data file.

High Controls: G6PD activity / site / assay.

(TIF)

Click here for additional data file.

G6PD activity / site / control by Spectrophotometry.

(TIF)

Click here for additional data file.

Ethics boards of participating sites.

(DOCX)

Click here for additional data file.

Lot numbers of ACS controls per phase.

(DOCX)

Click here for additional data file.

Coefficient of variation and median G6PD activity / Hb concentration measured per assay and device in Phase A.

(DOCX)

Click here for additional data file.

Summary of readings for Biosensor (G6PD activity and Hb), spectrophotometry, and Hemocue across 10 sites in Phase B.

(DOCX)

Click here for additional data file.

Phase B coefficient of variation (CV) of G6PD measurements by Biosensor vs Spectrophotometry and Haemoglobin measurements by Biosensor vs Hemocue.

(DOCX)

Click here for additional data file.

19 Aug 2021

Dear Dr. Ley,

Thank you very much for submitting your manuscript "The STANDARD G6PD test (SD Biosensor) shows good repeatability and reproducibility in a multi-laboratory comparison" for consideration at PLOS Neglected Tropical Diseases. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

J. Kevin Baird

Guest Editor

PLOS Neglected Tropical Diseases

Mary Lopez-Perez

Deputy Editor

PLOS Neglected Tropical Diseases

***********************

Reviewer's Responses to Questions

Key Review Criteria Required for Acceptance?

As you describe the new analyses required for acceptance, please consider the following:

Methods

-Are the objectives of the study clearly articulated with a clear testable hypothesis stated?

-Is the study design appropriate to address the stated objectives?

-Is the population clearly described and appropriate for the hypothesis being tested?

-Is the sample size sufficient to ensure adequate power to address the hypothesis being tested?

-Were correct statistical analysis used to support conclusions?

-Are there concerns about ethical or regulatory requirements being met?

Reviewer #1: This manuscript reports on and intra-laboratory and and inter-laboratory analysis of G6PD assays carried out by a commercial device called BIOSENSOR.

The results show that, overall, the agreement among multiple BIOSENSOR devices, is good, but with exceptions. The comparison among several distant laboratory must have posed considerable challenges and it is an enterprise for which the Authors are to be commended.

However, I have to raise several general criticisms.

1. The main motive for this study, as stated in the introduction and elsewhere, is to avoid hemolytic complications from primaquine or tafenoquine. The assessment of G6PDd in males is very easy: it does not require either a spectrophotometric assay or BIOSENSOR, as it can be done by the fluorescent spot test or by other inexpensive screening tests. The need for a quantitative tests regards only females. Surprisingly, this is not stated anywhere.

2. In view of the above, in order to assess BiOSENSOR the focus should be on the ‘intermediate’ group: specifically, on the success rate of the device in detecting heterozygotes with a range of G6PD activities. Unfortunately, by using a single intermediate control the work was not designed to assess this success rate. In addition, it is precisely in the intermediate group that problems were encountered.

3. The Authors should briefly explain how the BIOSENSOR works. Is G6PD measured by NADPH production, or by formazan production, or in what other way? Is Hb measured by cyanmethemoglobin production or in what other way?

4. Lyophilization is not a standard way to process blood samples for a G6PD assay (and it may be a factor in some of the problems encountered). Given the design of their study, the Authors should show what was the agreement between G6PD values obtained in one lab on a set of samples, fresh versus after lyophilisation and re-constitution; and on 0 to 48 hours after re-constitution.

5. In heterozygous (intermediate) females what matters from the point of view of hemolysis is, rather than the level of G6PD activity in a hemolysate, the proportion of G6PDd red cells within their red cell mosaicism. Of course the two do correlate, but somewhere in the paper this important fact should be mentioned.

Reviewer #2: The overall study design is appropriate, allowing direct comparison between sites. However, as the authors also mentioned, the devices used in this study (Biosensor and Hemocue) all require the use of fresh blood. So the use of lyophilised RBCs may not faithfully reflect the real situations.

Reviewer #3: The objectives of the study are very clearly articulated, the design is appropriate, the sample set and size are clearly described and justified, the statistical analysis is robust and there are no ethical or regulatory concerns that I can tell.

There are no major additional analyses/experiments required.

--------------------

Results

-Does the analysis presented match the analysis plan?

-Are the results clearly and completely presented?

-Are the figures (Tables, Images) of sufficient quality for clarity?

Reviewer #1: Line 145. It would be important to know exactly what these controls are: I could not find any G6PD controls on the ACS site. What was the original G6PD activity on the fresh blood sample of each? I think

Here and throughout the manuscript I assume ‘high’ means a sample from a G6PD normal person: why use the term high?

Fig. 2. In this figure there are several strange features on which there is no comment in the text until the discussion, and the legend is incomplete: are all the cricles outliers? Shaded areas are said to be the recommended ranges (presumably recommended by ACS). I could not find what were the original G6PD activity values of the control samples provided; but even the spectrophotometric assay gives values way below recommended ranges in the intermediate and ‘high’ areas. This suggests that the lyophilization creates a problem. The red dotted line is said to be the median of the low controls, but I think it is the median of the intermediate controls. Similar apparent mistakes in defining the horizontal lines are replicated in the captions of Figs. 5 and 6.

Fig. 3. In the Hb measurement the agreement among devices is clearly not very good: whatever the overall statistics say, most of the data from device number 5 (from the left) are above most of the data from device number 4. Since I expect Hb to be independent of G6PD, I don’t understand why the reference (Hemocue) is way higher than BIOSENSORs in low and intermediate, and about right in ‘high’.

Fig. 4. The overall correlation between BIOENSOR and spectrophotometry is obviously OK; but the fact remains that in the intermediate group the bulk is about 3 IU/G Hb by spectrophotometry, and it is instead about 2 by BIOSENSOR. The superiority of spectrophotometry is supported by Suppl figure S4.

Fig. 5. In phase B, by the Authors’ own admission, the low values of the ‘high’ controls and the lack of discrimination between low and intermediate controls are a bit of a disaster.

Fig. 6. In spite of considerable scattering of values among different labs, the results are clearly better by spectrophotometry than by BIOSENSOR.

Reviewer #2: Yes.

Reviewer #3: The results are clearly and completely presented and the supplementary data adds to the manuscript.

The figures and tables are of sufficient quality and clarity, I would however suggest to combine figures 2 & 3 and figures 5 & 6 in a panel figures so as to make direct comparison easier for the reader.

In addition, I would revise the tables and condense the information into fewer tables/add some of the tables to the supplementary data.

--------------------

Conclusions

-Are the conclusions supported by the data presented?

-Are the limitations of analysis clearly described?

-Do the authors discuss how these data can be helpful to advance our understanding of the topic under study?

-Is public health relevance addressed?

Reviewer #1: Overall, the Authors have given a reasonably critical account of the shortcomings of BIOSENSOR emerging from their study. The emphasis on reproducibility, related to precision, cannot overcome the rather glaring problem with accuracy.

The statement on lines 388-393 suggests that using lyophilized material was a weak point in the design of the study.

Line 414 onwards. It is a bit of an anti-climax that, having conducted a deliberate study on agreement among multiple sites, the authors have to quote previous studies on the reliability of BIOSENSOR; and they then proceed to state that their work needs to be “confirmed in clinical settings”, after it has been found wanting in experienced laboratory settings.

Reviewer #2: Yes.

Reviewer #3: The conclusions are supported by the data and the limitations of the analysis are described in some detail. The authors briefly touch on how the data can inform the topic under study and briefly touch on public health relevance.

--------------------

Editorial and Data Presentation Modifications?

Use this section for editorial suggestions as well as relatively minor modifications of existing data that would enhance clarity. If the only modifications needed are minor and/or editorial, you may wish to recommend “Minor Revision” or “Accept”.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

Reviewer #3: (No Response)

--------------------

Summary and General Comments

Use this section to provide overall comments, discuss strengths/weaknesses of the study, novelty, significance, general execution and scholarship. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. If requesting major revision, please articulate the new experiments that are needed.

Reviewer #1: 1. The main motive for this study, as stated in the introduction and elsewhere, is to avoid hemolytic complications from primaquine or tafenoquine. The assessment of G6PDd in males is very easy: it does not require either a spectrophotometric assay or BIOSENSOR, as it can be done by the fluorescent spot test or by other inexpensive screening tests. The need for a quantitative tests regards only females. Surprisingly, this is not stated anywhere.

2. In view of the above, in order to assess BiOSENSOR the focus should be on the ‘intermediate’ group: specifically, on the success rate of the device in detecting heterozygotes with a range of G6PD activities. Unfortunately, by using a single intermediate control the work was not designed to assess this success rate. In addition, it is precisely in the intermediate group that problems were encountered.

3. The Authors should briefly explain how the BIOSENSOR works. Is G6PD measured by NADPH production, or by formazan production, or in what other way? Is Hb measured by cyanmethemoglobin production or in what other way?

4. Lyophilization is not a standard way to process blood samples for a G6PD assay (and it may be a factor in some of the problems encountered). Given the design of their study, the Authors should show what was the agreement between G6PD values obtained in one lab on a set of samples, fresh versus after lyophilisation and re-constitution; and on 0 to 48 hours after re-constitution.

5. In heterozygous (intermediate) females what matters from the point of view of hemolysis is, rather than the level of G6PD activity in a hemolysate, the proportion of G6PDd red cells within their red cell mosaicism. Of course the two do correlate, but somewhere in the paper this important fact should be mentioned.

Reviewer #2: This study evaluated the repeatability and reproducibility of the quantitative G6PD device (SD Biosensor) using standard commercial controls with high, intermediate and low G6PD activities. The results showed high repeatability and reproducibility of the device in a multi-lab setting. However, the study showed significant variations of the results from spectrometry-based measurements, which the authors speculated may have resulted from the variations from Hemocue measurement of the hemoglobin (Hb) level. As the authors discussed, the major caveat is that both the SD Biosensor and Hemocue are designed for using fresh blood, whereas the study used lyophilized controls.

1. While the study design using the same lots of controls allows direct comparison of the performance of the SD Biosensor in different labs, all controls were non-anemic and had a similar level of Hb. In endemic conditions, many vivax patients are anemic with much lower levels of Hb. Given that the variations in results may derive from the Hb measurement, it would be appropriate if the measurement was also done using diluted controls to mimic anemic situation.

2. Comparison between the SB biosensor and spectrometry results showed that the latter could differentiate the low from intermediate levels of G6PD activities, whereas SD biosensor could not. It is doubted whether the discrepancy indeed reflects the lower sensitivity of SD biosensor in discriminating low vs intermediate level activity or the standard controls tested in this study are not appropriate.

Minor comments

1. Need to synchronize spellings (Br or Am, but not both) – e.g., line 37, haemolysis; line 81, hemolysis; line 43, lyophilised; line 54, lyophilized

2. Line 214-217: please clarify whether the comparison was done for Low, Intermidiate or High separately, or these reflect the combined results.

3. Figure 2: specify the shared areas are recommended….. for the Biosensor device. Also clarify what do the dots represent (individual measurement)?

4. Line 223-224. Why do different dotted lines represent the same? I think red line shows intermediate controls, while green dotted line shows the high control. The same mistakes are also found in Fig 5 and Fig 6 legends.

5. Line 236: It shows that the normalized activity was positively correlated. Are they significantly different (for Fig 1, Intermediate, it looks like the SD biosensor readings and spectrometry results are quite different).

6. For the supplementary tables, please include the statistical results in these tables too.

Reviewer #3: The authors present G6PD enzyme activity measurement comparisons by two different methods in their paper titled: The STANDARD G6PD test (SD Biosensor) shows good repeatability and reproducibility in a multi-laboratory comparison. The paper is very well written and the data analysis and presentation is clear.

Please find below some minor comments I would like to make:

1. Would the authors please elaborate in more detail on the differences in between the recommended range set by the manufacturer and the range found by Biosensor/spectrometry. How do the company determine G6PD activity ranges? Do the company give any more information with regards to the controls i.e. from male or female, SNPs in question, country of origin of controls?

2. Would the authors please discuss in more detail the inability of Biosensor to distinguish in between low and intermediate G6PD activity, especially in phase B. It seems that the reason why there is so much overlap in phase B, is that the median low G6PD activity is higher and the median intermediate G6PD activity is similar to phase A. How, if at all, would this be influenced by different batches?

3. Would the authors please discuss the potential relevance of the inability to distinguish in between low and intermediate G6PD activity? What will it mean in terms of giving PQ to people/deciding on cut-offs/dosage regimes etc.?

The authors only describe data acquisition on controls and in well-equipped laboratory settings. It will be highly interesting to see results from field studies using the Biosensor.

--------------------

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Lynn Grignard

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

20 Sep 2021

Submitted filename: Ley_B-et al_Reponse to reviewers_v1.0.docx

Click here for additional data file.

25 Oct 2021

Dear Dr Ley,

Thank you very much for submitting your manuscript "The STANDARD G6PD test (SD Biosensor) shows good repeatability and reproducibility in a multi-laboratory comparison" for consideration at PLOS Neglected Tropical Diseases. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

J. Kevin Baird

Guest Editor

PLOS Neglected Tropical Diseases

Mary Lopez-Perez

Deputy Editor

PLOS Neglected Tropical Diseases

***********************

Reviewer's Responses to Questions

Key Review Criteria Required for Acceptance?

As you describe the new analyses required for acceptance, please consider the following:

Methods

-Are the objectives of the study clearly articulated with a clear testable hypothesis stated?

-Is the study design appropriate to address the stated objectives?

-Is the population clearly described and appropriate for the hypothesis being tested?

-Is the sample size sufficient to ensure adequate power to address the hypothesis being tested?

-Were correct statistical analysis used to support conclusions?

-Are there concerns about ethical or regulatory requirements being met?

Reviewer #1: As I stated before, the comparison among several distant laboratory must have posed considerable challenges and it is an enterprise for which the Authors are to be commended.

Reviewer #2: (No Response)

Reviewer #3: Yes

--------------------

Results

-Does the analysis presented match the analysis plan?

-Are the results clearly and completely presented?

-Are the figures (Tables, Images) of sufficient quality for clarity?

Reviewer #1: The Authors have responded to all queries, but with respect to some important ones I find their response not satisfactory.

General criticism 1. The Authors refuse to make the simple statement that the diagnosis of G6PD deficiency in males is easy and does not require a Biosensor. They prefer to say “especially females”.

General criticism 2. I understand that the purpose of the study was precision, because on accuracy ‘there are already published data’. To me it is a reason of concern that, with accuracy already established, it is not confirmed in this work. It does not make sense to me that the need for a quantitative test is at the basis of promoting Biosensor, and then the Authors do not know the original (true) values of G6PD activity in the samples they have distributed: I find this unacceptable.

General criticism 3. This clarification is appropriate: I think it is a good addition to the manuscript.

General criticism 4. The Authors essentially seem to agree that lyophilization was not a good idea (as pointed out also by another Reviewer). In principle, one might question whether the repeatability assessed on lyophilized samples would be valid for fresh samples. This is buttressed by the Authors’ own statement that “Determining accuracy of Biosensor Hb readings against the Hemocue reference assay with reconstituted lyophilized controls is of limited clinical relevance.” This puts into question the entire Hb data, that are half of the data in the paper.

General criticism 5. The added sentence is not well phrased. The G6PD activity measured in a hemolysate depends on the ratio between the two cell populations; the potential severity of hemolysis depends on the proportion of G6PD deficient red cells, because these are the ones susceptible to hemolysis from primaquine or tafenoquine.

Additional comments.

I was interested and not surprised that the “ACS recommend that laboratories develop their own in-house ranges”. This has either not been done, or the data are not shown: in my view this should be mentioned in the discussion as a limitation. In addition, I still think that readers may be confused by the fact that activity of normal samples is called “high”. As for intermediate activity, I can only reiterate that since this is the important range, it is unfortunate that only one set of samples was used within this range.

The answer to my criticism of Fig. 5 is unsatisfactory. The Authors now say that “Clinical data will be important to further investigate the poor discriminatory power of the Biosensor at low G6PD activities“: how on earth will “clinical data” help, when the device has failed in highly qualified labs?

Reviewer #2: (No Response)

Reviewer #3: Yes

--------------------

Conclusions

-Are the conclusions supported by the data presented?

-Are the limitations of analysis clearly described?

-Do the authors discuss how these data can be helpful to advance our understanding of the topic under study?

-Is public health relevance addressed?

Reviewer #1: Considering all of the above, and the many limitations of this study, some of which are admitted by the Authors, I find the claim of ‘consistent and robust performance’ (line 428) seriously inconsistent.

Reviewer #2: (No Response)

Reviewer #3: Yes

--------------------

Editorial and Data Presentation Modifications?

Use this section for editorial suggestions as well as relatively minor modifications of existing data that would enhance clarity. If the only modifications needed are minor and/or editorial, you may wish to recommend “Minor Revision” or “Accept”.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

Reviewer #3: N/A

--------------------

Summary and General Comments

Use this section to provide overall comments, discuss strengths/weaknesses of the study, novelty, significance, general execution and scholarship. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. If requesting major revision, please articulate the new experiments that are needed.

Reviewer #1: In summary, there are many unsatisfactory results in the performance of Biosensor in this study, with respect to both Hb and G6PD activity. The whole point of a quantitative test versus a qualitative test ought to be to detect heterozygotes at risk, and the device has proven wanting precisely in this respect. The Authors correctly recognize that Biosensor is inferior to the standard spectrophotometric assay. None of these serious problems is reflected in the Abstract, that gives instead the impression that everything is OK. The abstract should state instead that there are problems with the Biosensor and that the device needs to be improved.

Reviewer #2: The manuscript has been satisfactorily revised to address all my comments and suggestions.

Reviewer #3: The authors have addressed all the reviewers' comments and made changes to the manuscript accordingly.

--------------------

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Liwang Cui

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

18 Nov 2021

Submitted filename: BREIN_Rebuttal2_v2.docx

Click here for additional data file.

10 Dec 2021

Dear Dr Ley,

Thank you very much for submitting your manuscript "The STANDARD G6PD test (SD Biosensor) shows good repeatability and reproducibility in a multi-laboratory comparison" for consideration at PLOS Neglected Tropical Diseases. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Reviewer #1 of the original and revised submissions recommended rejection of the first revision and has declined to review the second revision. As the serving guest editor of this manuscript, I am therefore forced to weigh the merits of Reviewer #1’s strong criticisms and those of the responses and revisions of the authors. After careful study of these factors, along with a read of the second revision (unseen by Reviewer #1, henceforth “Reviewer”), I would be prepared to accept this manuscript with the following recommendations:

1. The title should be revised to describe the study theme and design. It is not necessary to declare that which may arguably mislead one to perceive proven suitability for purpose. This study does not do that. Something like; “Repeatability and reproducibility of a portable quantitative G6PD test device within and among users and laboratories”

2. In the abstract, line 60, the authors seem to infer inferiority of the spectrophotometric standard compared to the POC device. This is unnecessary. Simply express that good repeatability and inter-laboratory reproducibility occurred, but the device did not reliably discriminate low and intermediate samples. The argument for set global cut-off values guiding 8-aminoquinoline treatment decisions (line 78 & elsewhere) may not be supported by observations limited to relatively very few specimens, none of which represented fresh peripheral blood.

3. If the technicians involved were not trained and certified as competent/capable in the spectrophotometric assay (as they were with the Biosensor), this should perhaps be expressed. If that is so, you may wish to express less firmly that statement at line 374-6.

4. The authors have failed to make an important point (in my opinion). The intended use of the instrument is to generate a number that informs a “go” vs. “no go” decision on 8-aminoquinoline administration. That number is 70-80% of normal G6PD activity (GSK or WHO), below which fall all “intermediate” and “deficient” phenotypes. The lack of discernment of “low” and “intermediate” by the Biosensor is thus of no practical consequence. You see my opinion on this diametrically opposes that of the Reviewer. Separating “low” and “intermediate” is certainly critical for scientific study of the phenomenon and solving the clinical problem, but in the hands of the intended end-user there is no need to separate them – both safely get the “no go” on treatment. It is a single number, greater than, or less than – the only precision of importance is in that classification.

5. Your concluding statement (line 438-40) exceeds the limitations imposed by your experimental approach. Your work convincingly demonstrates reproducibility and repeatability of the Biosensor. Your work certainly does not prove the instrument “offers reliable quantitative diagnosis of G6PD status” – how could you possibly prove that with this design? Your evidence, in fact, seems to argue the opposite conclusion. You may articulate that the findings are consistent with an expectation that the instrument may consistently inform a safe treatment decision based on locally relevant thresholds for that decision.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

J. Kevin Baird

Guest Editor

PLOS Neglected Tropical Diseases

Mary Lopez-Perez

Deputy Editor

PLOS Neglected Tropical Diseases

***********************

Reviewer #1 of the original and revised submissions recommended rejection of the first revision and has declined to review the second revision. As the serving guest editor of this manuscript, I am therefore forced to weigh the merits of Reviewer #1’s strong criticisms and those of the responses and revisions of the authors. After careful study of these factors, along with a read of the second revision (unseen by Reviewer #1, henceforth “Reviewer”), I would be prepared to accept this manuscript with the following recommendations:

1. The title should be revised to describe the study theme and design. It is not necessary to declare that which may arguably mislead one to perceive proven suitability for purpose. This study does not do that. Something like; “Repeatability and reproducibility of a portable quantitative G6PD test device within and among users and laboratories”

2. In the abstract, line 60, the authors seem to infer inferiority of the spectrophotometric standard compared to the POC device. This is unnecessary. Simply express that good repeatability and inter-laboratory reproducibility occurred, but the device did not reliably discriminate low and intermediate samples. The argument for set global cut-off values guiding 8-aminoquinoline treatment decisions (line 78 & elsewhere) may not be supported by observations limited to relatively very few specimens, none of which represented fresh peripheral blood.

3. If the technicians involved were not trained and certified as competent/capable in the spectrophotometric assay (as they were with the Biosensor), this should perhaps be expressed. If that is so, you may wish to express less firmly that statement at line 374-6.

4. The authors have failed to make an important point (in my opinion). The intended use of the instrument is to generate a number that informs a “go” vs. “no go” decision on 8-aminoquinoline administration. That number is 70-80% of normal G6PD activity (GSK or WHO), below which fall all “intermediate” and “deficient” phenotypes. The lack of discernment of “low” and “intermediate” by the Biosensor is thus of no practical consequence. You see my opinion on this is diametrically opposes that of the Reviewer. Separating “low” and “intermediate” is certainly critical for scientific study of the phenomenon and solving the clinical problem, but in the hands of the intended end-user there is no need to separate them – both safely get the “no go” on treatment. It is a single number, greater than, or less than – the only precision of importance is in that classification.

5. Your concluding statement (line 438-40) exceeds the limitations imposed by your experimental approach. Your work convincingly demonstrates reproducibility and repeatability of the Biosensor. Your work certainly does not prove the instrument “offers reliable quantitative diagnosis of G6PD status” – how could you possibly prove that with this design? Your evidence, in fact, seems to argue the opposite conclusion. You may articulate that the findings are consistent with an expectation that the instrument may consistently inform a safe treatment decision based on locally relevant thresholds for that decision.

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article's retracted status in the References list and also include a citation and full reference for the retraction notice.

22 Dec 2021

Submitted filename: BREIN_rebuttal3.docx

Click here for additional data file.

17 Jan 2022

Dear Dr Ley,

We are pleased to inform you that your manuscript 'Repeatability and reproducibility of a handheld quantitative G6PD diagnostic' has been provisionally accepted for publication in PLOS Neglected Tropical Diseases.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Neglected Tropical Diseases.

Best regards,

J. Kevin Baird

Guest Editor

PLOS Neglected Tropical Diseases

Mary Lopez-Perez

Deputy Editor

PLOS Neglected Tropical Diseases

***********************************************************

26 Jan 2022

Dear Dr Ley,

We are delighted to inform you that your manuscript, " Repeatability and reproducibility of a handheld quantitative G6PD diagnostic ," has been formally accepted for publication in PLOS Neglected Tropical Diseases.

We have now passed your article onto the PLOS Production Department who will complete the rest of the publication process. All authors will receive a confirmation email upon publication.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any scientific or type-setting errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript. Note: Proofs for Front Matter articles (Editorial, Viewpoint, Symposium, Review, etc...) are generated on a different schedule and may not be made available as quickly.

Soon after your final files are uploaded, the early version of your manuscript will be published online unless you opted out of this process. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting open-access publishing; we are looking forward to publishing your work in PLOS Neglected Tropical Diseases.

Best regards,

Shaden Kamhawi

co-Editor-in-Chief

PLOS Neglected Tropical Diseases

Paul Brindley

co-Editor-in-Chief

PLOS Neglected Tropical Diseases