WHO multicenter evaluation of FACSCount CD4 and Pima CD4 T-cell count systems: instrument performance and misclassification of HIV-infected patients.

BACKGROUND: CD4⁺ T-cell counts are used to screen and follow-up HIV-infected patients during treatment. As part of the World Health Organization prequalification program of diagnostics, we conducted an independent multicenter evaluation of the FACSCount CD4 and the Pima CD4, using the FACSCalibur as reference method.
METHODS: A total of 440 paired capillary and venous blood samples were collected from HIV-infected patients attending the HIV outpatient clinic in Antwerp, Belgium, and the HIV care and treatment center in Dar es Salam, Tanzania. Capillary blood was run on Pima analyzer, whereas venous blood was analyzed on FACSCount, Pima, and FACSCalibur instruments. Precision and agreement between methods were assessed.
RESULTS: The FACSCount CD4 results were in agreement with the FACSCalibur results with relative bias of 0.4% and 3.1% on absolute CD4 counts and an absolute bias of -0.6% and -1.1% on CD4% in Antwerp and Dar es Salam, respectively. The Pima CD4 results were in agreement with the FACSCalibur results with relative bias of -4.1% and -9.4% using venous blood and of -9.5% and -0.9% using capillary blood in Antwerp and Dar es Salam, respectively. At the threshold of 350 cells per microliter, the FACSCount CD4 and Pima CD4 using venous and capillary blood misclassified 7%, 9%, and 13% of patients, respectively.
CONCLUSIONS: The FACSCount CD4 provides reliable CD4 counts and CD4% and is suitable for monitoring adult and pediatric HIV patients in moderate-volume settings. The Pima CD4 is more suitable for screening eligible adult HIV patients for antiretroviral treatment initiation in low-volume laboratories.

INTRODUCTION

Laboratory monitoring of HIV-infected patients receiving antiretroviral treatment (ART) is currently done by measuring HIV viral load and counting CD4 T cells. As many middle- and low-income countries do not have regular access to viral load testing, CD4 T-cell enumeration is the more common biological assay used to monitor ART.1,2 CD4 T-cell enumeration is used to start chemoprophylaxis against opportunistic infections and is also recommended as marker to identify patients in need of ART.3 Single-platform (SP) flow cytometry is the most preferred reference method for CD4 T-cell enumeration.4 However, classical and dedicated flow cytometers are still very expensive and operated by highly trained personnel. In addition, they require stable electricity supply, a cold chain to transport and store reagents, and regular instrument maintenance services, which are not readily available in most low-income countries.3,5,6 The FACSCount (BD Biosciences, Erembodegem, Belgium) is the first flow cytometer dedicated for absolute CD4 counting and is mainly used in resource-limited countries since 1996.7 Recently, new reagents and software have been developed to allow the additional measurements of CD4% essential for monitoring pediatric patients.8,9 Other small flow cytometers dedicated for CD4 counting have been introduced in the past decade and are also mainly used in resource-limited countries. These include CyFlow Counter (Partec, Munster, Germany), Apogee Auto40 (Apogee Flow System, Hemel Hempstead, United Kingdom), and Guava EasyCD4 (Merck Millipore, Billerica, MA).10–17 Most of these instruments can provide both CD4 T-cell counts and CD4%.8,17–19 More recently, alternative non–flow cytometry-based CD4 T-cell counting devices have been released in the market. Pima CD4 (Alere, Jena, Germany) has been introduced a few years ago as a point-of-care (POC) device, which uses either venous or capillary blood. However, it only provides absolute CD4 T-cell counts and thus is less suited for monitoring pediatric HIV-infected patients.20–26 Other CD4 POC technologies are in the pipeline; some are currently evaluated but are not yet available in the market.27–30 An overview of all existing and newly emerging CD4 technologies is provided in a recent UNITAID technical report on the diagnostic technology landscape and in recently published reviews.31,32

The World Health Organization (WHO) recommended initiating ART in HIV patients with CD4 count up to 350 cells per microliter in 2010 and in 2013 raised the threshold up to 500 cells per microliter in specific patient populations.33,34 Misclassification of HIV patients at these thresholds may sometimes have grave outcomes. Very few studies have reported the ability of these assays to correctly classify patients at these critical thresholds.20,23,25,35 Laboratory and field evaluation of the CD4 systems is part of the prequalification of diagnostic program and is conducted after a successful review of the product dossier by the WHO. The assessment of the performance and operational characteristics of the product is one of the final phases and is carried out by WHO collaborating centers. As part of this program, we conducted an independent multicenter evaluation of the recently introduced FACSCount CD4 to measure absolute CD4 T-cell counts and CD4% and of the Pima CD4 using venous and capillary blood. The main objective was to assess their operational characteristics and performance, including patients' misclassification probabilities.

MATERIALS AND METHODS

Study Population

The study was approved by the Institutional Review Board of the Institute of Tropical Medicine (ITM) and the University of Antwerp, Belgium, and by the Senate Research and Publications Committee of Muhimbili University of Health and Allied Sciences in Dar es Salam, Tanzania. Study participants were recruited among HIV-infected patients presenting for routine CD4 T-cell counting at the HIV outpatient clinic of the ITM Antwerp and at the Infectious Diseases Clinic in Dar es Salam. At least 200 participants were enrolled per site, targeting 50 patients with CD4 T cells <200 per microliter, 100 with CD4 T cells between 200 and 500 per microliter, and 50 with CD4 T cells >500 per microliter. The recruitment was initially done consecutively and was then focused on patients with low CD4 count looking at their previous CD4 T-cell count. CD4 enumeration was part of the routine follow-up, and study participants signed an informed consent before enrollment to provide an additional capillary blood sample. Capillary blood was collected from a finger stick directly onto Pima cartridges and venous blood from venepuncture in K3 EDTA vacutainer tubes.

Description of the Technologies

The new BD FACSCount CD4 assay is an updated version of the BD FACSCount with an ability to provide percentage of CD4 T cells (CD4%) in addition to absolute CD4 T cells.

The Alere Pima CD4 is an automated image-based immune hematology POC test intended for rapid in vitro quantitative measurement of absolute CD3+CD4+ T cells from capillary or venous blood samples.

Precision Assessment

Precision of FACSCount CD4 included instrument precision, intra-assay variation, inter-assay variation, and carry over. Precision of Pima CD4 was evaluated on venous blood only and included intra-assay, inter-assay, intra-instrument, and inter-instrument variabilities. Instrument precision, not applicable on blood samples (single-use cartridges), was determined using the Pima control beads. Carry over was not applicable on Pima CD4.

For the FACSCount CD4, the instrument precision (run to run) was assessed on 15 different blood specimens containing 100–300 cells per microliter. Each stained sample was run 10 times or as many times as possible if less than 10 because of volume shortage.

The intra-assay variability assesses the tube-to-tube variability and includes the variation induced by pipetting errors made by the operator. The intra-assay variability was determined on 10 different blood samples with CD4 T cells ranging from 100 to 300 per microliter. For each blood sample, the entire CD4 staining procedure and sample acquisition were repeated 10 times. In Antwerp, for Pima CD4, each of the 10 venous blood samples was run twice on each of the 5 Pima analyzers, for a total of 10 readings per blood sample. In Dar es Salam, each blood sample was read 10 times on the same device, and 10 different blood samples were run using 6 different Pima analyzers.

The inter-assay variation, which assesses the day-to-day variation, was determined on 10 consecutive blood samples (7 with 100–300 cells per microliter and 3 with 301–550 cells per microliter). An aliquot from the blood samples was stained 3 times: at 6, 24, and 48 hours after specimen collection, with storage at room temperature.

The carry over assessment determines if the result of a high–CD4 count sample has an influence on the result of a subsequent low–CD4 count sample. This effect was studied by analyzing 5 batches of 2 different blood samples, one with a high CD4 count and the other with low CD4 count. The high–CD4 count sample (>600 per microliter) was read in duplicate (recorded as a1 and a2) followed by the duplicate reading of the low–CD4 count (100–300 cells per microliter) sample (recorded as b1 and b2). The carry over (k) is defined by k = (b1 − b2) × 100/(a2 − b2).36

Additionally, in Antwerp, we assessed the true intra-assay variability by running 1 fresh venous blood sample 10 times on each of the 5 Pima devices. By this, we also calculated the inter-instrument variability (device to device), comparing the 5 Pima devices used in this study.

An acceptable assay should have the percent coefficient of variation (%CV) less than 15% (or 30 cells) for CD4 counts ≤200 cells per microliter and less than 10% for CD4 counts >200 cells per microliter as agreed in WHO prequalification protocol.37,38 The acceptable carry over must be less than 2%.

Agreement Between Methods

Routine venous blood samples brought into the laboratory and capillary blood samples collected by finger stick from the same patients were used to determine agreement between methods. Capillary blood samples were run on 1 of the 5 different Pima CD4 analyzers within 5 minutes after finger-prick collection. Venous blood samples were collected and stored at ambient temperature (17–25° C) in the outpatient clinic before being transported to the laboratory for analysis within 6 hours after venipuncture on FACSCalibur using Trucount tubes, FACSCount, FACSCount CD4, and Pima CD4 analyzers. For each capillary sample, the corresponding venous sample was run on the same Pima analyzer to avoid inter-instrument variability. All tests were performed according to the manufacturer's instructions. Before running samples, specific control beads were successfully tested daily on FACSCalibur and Pima analyzers and weekly on FACSCount. In addition, the Multicheck controls (BD Biosciences) were run daily before samples to ensure the accuracy and reliability of the FACSCalibur reference system. CD4 T-cell measurements on FACSCalibur reference and on the evaluated instruments were performed by different operators to allow a blind reading.

The reference method for CD4 counting used in this study was the FACSCalibur in combination with Trucount (BD Biosciences). Trucount is an established flow cytometric CD4 cell counting assay, which allows SP measurements of both absolute CD4 T-cell counts and CD4% and has excellent repeatability and quality assurance scores.39 Briefly, 50 µL of whole venous blood and 20 µL of Multitest monoclonal antibody (mAb) reagents (CD3-FITC/CD4-PE/CD8-PerCP/CD45-APC) were pipetted into a Trucount tube, mixed, and incubated for 15 minutes. Subsequently, 450 µL of lysis solution (BD Biosciences) was added to each tube and incubated for 15 minutes before reading the samples on the FACSCalibur.

The FACSCount CD4 reagent kit consists of 50 single tubes containing each a mixture of 3 mAbs (CD4-PE/CD14-PE-Cy5/CD15-PE-Cy5), a fluorescent nuclear dye, and fluorescent beads. The FACSCount reagent kit consists of 50 twin tubes containing each a mixture of mAbs (CD3-PE-Cy5 and CD4-PE or CD8-PE) and fluorescent beads. Fifty microliters of venous blood was added into single tube and into each of the twin tubes. The tubes were capped, vortexed, and incubated for 30 minutes (single tubes) or 60 minutes (twin tubes). After incubation, 50 µL of fixative solution was added into each tube, and samples were run on the FACSCount instrument with the respective software. All incubation steps were done in the dark at room temperature.

The Pima CD4 uses disposable anticoagulant-coated cartridges preloaded with antihuman CD3-dye1 and CD4-dye2 mAbs. Capillary blood was directly collected from the finger into a cartridge according to the manufacturer's instructions. Twenty-five microliters of venous blood was loaded into a disposable cartridge. Once the control window was filled, the blood collector was removed and the cartridge capped and immediately inserted into a Pima analyzer for 20 minutes incubation followed by automatic analysis of the test sample.

Statistical Analyses

Data were analyzed using MedCalc version 10.0.2.0 (MedCalc Software, Mariakerke, Belgium). Precision expressed as the CV was determined by dividing the SD of the measurements by the mean (CV = SD × 100/mean). We calculated the CV for instrument precision and intra-assay, inter-assay, and inter-instrument variations. Measurement of linear regression was determined using Passing–Bablok regression analysis.40 Pollock and Bland–Altman41,42 analyses were used to determine the mean biases and the limits of agreement (LOA = mean ± 1.96 SD) on CD4 count and on CD4%, respectively. Percentage similarity was calculated for each sample as Similarity = Average of methods A and B × 100/Method A (with A = reference and B = evaluated). For each group, the mean percent similarity and the CV were determined.43 We first performed comparisons between each alternative method (FACSCount CD4 or Pima CD4) and the FACSCalibur (Trucount) reference system in the overall data and then within each of the 3 CD4 T-cell count categories. Second, comparisons were done between FACSCount CD4 and FACSCount and between CD4 counts from capillary blood and those from venous blood on the Pima CD4.

Misclassification probabilities were calculated at ART initiation thresholds of 200, 350, and 500 cells per microliter for CD4 counts and 25% for CD4% setting of the FACSCalibur as the reference method to determine eligible patients on ART.

RESULTS

A total of 440 HIV-infected patients were recruited at the HIV outpatient clinic of the ITM Antwerp (Belgium) and at the Infectious Diseases Clinic HIV care and treatment center in Dar es Salam (Tanzania). The characteristics of the study population recruited from the 2 sites are summarized in Table 1.

TABLE 1

Characteristics of Study Population

The intra-laboratory variability of FACSCount CD4 and Pima CD4 expressed as CV is summarized in Table 2.

TABLE 2

Intra-Laboratory Variability (%CV) of FACSCount CD4 and Pima CD4

FACSCount CD4

The FACSCount CD4 showed an instrumental (run-to-run) variability with mean CVs ranging from 3% to 6%. The intra-assay variation with mean CVs ranging from 3.2% to 6.8% is similar to the CVs of the instrumental precision.

The inter-assay (day-to-day) variation showed CVs ranging from 4% to 9% for absolute CD4 counts and from 2% to 8% for CD4%. The FACSCount CD4 provided both CD4 counts and CD4% without any significant carry over (k < 0) in both Antwerp and Dar es Salam sites.

Pima CD4

The instrumental precision of Pima CD4 was determined using values from the control beads and resulted in a CV of less than 4%.

The Pima CD4 showed an average mean intra-assay CV >10% (CV = 13.4%). However, all blood samples with CD4 T cells ≤200 per microliter showed acceptable variability (SD) of <35 per microliter as recommended by the manufacturer. For samples with CD4 counts between 200 and 300 per microliter, the mean CV was higher than 10% but still less than 15% in both study sites except for 1 sample in Dar es Salam. In Antwerp, based on 1 sample, the Pima devices showed true intra-assay CVs ranging from 6% to 12% with a mean of 9.4%. The 5 Pima CD4 devices showed an inter-instrument CV of 4.6% calculated as the statistical mean of the intra-instrument CVs.

The Pima CD4 showed inter-assay CVs ranging from 7% to 15%.

When the Multicheck controls were run on the FACSCalibur, all values provided in the 2 study sites were within the expected ranges, with a CV <6% for normal CD4 counts and <8% for low CD4 counts.

Comparison Between FACSCount CD4 and FACSCalibur (Trucount)

Figures 1A–D and E–H, respectively, show the comparisons of absolute CD4 counts and CD4% obtained from FACSCount CD4 and FACSCalibur.

FIGURE 1

Comparison between FACSCount CD4 and FACSCalibur Trucount: absolute CD4 counts obtained by FACSCount CD4 and FACSCalibur Trucount were compared by Passing–Bablok regression in Antwerp (A) and Dar es Salam (B). The corresponding graphs depicting the relative bias between the 2 instruments are represented in Pollock plots for Antwerp (C) and Dar es Salam (D). CD4% obtained by FACSCount CD4 and FACSCalibur Trucount are compared by Passing–Bablok regression in Antwerp (E) and Dar es Salam (F). The corresponding graphs depicting the absolute bias between the 2 instruments are depicted in Bland–Altman plots for Antwerp (G) and Dar es Salam (H). In Passing–Bablok regression graphs, the solid blue line represents the regression line and the dashed lines represent the 95% confidence interval for the regression line. In the Pollock and Bland–Altman graphs, the solid blue line represents the mean bias. The dashed lines represent mean bias ± 1.96 SD, which are the upper and lower LOA.

On Absolute CD4 T-Cell Counts

These 2 systems showed an excellent correlation with a slope of 1.02 in Antwerp and 1.06 in Dar es Salam. The agreement between the 2 methods was assessed by Bland–Altman and similarity analyses. The mean relative bias (LOA) and the mean similarity (CV) were 0.4% (−22.4 to 23.2) and 101% (5%) in Antwerp and 3.1% (−30.1 to 36.3) and 102% (9%) in Dar es Salam, respectively. Agreement within the different CD4 categories is summarized in Table 3.

TABLE 3

Comparison Between Alternative Methods and FACSCalibur Reference System (A) Pollock Bias (With LOA) on CD4 Counts; (B) Mean Percentage of Similarity (With %CV); and (C) Bland–Altman Bias (With LOA) on CD4%

At the threshold of 200 cells per microliter, the FACSCount CD4 misclassified 2% (10/435) of patients in both sites. Of these 10 misclassifications, the FACSCount CD4 would delay the ART initiation of only 2 patients. The sensitivity was 100% (34/34) in Antwerp and 95% (38/40) in Dar es Salam, and the specificity was 99% (201/204) and 97% (152/157) in Antwerp and Dar es Salam, respectively. At the threshold of 350 cells per microliter, the FACSCount CD4 misclassified 7% (30/435) of patients. Thus, sensitivity and specificity were 97% (91/94) and 96% (138/144) in Antwerp and 89% (86/97) and 98% (98/100) in Dar es Salam, respectively. At the threshold of 500 cells per microliter, 5% (23/435) of patients were misclassified by the FACSCount CD4, which showed sensitivity and specificity of 95% (150/158) and 96% (74/80) in Antwerp and 96% (136/142) and 95% (52/55) in Dar es Salam, respectively. The performances (agreement) of the FACSCount CD4 on absolute CD4 counts were different between Antwerp and Dar es Salam.

On percentage of CD4 T-Cells (CD4%)

The Passing–Bablok analysis on CD4% showed excellent correlation between FACSCount CD4 and FACSCalibur with a slope of 0.99 in Antwerp and 0.96 in Dar es Salam. These 2 methods showed excellent agreement with mean bias (LOA) and mean similarity (CV) of −0.6% (−3.6 to 2.4) and 99% (3%) in Antwerp and −1.1% (−5.3 to 3.1) and 98% (7%) in Dar es Salam, respectively. Agreement within different CD4 categories is summarized in Table 3. At the threshold of 25%, the FACSCount CD4 showed sensitivity of 97% (137/141) in Antwerp and 96% (123/128) in Dar es Salam and specificity of 84% (78/93) and 97% (62/64), respectively, in Antwerp and Dar es Salam. Except for the bias with a P value of 0.0053, the FACSCount CD4 showed a similar performance between Antwerp and Dar es Salam.

Comparison Between FACSCount CD4 and Standard FACSCount on Absolute Counts

The supplemental Figure (see Supplemental Digital Content 1, http://links.lww.com/QAI/A533) shows the comparison of the absolute CD4 counts obtained from FACSCount CD4 and FACSCount. The 2 absolute CD4 counts obtained on the 2 different FACSCount versions correlated well and showed excellent agreement in both Antwerp and Dar es Salam sites. The slopes were 1.02 and 0.96 in Antwerp and Dar es Salam, respectively. In addition, the mean relative bias (LOA) and similarity (CV) were 1.1% (−14.4 to 16.7) and 101% (4%) in Antwerp and 3.9% (−17.1 to 24.8) and 102% (5%) in Dar es Salam, respectively.

Comparison Between Pima CD4 and FACSCalibur (Trucount)

Figures 2A–D and E–H, respectively, illustrate the comparisons between Pima CD4 using venous blood or capillary blood and FACSCalibur.

FIGURE 2

Comparison between Pima CD4 and FACSCalibur Trucount: absolute CD4 counts obtained by Pima CD4 using venous blood and FACSCalibur Trucount were compared by Passing–Bablok regression in Antwerp (A) and Dar es Salam (B). The corresponding graphs depicting the relative bias between the 2 instruments are represented in Pollock plots for Antwerp (C) and Dar es Salam (D). Absolute CD4 counts obtained by Pima CD4 using capillary blood and FACSCalibur Trucount were compared by Passing–Bablok regression in Antwerp (E) and Dar es Salam (F). The corresponding graphs depicting the relative bias between the 2 instruments are represented in Pollock plots for Antwerp (G) and Dar es Salam (H). In Passing–Bablok regression graphs, the solid blue line represents the regression line and the dashed lines represent the 95% confidence interval for the regression line. In the Pollock graphs, the solid blue line represents the mean bias. The dashed lines represent mean bias ± 1.96 SD, which are the upper and lower LOA.

On Venous Blood Samples

These methods showed good correlation with a slope of 0.93 in Antwerp and 0.85 in Dar es Salam. Pima CD4 using venous blood showed a mean bias (LOA) of −4.1% (−29 to 20.8) and similarity (CV) of 98% (7%) in Antwerp and a mean bias of −9.4% (−54.4 to 35.6) and similarity (CV) of 98% (21%) in Dar es Salam. Agreement within the different CD4 categories is summarized in Table 3. The bias and correlation coefficients of Pima CD4 using venous blood were different between Antwerp and Dar es Salam.

At the threshold of 200 cells per microliter, 3% (14/440) of patients were misclassified by the Pima CD4 using venous blood in both sites. Only 2 patients would have their ART initiation delayed when relying on the Pima CD4 compared with FACSCalibur. The sensitivity and specificity were 97% (34/35) and 98% (201/205) in Antwerp and 98% (40/41) and 95% (151/159) in Dar es Salam, respectively. At the threshold of 350 cells per microliter, 9% (40/440) of HIV patients were misclassified relying on the Pima CD4 using venous blood, which would delay the ART initiation in 7 patients. The sensitivity and specificity were 96% (92/96) and 91% (131/144) in Antwerp and 97% (96/99) and 80% (81/101) in Dar es Salam, respectively. At the threshold of 500 cells per microliter, sensitivity and specificity were 99% (158/160) and 85% (68/80) in Antwerp and 99% (143/145) and 78% (43/55) in Dar es Salam, respectively.

On Capillary Blood Samples

The Passing–Bablok regression plots showed slopes of 0.89 and 0.94 in Antwerp and Dar es Salam, respectively. The Pima CD4 using capillary blood showed a mean bias (LOA) and a mean similarity (CV) of −9.5% (−46.8 to 27.9) and 96% (12%) in Antwerp and −0.9% (−57.3 to 55.6) and 102% (21%) in Dar es Salam, respectively. The Pima CD4 using capillary blood showed different performances between Antwerp and Dar es Salam.

At the threshold of 200 cells per microliter, 4% (16/410) of patients were misclassified by the Pima CD4 using capillary blood in both sites. None of the patients would have their ART initiation delayed relying on Pima CD4 using capillary blood. The sensitivity and specificity were 100% (26/26) and 96% (177/184) in Antwerp and 100% (41/41) and 94% (150/159) in Dar es Salam, respectively. At the threshold of 350 cells per microliter, 13% (55/410) of patients were misclassified by the Pima CD4 using capillary blood. Sixteen out of 55 patients would have the initiation of ART delayed. The sensitivity and specificity were 96% (75/78) and 78% (103/132) in Antwerp and 87% (86/99) and 90% (91/101) in Dar es Salam, respectively. At the threshold of 500 cells per microliter, the sensitivity was 98% (134/137) and 97% (140/145) and the specificity was 73% (53/73) and 82% (45/55) in Antwerp and Dar es Salam, respectively.

Comparison Between Capillary Blood and Venous Blood on Pima CD4

The Figure, SDC 2, http://links.lww.com/QAI/A533, compares CD4 counts in capillary and venous blood read on Pima CD4. Absolute CD4 counts measured in capillary blood correlated well with those in venous blood in both Antwerp and Dar es Salam. The Passing–Bablok regression showed a slope of 0.97 in Antwerp and 1.12 in Dar es Salam. The mean bias (LOA) and similarity (CV) were −5.5% (−48.0 to 37.0) and 98.6% (14%) in Antwerp and 8.4% (−38.2 to 55.1) and 106.3% (15%) in Dar es Salam, respectively.

Error Rates

A total of 33 errors were reported during the study on FACSCount CD4. Of these, 15 errors were from agreement assessment and 18 from precision assessment. Of the 33 errors reported, 3 samples failed to provide values even after repeating the tests; the 30 remaining provided CD4 counts but failed to provide CD4%. On Pima CD4 analyzers, 54 errors were reported during the study. Of these, 31 were from agreement assessment, 10 from precision assessment, and 13 from Pima standard beads. In Antwerp, capillary blood showed more errors (30/32) than venous blood, and 63.2% were attributed to 1 of the 3 nurses performing the finger-stick collections. In Dar es Salam, the venous blood showed a higher rate of errors of 59% (13/22) than the capillary blood, and 61.5% of errors were attributed to 1 of the 6 Pima CD4 operators.

DISCUSSION

Whereas previous studies assessed the performance of FACSCount CD48 and Pima CD4,21–26,35,44,45 our study is the first independent study conducted by the WHO to evaluate the operational characteristics of these systems together with their performance, including error rates.

The FACSCount CD4 system provides absolute CD4 counts up to 5000 cells per microliter and associated CD4% rendering this system particularly suitable for monitoring pediatric HIV-infected patients. The FACSCount CD4 showed very good intra-assay CVs similar to the CVs of the instrumental variation, suggesting that the variability induced by the operator on FACSCount CD4 results is limited. The FACSCount CD4 was compared with the Double-Platform FACSCan and FACSCount in Thailand, but our study is the first to compare it with the SP FACSCalibur. Our results are in agreement with those reported by Pattanapanyasat et al8 who showed an R2 >0.97, an absolute bias (LOA) of 3.39 (−52.5 to 59.3) cells per microliter, and a mean similarity of 98.2%.

The Pima CD4 is a battery-powered imaging device that only provides absolute CD4 counts from either capillary or venous blood using disposable cartridges containing lyophilized mAbs. The use of the Pima CD4 system reduces the loss of patients from follow-up, thanks to the short turnaround time for providing CD4 results (20 min). Thus, the physician can immediately take the adequate decision to start or switch ART regimen. The use of Pima CD4 system eliminates the need of a cold chain and air conditioning (except in very hot areas), as its disposable cartridges can be stored up to 30° C and the instrument can be operated up to 40° C. The Pima CD4 showed an intra-assay variation with mean %CV equal or larger than the recommended value of 10% (or 15% in low CD4 counts). This could be partly because of the fact that the precision was measured on 5 different instruments, adding an extra inter-instrument variability to the intra-assay variability itself as was shown in Antwerp (CV = 9.4%). In addition, the Pima CD4 showed intra-assay CVs higher than the instrumental CVs, suggesting that operator added a significant variability. This is in line with the results of previous studies that showed excellent instrumental precision with mean %CV of <5% reflecting the significant contribution of operators to the %CV.20,22 The Pima CD4 results using venous blood showed acceptable correlation and agreement with the FACSCalibur results, similar to those reported by previous studies20–23,26,46 and partially in agreement with the study by Mwau et al47 in Kenya, comparing Pima CD4 with FACSCount. Mwau et al47 obtained discordant results comparing Pima CD4 with FACSCalibur but concordant results comparing Pima CD4 with FACSCount. This discrepancy could be because of less reliable results generated by the FACSCalibur used, which showed a substantial absolute bias (LOA) against the FACSCount of −76.5 cells per microliter (−316 to 163). Using capillary blood, the Pima CD4 showed good correlation and acceptable agreement with the FACSCalibur similar to those reported by previous studies.21,23,25,46 The number of operators (6 in Dar es Salam, against 1 operator for venous blood and 3 nurses performing finger prick in Antwerp) may explain the performance differences observed between the 2 study sites, as shown in previous studies.21,22

Looking at screening of patients in need of treatment, rates of misclassification are in agreement with those reported in previous studies, with a better identification of patients based using venous blood instead of capillary blood on the Pima CD4, even though other studies reported higher misclassification rates using capillary blood.21,22 A higher threshold for ART initiation increased the number of misclassified patients. However, the consequences for patients who were upward misclassified at 350 or 500 cells per microliter will be less severe than for patients misclassified upward at 200 cells per microliter. In practice, this delay might be of little importance when the decision is postponed for one visit or in settings where the clinicians take the CD4 count decrease into account instead of the CD4 threshold or even other parameters. Fortunately, when relying on the FACSCount CD4 or the Pima CD4 results rather than on the FACSCalibur results, the proportion of patients for whom the ART initiation would be delayed is much smaller than the proportion of patients who would have been treated too early. This early initiation will improve the well-being of patients and consequently reduce their risk of transmitting HIV.48,49 Even if this means that the national programs would have to spend more money to ART, early treatment reduces the risk of opportunistic infections. The only drawback could be that upon ART shortage, some patients who do not require immediate treatment are treated at the expense of patients who require ART more urgently.

Besides the misclassification because of poor instrument precision, physicians need to take into account physiological variation of the CD4 T-cell counts, which also might lead to misclassification. During follow-up of individual HIV-infected patients, it is highly recommended to collect blood for routine CD4 counting at the same time (either in the morning or in the afternoon) to avoid bias between 2 consecutive measurements because of diurnal variations.50,51

The failure rates (percentage of samples where no results were generated) reported in our study on Pima CD4, either on venous or capillary blood, were in agreement with those in previous studies.21,22,46 The occurrence of errors may partially be explained by the lack of experience of personnel who performs finger stick (eg, Pima CD4) and partially by samples that fail to meet the internal quality acceptance criteria implemented by the instrument software itself (eg, Pima CD4 and FACSCount CD4). Errors will obviously increase the cost per test, and repeating a finger prick may not be as simple as repeating a test using venous blood.

In Antwerp, nurses occasionally experienced difficulties to obtain a good capillary blood drop especially without squeezing the finger, which may dilute a blood sample. Alternatively, other patients experienced rather heavy finger bleeding after the finger prick, resulting in either incorrect filling of the cartridges or blood dripping which increased the risks of exposure of blood to the nurses who performed the finger prick. All these aspects make the finger prick more difficult to perform than originally expected, in particular because the training skills required for CD4 counting are more critical than, for instance, for malaria or diabetes finger-prick testing.

The operators experienced the FACSCount CD4 as a simple flow cytometer requiring 2–3 days of training. The FACSCount CD4 can run a batch of 20 samples in 90 minutes and up to 100 samples a day. The instrument operators at the 2 study sites considered the Pima CD4 as a very simple system requiring only one day for training, no need for reagent preparation or extra equipment (except a pipet for pipetting venous blood in the cartridge) to perform the assay. There is no need for a cold chain, and the instrument can be moved from one laboratory to another without the need of instrument recalibration, which is an asset when used in mobile units or for itinerant quality control programs. The Pima CD4 is suitable for a limited number of samples with a maximum daily throughput of 15 samples per instrument (See Table, SDC 3, http://links.lww.com/QAI/A533, which summarizes the characteristics of FACSCount CD4 and Pima CD4 systems).

This study was conducted in 2 reference laboratories with highly skilled personnel who received appropriate training before the start of the study. In both laboratories, the FACSCalibur was used as reference method and both instruments were monitored by external quality assessment programs (QASI means Quality Assessment and Standardization for Immunological measures relevant to HIV/AIDS and UK NEQAS means United Kingdom National External Quality Assessment Service). The 2 sites performed daily calibration of the instruments and ran blood control samples to test the accuracy of the systems. The lack of experience of nurses in Antwerp in performing finger prick for CD4 counting, the different study population at the 2 sites, and the different intra-assay variability method implemented at the 2 sites for the Pima CD4 were noticed as limitations of the study.

In conclusion, the FACSCount CD4 provides reliable absolute CD4 counts and CD4%, which are in excellent agreement with the results obtained on the FACSCalibur as CD4 counting reference method. The FACSCount CD4 is suitable for monitoring HIV-infected adults and children. The results of the Pima CD4 are in acceptable agreement with the FACSCalibur results using either capillary or venous blood. This instrument, which only provides absolute CD4 counts, is primarily suitable for screening adult HIV patients for eligibility to initiate ART in resource-poor settings. Although the Pima CD4 showed a higher misclassification probability than the FACSCount CD4, its high mobility (light and battery powered) and its independence of a cold chain make this instrument a very attractive POC CD4 device, which will increase access to fast CD4 results in more remote areas.