Applicability of the long-term uncertainty in measurement (LTUM) method for analytical performance assessment in clinical cytometry laboratories.

BACKGROUND: The estimation of uncertainty in measurement for quantitative analyses is an international obligation of the ISO15189 standard for laboratories. The most widespread method is the Internal Quality Control and External Quality Assessment (IQC + EQA).
METHODS: We compared two methods to assess uncertainty in measurement for the quantification of the number of CD34+ stem cells and of the different lymphocyte subpopulations in blood samples: the IQC + EQA method and the Long-Term Uncertainty in Measurement (LTUM) method.
RESULTS: We focused on the CD3+/CD4+ T lymphocyte subpopulation for a target value of 350 CD3+/CD4+/μl. The range in terms of uncertainty in the measurement of 350 CD3+/CD4+ cells/μl with the IQC + EQA method was [292.8; 407.2]. With the LTUM method, the uncertainty was 19.1% of the measured value. This represented a range of [283.2; 416.9].
CONCLUSIONS: The relative uncertainty calculated with the LTUM method can be adapted to any level of the measured parameter. IQC and EQA calculate the absolute uncertainty and need a clustering of values at different levels. This clustering can lead to some approximations in the uncertainty in measurement determination, particularly around the cut-off values. Unlike previous reports, uncertainty values were higher when calculated with the LTUM than with the IQC + EQA method. However, LTUM might be more representative of the daily routine practice with patient samples.

INTRODUCTION

The determination of uncertainty in measurement allows estimating the accuracy of measurements by comparing the values obtained by laboratory quantification of patient samples to reference values. The International Organization for Standardization ISO15189 (ISO 15189, 2012) requires calculating uncertainty in measurement.

Uncertainty in measurement comes from the total error of a parameter that can be subdivided in random error and systematic error. Random errors in experimental measurements are caused by unknown and unpredictable changes in the experiment. They lead to precision defects. Precision in measurement is the tightness of the values obtained by repeated measurements in specified conditions. Systematic errors are recurrent errors usually caused by measuring instruments that are incorrectly calibrated and/or used. The bias quantifies this systematic error and represents the difference between the value analyzed by the laboratory and a reference value (defined here as the mean value of each parameter provided by the UK NEQUAS expert group).

Three methods are recommended for uncertainty in measurement determination (COFRAC, SH‐GTA 14, 2011). The Guide to the expression of Uncertainty in Measurement (GUM) method (ISO/IEC GUIDE 98, 1993) is a mathematical model created for the measurement process in which the standard uncertainty of each influencing factor has to be quantified. The advantage of this method is that it allows determining the uncertainty burden of each influencing factor. Its major drawback is that it is relatively tedious and not applicable in the routine practice of a medical laboratory.

In France, the Internal Quality Control and External Quality Assessment (IQC + EQA) is the most widespread method (Martinello et al., 2020). It is described in the technical accreditation guide for the evaluation of uncertainty in measurement published by COFRAC. In this method, IQC allows estimating the random error, and EQA establishes the bias and thus estimates the systematic error. The advantage of this method is its simplicity. The controls are clustered into levels, and the uncertainty values are calculated for each control level. However, this clustering can lead to inaccuracies (Bouveyron et al., 2019) in the establishment of the uncertainty in measurement.

Finally, the Long‐Term Uncertainty in Measurement (LTUM) method, described by Meijer et al. in 2002 (Meijer et al., 2002), is based only on EQA. In this method, a regression line is drawn between the laboratory values and the expected values (EQA). The random error is represented by the Long‐Term Coefficient of Variation (LTCV) that reflects the point dispersion around the regression line. The systematic error is represented by the long‐term bias (LTB). It reflects the deviation of the regression line compared with the identity line that can be drawn if each laboratory value corresponds to the expected value. The LTUM method has been implemented by Matar et al. (2015) to other disciplines, such as hormonology, immunology and hemostasis. In this method, quality controls do not need to be clustered by levels. However, it requires many values for robust evaluation. It has been estimated that with fewer than eight values, the calculated uncertainty is not relevant (Martinello et al., 2020).

Multiparameter flow cytometry (MFC) is a technique based on the measurement of the fluorescence intensity emitted by antibodies coupled to fluorochromes. MFC uses internal beads as standard to provide accurate absolute cell counting data. The quantification of the different lymphocyte subpopulations used for the diagnosis of immunodeficiencies is commonly done by MFC. In clinical practice, the level of CD3+/CD4+ T lymphocytes is used to measure disease progression in HIV‐positive patients (Noël et al., 2019; World Health Organization, 2016), and when this level is below a given threshold, treatment needs to be adjusted. MFC is also used to quantify the level of CD34+ stem cells in blood. This measurement is important in patients undergoing stem cell transplantation because it gives an estimation of the circulating stem cell level. If this level is higher than a pre‐defined cut‐off value, apheresis for stem cell grafting can be performed. Therefore, uncertainty in measurement assessment is essential in these two areas and is required to obtain reliable results.

The aim of our study was to compare the uncertainty in measurement values obtained using the IQC + EQA and LTUM methods, and to demonstrate LTUM applicability for a specialized technique, such as flow cytometry.

MATERIALS AND METHODS

Multiparameter flow cytometry (MFC) analysis

Samples were analyzed using two FACSCanto II cytometers (BD Biosciences®) that were standardized using fluorescent beads according to the Euroflow protocol (Kalina et al., 2012). The internal quality of the cytometer used on a given day was evaluated before starting the analyses. Two quality levels were used to frame the series of tests with normal and pathologic values. Streck provided the IQCs®. The EQAs were provided by UK Nequas® every 2 months (six times per year). The EQAs and IQCs were carried out using stabilized blood.

For this study, the following parameters were analyzed: CD34+ stem cells, and also the lymphocyte subpopulations characterized as CD19+ (B lymphocytes), CD3+/CD4+ (T helper lymphocytes) and CD3+/CD8+ (T cytotoxic lymphocytes), and CD3‐/CD16+/CD56+ (natural killer cells).

The number of cells in each subpopulation was quantified using reagents from two CE‐marked IVD kits: BD Multitest™ and BD™ Stem Cell Control Kit (both from BD Biosciences®). Each antibody was coupled to a specific fluorochrome: the CD3 antibody to FITC, the CD16 and CD56 antibodies to PE, the CD45 antibody to PerCP‐Cy5.5, the CD4 antibody to PE‐Cy7, the CD19 antibody to APC, the CD8 antibody to APC Cy‐7 (BD Multitest™), and the CD34 antibody to PE (BD™ Stem Cell Control Kit).

These kits allow determining the absolute value (cells/μl) of positive cells in the samples by comparing cell events and fluorescent bead events. Therefore, the absolute value of a cell population (A) can be obtained by dividing the number of positive cell events (X) by the number of bead events (Y), and then multiplying by the bead concentration after reconstitution in the BD™ Trucount tube (provided by BD Biosciences®) (N/V, where N is the number of beads per test and V is the test volume): A = X/Y × N/V.

The study period was from June 2017 to October 2018. The BD™ FACSCanto clinical software was used for the analysis.

Samples

Control samples

BD Multi‐Check Control® and BD Stem Cell Control® are stabilized blood controls with assigned values that can be used to monitor immunophenotyping.

Sample preparation

A single operator prepared all samples for quantification of the number of CD34+ stem cells and lymphocyte subpopulations. Briefly, 20 μl of BD™ Stem Cell Control reagent and 100 μl of BD™ stabilized blood Stem Cell Control CD34+ were added using an electronic pipette in a BD™ Trucount tube. After incubation at room temperature (20–25°C) protected from light for 20 min, 2 ml of BD™ CD34+ lysis buffer was added followed again by incubation at room temperature protected from light for 10 min. For lymphocyte quantification, 50 μl of stabilized whole blood was added to a BD™ Trucount tube (with an electronic pipette) followed by 20 μl of BD Multitest™ antibody suspension. Following incubation in the dark at room temperature for 15 min, 450 μl of BD FACS lysing solution was added to the tube. After 15 min of incubation, samples were analyzed.

Each sample was analyzed with the two cytometers consecutively.

Sample analysis

The flow cytometer IQC data were analyzed by laboratory technicians. They adjusted the gates and verified the acceptability of the obtained values according to the Levey‐Jennings rules (Levey & Jennings, 1950; Westgard et al., 1981).

Figure 1 shows the results of a typical analysis of the different lymphocyte subpopulations. After gating for lymphocytes on the basis of their low internal complexity (SSC) and high CD45 expression (Figure 1a), beads were isolated on the basis of their high side scatter and APC fluorescence (Figure 1b). T lymphocytes were identified on the basis of CD3 expression (CD3+) (Figure 1c). Then, in the T lymphocyte population, CD4 and CD8 expression allowed the identification of the different T cell subpopulations (Figure 1d). CD34+ stem cells were gated according to the ISHAGE protocol (Sutherland et al., 1996). Lymphocytes and beads were gated using the same method as for the lymphocyte subpopulations. For CD34+ stem cell quantification, 7AAD expression allowed excluding dead cells, and the bright CD34 expression was used to isolate CD34+ stem cells.

FIGURE 1

Dot plots of the lymphocyte subpopulations. (a) CD45 is a pan‐leucocyte marker; the side scatter provides information on the cell internal complexity. The use of CD45 and side scatter allows gating all lymphocytes in the sample (red‐colored cells). (b) The Trucount™ beads constitute the internal standard for the enumeration of the different subpopulations. They are gated based on their high APC fluorescence and high complexity (blue‐colored cells). (c) In the lymphocyte population, CD3 expression separates T cells (CD3+) (blue‐colored cells) from all other lymphocytes (pink‐colored cells). (d) Among T cells (CD3+), the expression of CD4 and CD8 allows distinguishing the different T cell subpopulations: CD3+/CD4+ (green‐colored cells) and CD3+/CD8+ (yellow‐colored cells) [Color figure can be viewed at wileyonlinelibrary.com]

IQC + EQA method

The random error was determined with the daily IQC, whereas the systematic error (i.e., the bias) was given by the EQA. Therefore, in this method, the uncertainty in measurement value was given by the following equation (COFRAC, SH‐GTA 14): where u(IQC) represents the inaccuracy and u(EQA) the bias.

LTUM method

This method is based on a linear regression (y = bx + a) in which the laboratory value (y) is expressed as a function of the target value, “b” represents the slope, and “a” the intercept. The total error can be divided into random error and systematic error.

LTB represents the systematic error. It includes the constant bias (CB) and the proportional bias (PB). CB is equal to the intercept (a). It can be attributed to a matrix effect. PB is represented, at each concentration, by the difference between each point of the regression line and the parallel line that can be drawn if the intercept (a) is added at each point of the identity line (line where all the laboratory values correspond to the target value and where y = x). The slope (b) of the regression line is an estimation of PB, mainly resulting from calibration errors. The random error corresponds to the dispersion of the data points around the regression line caused by the routine operating condition variability. It is quantified by the LTCV.

Therefore, in LTUM, uncertainty is given by the following equation: where LTCV(a) represents the random error and LTB (COFRAC, SH‐GTA 14) the bias.

RESULTS

The uncertainties in measurement values obtained with the two methods are summarized in Table 1. The major difference was that the IQC + EQA method calculated the absolute uncertainty (i.e., a constant value for each cluster level), whereas the LTUM calculated the relative uncertainty that can be adapted to any level of the measured parameter.

TABLE 1

Uncertainty in measurement calculated using the IQC + EQA and LTUM methods from June 2017 to October 2018

	IQC + EQA			LTUM
	Cytometer 1	Cytometer 2		Cytometer 1	Cytometer 2
CD3+ cells (/μl) level high	213.2	199.8	CD3+ cells %	12.9	12.4
CD3+ cells (/μl) level low	184.7	206.7
CD3+/CD8+ cells (/μl) level high	186.4	185	CD3+/CD8+ cells %	16.2	16.5
CD3+/CD8+ cells (/μl) level low	99.5	99.9
CD3+/CD4+ cells (/μl) level high	147.1	148	CD3+/CD4+ cells %	19.1	18.4
CD3+/CD4+ cells (/μl) level low	57.2	55.3
CD3neg/CD16+/CD56+ cells (/μl)	83.6	98	CD3neg/CD16+/CD56+ cells %	19.3	24
CD19+ cells (/μl)	83.8	86.4	CD19+ cells %	16.7	16.7
CD34+ stem cells (/μl) level high	10.67	11.06	CD34+ stem cells %	14.4	13.7
CD34+ stem cells (/μl) level low	4.55	5.64

To illustrate this difference, uncertainty in measurement was determined for 350 CD3+/CD4+ cells/μl, which is the therapeutic threshold for HIV treatment adjustment (Noël et al., 2019; World Health Organization, 2016). With the IQC + EQA method, the absolute uncertainty value was 57.2 CD3+/CD4+ cells/μl for the low level (<400 CD3+/CD4+ cells/μl). The range for a measured value of 350 CD3+/CD4+ cells/μl was [292.8; 407.2]. With the LTUM method, the uncertainty was 19.1% of the measured value. For the value of 350 CD3+/CD4+ cells/μl, this represented a range of [283.2; 416.9], indicating a larger range of uncertainty with LTUM.

Figure 2 presents the comparison of uncertainty in measurement values for each CD3+/CD4+ cell level obtained with the two methods. With the IQC + EQA method, uncertainties could be very different, particularly around the cut‐off value, with a risk of underestimation below and of overestimation above this value. In the example on the therapeutic threshold for HIV treatment adjustment, the cut‐off between high and low CD3+/CD4+ values was 400 CD3+/CD4+ cells/μl. This value was chosen because it is located between the two IQC mean values during the study period that were 694.87/μl [641.94; 747.8] for the high level and 150.7/μl [143.67; 157.73] for the low level. Although 350 CD3+/CD4+ cells/μl was among the highest values below this cut‐off, with the IQC + EQA method it showed the same uncertainty as very low values, for example, 20 CD3+/CD4+ cells/μl. This helps to understand why for the value of 350 CD3+/CD4+ cells/μl, uncertainty was higher with the LTUM than with the IQC + EQA method.

FIGURE 2

Uncertainty in measurement for each CD3+/CD4+ cell/μl value with the two methods (IQC + EQA and LTUM). (a) For the IQC + EQA method, 400 CD3+/CD4+ cells/μl represents the cut‐off value for clustering the uncertainties values. Inside each cluster level, uncertainties (in black) are constant regardless of the laboratory values (in blue). (b) For the LTUM method, uncertainties are adapted to each CD3+/CD4+ laboratory value [Color figure can be viewed at wileyonlinelibrary.com]

Analysis of the IQC and EQA data for the CD3+/CD4+ T lymphocyte quantification (Figure 3) obtained with one of the two standardized cytometers from June 1, 2017 to October 10, 2018 showed that the IQC values (colored dots) were very close to each other, although some inter‐batch variation was observed. Conversely, the EQA values (white squares in Figure 3) within each level were more dispersed than the IQC values. This higher dispersion better represented the values observed in the routine use of these cytometers with patients' samples.

FIGURE 3

IQC and EQA data for CD4+ cell populations (cells/μl) from June 2017 to October 2018. The EQA values are represented by white squares, the IQC values by colored dots. Each color represents a new batch [Color figure can be viewed at wileyonlinelibrary.com]

The estimation of the random error by EQA (LTCV) was higher than by IQC, probably due to the lower variation of the IQC values (Figure 3).

DISCUSSION

In this study, we compared the uncertainty in measurement values obtained with the IQC + EQA and LTUM methods for lymphocyte subpopulations and CD34+ stem cells. As results were comparable, we only chose to focus on CD3+/CD4+ T cells to avoid redundancy. The advantage of LTUM is that the bias variations in function of the concentration are taken into account, and therefore value clustering by levels is not required. Moreover, value clustering, as done in the IQC + EQA method, leads to some approximation of the uncertainty values. This approximation is greater if the clustered values are far from the IQC level cut‐off. For example, our study highlighted a large difference between IQC and EQA values. Specifically, for CD3+/CD4+ cell quantification, the two IQC levels (694.87 and 150.78 cells/μl) were often very different from the EQA values (Figure 3). Furthermore, although the IQC values were very reproducible, they did not correspond to the treatment decision‐making thresholds (for example, 350 CD3+/CD4+ cells/μl). Conversely, the EQA gave a large range of values that allowed better estimating the cytometer reliability relative to the physiological values managed by a routine laboratory. Indeed, it has been shown that in a single individual, many factors can affect CD3+/CD4+ cell levels, for example, circadian rhythm (Silva‐Sanchez & Randall, 2017) through transcriptional/translational feedback loops that involve different clock genes/proteins (Beam et al., 2020). For instance, lymphocyte and monocyte levels are at their maximum during the night.

The use of stabilized blood coupled to fluorescent antibodies for IQC determination should allow the verification of the analytical process and is required by the ISO15189 standard for laboratories. Chapter 5.6.1 specifies that: “The laboratory shall ensure the quality of examinations by performing them under defined conditions. Appropriate pre‐ and post‐examination processes shall be implemented (see 4.14.7, 5.4, 5.7, and 5.8). The laboratory shall not fabricate any results.” The main aims are to identify possible dysfunctions in the analytical process, to demonstrate the mastering of the analytical system, to prevent abnormalities and to obtain data for determining the uncertainty in measurement. Cytometer standardization using fluorescent beads according to the Euroflow protocol (Kalina et al., 2012) reaches some of these goals, particularly guaranteeing the cytometer calibration to control the analytical process. The two major drawbacks are that they do not allow determining the uncertainty in measurement and they do not have the same or similar composition as patients' samples. However, as the cytometer only detects fluorescence, the conditions are the same as for patients' sample analysis. These beads correspond to the IQC definition, if they are considered as “a set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released” (Thompson & Wood, 1995). Unlike stabilized blood, beads are not subject to inter‐batch variations, and this explain why the stabilized blood used for the IQC assessments leads to some approximation in the target value determination. Beads can guarantee cytometer performances better than stabilized blood and can be used alone as IQC.

Several studies demonstrated that LTUM is equivalent to the IQC + EQA method for uncertainty in measurement determination (Matar et al., 2015; Meijer et al., 2002). Our results suggest that LTUM is superior to IQC + EQA for different reasons. First, the IQC method uses stabilized blood that does not reflect exactly the patients' sample matrix. Furthermore, the process of IQC stabilization leads to some approximation in the target values that is materialized by inter‐batch variations (Figure 3). Second, the uncertainty in measurement values calculated by the IQC + EQA method were lower than those obtained with the LTUM method because the IQC values almost never vary, which can lead to an underestimation of the real‐life reproducibility. Third, the IQC method does not test the treatment decision‐making threshold values (e.g., 350 CD3+/CD4+ cells/μl). Therefore, it does not assess the reproducibility around these critical values. Finally, as the EQA tests a larger range of values, it offers a more representative panel of what routinely encountered in a laboratory. The evaluation of the coefficient of variation is more precise when the EQA is considered on its own.

Here, we demonstrated that the LTUM method could be used to determine the uncertainty in measurement for cytometry analyses. We chose to focus only on CD3+/CD4+ T cells to avoid redundancy, but we obtained similar results for the other lymphocyte subpopulations and for CD34+ stem cells (Table 1). We think that the LTUM method is useful for laboratories that analyze lymphocyte subpopulations and for laboratories specialized in the quantification of circulating stem cells in blood. Stabilized blood‐based IQC leads to an approximation of the uncertainty in measurement determination and does not provide reliable information on the analysis reproducibility. The periodic utilization of fluorescent beads can be considered as a good IQC to monitor cytometer performances.

CONFLICT OF INTEREST

The authors have no competing conflict of interest to declare.