The significance of reporting to the thousandths place: Figuring out the laboratory limitations.

OBJECTIVES: A request to report laboratory values to a specific number of decimal places represents a delicate balance between clinical interpretation of a true analytical change versus laboratory understanding of analytical imprecision and significant figures. Prostate specific antigen (PSA) was used as an example to determine if an immunoassay routinely reported to the hundredths decimal place based on significant figure assessment in our laboratory was capable of providing analytically meaningful results when reported to the thousandths places when requested by clinicians. DESIGN AND METHODS: Results of imprecision studies of a representative PSA assay (Roche MODULAR E170) employing two methods of statistical analysis are reported. Sample pools were generated with target values of 0.01 and 0.20 μg/L PSA as determined by the E170. Intra-assay imprecision studies were conducted and the resultant data were analyzed using two independent statistical methods to evaluate reporting limits.
RESULTS: These statistical methods indicated reporting results to the thousandths place at the two assessed concentrations was an appropriate reflection of the measurement imprecision for the representative assay. This approach used two independent statistical tests to determine the ability of an analytical system to support a desired reporting level. Importantly, data were generated during a routine intra-assay imprecision study, thus this approach does not require extra data collection by the laboratory.
CONCLUSIONS: Independent statistical analysis must be used to determine appropriate significant figure limitations for clinically relevant analytes. Establishing these limits is the responsibility of the laboratory and should be determined prior to providing clinical results.

Introduction

The discussion of significant figures in result reporting is given relatively little formal attention in the field of laboratory medicine. While a few well-written discussions can be found in the literature [1], [2], [3], it is clear that available guidelines or requirements are not always practiced or well known. Further complicating the topic is the futility of a discussion about significant figures when laboratory information systems are only capable of reporting in reference to a decimal place. The available literature provides several useful mechanisms for establishing significant figures for the reporting of a given assay. However, less guidance is provided on determining how to establish reporting limits in situations where decimal place consistency is more important than significant figures. For example, prostate specific antigen (PSA) may be measured by methods referred to as “ultrasensitive” with performance claims allowing for the detection of PSA below 0.10 μg/L (0.10 ng/mL). The complication comes from a claimed sensitivity of 0.010 μg/L; an indication of reporting to the hundredths decimal place, but suggestion of a potentially clinically meaningful digit in the thousandths decimal place. Further, strict adherence to the two significant figures claim would allow reporting of 0.011, 0.015 and 0.019 μg/L but not 0.111, 0.115 and 0.119 μg/L. The latter set would require reporting as 0.11, 0.12, 0.12 μg/L causing a perceived loss of resolution between results.

PSA plays a prominent role in the early detection, management, and staging of prostate cancer [4], [5].”Ultrasensitive” PSA assays may be used by some clinicians to detect residual or recurrent disease in patients post-prostatectomy [6], [7]. Some manufacturers offer assays which claim to have a functional sensitivity (coefficient of variation ≤20%) as low as 0.010 μg/L [8], [9], [10] or results that can be reported to two significant figures using the thousandths place. Despite the many advances in the sensitivity and precision of the PSA assay, laboratories commonly report results in this range to two decimal places since the precision of these assays has not been well studied at these low concentrations. However, it has been brought to our attention that values reported to the thousands place are believed by some to aid in patient surveillance.

The goal of this study was to investigate the analytical validity of reporting to the thousandths place regardless of significant figure protocol for PSA at concentrations typically measured with an “ultrasensitive” method. Here we report the results of imprecision studies of a PSA assay with a reported functional sensitivity of 0.030 ng/mL (Roche MODULAR E170, Indianapolis, IN) employing two unique methods of statistical analysis. While straightforward in the approach, the laboratory's assessment of the precision of high sensitivity assays may have considerable clinical implications.

Materials and methods

Patient samples

Residual serum samples submitted to ARUP Laboratories for PSA testing were de-identified stored frozen (-20C) for 10–14 days prior to analysis. This project and its protocols were approved by the University of Utah Institutional Review Board (IRB protocol #00007275).

Data collection

Data was obtained by analyzing twenty replicates each of a low value (target 0.01 μg/L) and a high value (target 0.20 μg/L) PSA sample pool. Selected sample values were chosen to represent clinically relevant PSA concentrations [11], [12] and were within the analytical measurement range of the Roche MODULAR E170 PSA assay 0.014–100 μg/L). After preparation, sample pools were assayed to obtain an initial value and appropriately adjusted using a high value sample or a low value sample until the desired target values were obtained.

Aliquots were tested using the Roche MODULAR E170 automated chemistry analyzer. In order to reduce imprecision, all replicates were performed simultaneously and one measuring cell was inactivated to eliminate any cell-to-cell variation. Testing was performed according to manufacturer's guidelines and using Roche proprietary reagent for the total PSA assay (Catalog #04491734).

Statistical methods

Two methods were used to evaluate statistical precision and, thereby, assess the appropriateness of reporting. Method I is recommended by the National Resources Management and Environment Department [13] that involves using the within-run variation to direct significant figure reporting. Method II uses a χ2 test to compare performance claim standard deviation (σ) to observed standard deviation (S). Both of these methods are described in detail below.

Method I

The Natural Resources Management and Environment Department guideline recommends the following procedure that uses within-run variation to direct reporting of significant figures and determination of rounding rules (n ≥20):

Calculate the upper boundary b of the rounding interval a using the standard deviation (s) of the unrounded results, by letting: b=s/2. Then choose a equal to the largest decimal unit (e.g., 0.1, 0.01, 0.001 etc.) which does not exceed the calculated b.

Method II

The Clinical Laboratory Standards Institute global consensus guideline [14] uses a χ2 test to compare performance claim standard deviation (σ) to observed standard deviation (s), where s2 is the sample variance, σ2 is the claimed variance, and R is the total number of determinations or measurements: χ2 =(s2·R) /σ2. The calculated χ2 result is then compared to an upper 95% critical value for R degrees of freedom and can be treated as a formal hypothesis test of the claimed variance.

Data analysis

Data analysis was conducted using Excel (Microsoft Corporation, Bellevue, WA) and the open-source statistical language R (R Working Group, Vienna, Austria) [15]. Graphs were generated using Adobe Illustrator (Adobe Systems Incorporated, San Jose, CA). Kolmogorov-Smirnov significance testing was performed in R.

Results

Method I: rounding of test results

Table 1 contains the data from twenty replicates of the 0.01 μg/L and 0.20 μg/L PSA sample pools and calculated statistics for standard deviation (s), upper boundary (bt), and the rounding interval (a). Using this method and these data, the results indicated acceptability (aPSA pool and to the thousandths place with (0.001) for the 0.20 μg/L PSA pool.

Table 1

Raw PSA data used to determine significant figure reporting limits.

Replicate	0.010 μg/L target value	0.200 μg/L target value
	Observed value (μg/L)	Observed value (μg/L)
1	0.011	0.198
2	0.012	0.196
3	0.012	0.194
4	0.011	0.200
5	0.012	0.196
6	0.010	0.198
7	0.009	0.198
8	0.011	0.202
9	0.011	0.200
10	0.010	0.201
11	0.011	0.201
12	0.010	0.200
13	0.010	0.199
14	0.012	0.203
15	0.011	0.197
16	0.013	0.201
17	0.012	0.202
18	0.012	0.198
19	0.010	0.200
20	0.012	0.201
	s= 0.001020836	s= 0.002336777
Method I	bt= 0.0005104178	bt= 0.001168388
	a= 0.0001	a= 0.001
Method II	χ2= 5.21	χ2= 27.3

s, standard deviation; b, upper boundary; a, rounding interval.

Method II: testing (implied) performance claims

Using a manufacturer claim for precision of 0.001 and an implied standard deviation (σ) of 0.002, we compared the calculated χ2 result to the upper 95% critical value for R degrees of freedom. For our data, R=20 repeated measurements so the 95% cutoff value for χ2=31.41. Table 1 contains data from twenty replicates of the 0.01 μg/L and the 0.20 μg/L PSA sample pools and the calculated χ. Using a 2-sided test, the 0.01 μg/L PSA pool had a statistically significantly smaller variance than that which is implied by the precision, whereas the 0.20 μg/L PSA pool was within specified tolerance limits. This indicated reporting three decimal places was an appropriate reflection of the precision for both the 0.01 μg/L and the 0.20 μg/L PSA sample pools (Table 1).

Distribution of raw and rounded results

The distributions of 518 de-identified patient results using raw values (three decimal places) and rounded values (two decimal places) are presented in Fig. 1 using fixed bin widths of 0.01 and 0.1 µg/L for values less than 0.1 and 1 μg/L, respectively. A two sample Kolmogorov-Smirnov test for distribution equality produced a p-value of 0.6869 indicating the overall distributions were not affected when using raw values or rounded values.

Fig. 1

Distributions of raw values and rounded values from 518 patient results. Fixed bin widths of 0.01 μg/L and 0.1 μg/L were used for values less than 0.1 μg/L and 1 μg/L, respectively. All results greater than 1 were placed into a single “More” bin to highlight the lower end of the range. Black bars represent raw values (3 decimal places), gray bars represent the same values after rounding (2 decimal places).

Use of a third decimal place had a potentially significant effect for 46 patient results surrounding the 0.01 μg/L concentration associated with an increased probability of biochemical recurrence [12]. Using three decimal places, 23 patient results were <0.01 μg/L 0.006–0.009 μg/L) while a separate 23 patient results were >0.01 μg/L 0.011–0.014 μg/L). In contrast, rounding these results to 2 decimal places resulted in all 46 being reported as 0.01 μg/L.

Discussion

New or revised recommendations for treatment or testing may be initiated by physicians pushing the boundaries of what is known regarding test interpretation. Critical to the success of this paradigm is the general assumption that, in the case of significant figures and result rounding, the laboratory has identified the method limitations prior to providing any requested information. Responsibility in test result reporting rests with the clinical laboratory. In the present study, we describe the ability to accurately report results to a desired number of decimal places rather than adhering to strict significant figure reporting. We illustrated this using the PSA assay and verified reporting to three decimal places when adherence to strict significant figure reporting presents a challenge. The results of two separate statistical tests indicated that the assay tested was capable of generating results with statistically acceptable precision to three decimal places. Furthermore, as demonstrated by Fig. 1, the overall distribution of the data was unchanged with the use of either 2 or 3 decimal places.

It is important to recognize that the conclusions drawn regarding appropriate reporting of decimal places applies only to the PSA assay described here (Roche E170). Data gathered from other instruments and assays may differ.

Other approaches towards the determination of significant digits in the clinical laboratory exist and have been reviewed previously [2]. The approach presented here uses two independent statistical tests to determine the ability of the analytical system to support a desired precision target. Importantly, the data used were generated during a routine intra-assay imprecision study and assessment of reporting would therefore not require any extra data collection by the laboratory. Further, using saved data containing excess significant digits, a laboratory can retrospectively assess precision in the manner described here on demand. Alternatively, between-run imprecision data could also be assessed.

Although the current report uses PSA as an example, the data presented serves as an important reminder that the laboratory must fully understand the characteristics of testing methods to ensure integrity of all results.

Conflict of interest

None.