The total deviation index estimated by tolerance intervals to evaluate the concordance of measurement devices.

BACKGROUND: In an agreement assay, it is of interest to evaluate the degree of agreement between the different methods (devices, instruments or observers) used to measure the same characteristic. We propose in this study a technical simplification for inference about the total deviation index (TDI) estimate to assess agreement between two devices of normally-distributed measurements and describe its utility to evaluate inter- and intra-rater agreement if more than one reading per subject is available for each device.
METHODS: We propose to estimate the TDI by constructing a probability interval of the difference in paired measurements between devices, and thereafter, we derive a tolerance interval (TI) procedure as a natural way to make inferences about probability limit estimates. We also describe how the proposed method can be used to compute bounds of the coverage probability.
RESULTS: The approach is illustrated in a real case example where the agreement between two instruments, a handle mercury sphygmomanometer device and an OMRON 711 automatic device, is assessed in a sample of 384 subjects where measures of systolic blood pressure were taken twice by each device. A simulation study procedure is implemented to evaluate and compare the accuracy of the approach to two already established methods, showing that the TI approximation produces accurate empirical confidence levels which are reasonably close to the nominal confidence level.
CONCLUSIONS: The method proposed is straightforward since the TDI estimate is derived directly from a probability interval of a normally-distributed variable in its original scale, without further transformations. Thereafter, a natural way of making inferences about this estimate is to derive the appropriate TI. Constructions of TI based on normal populations are implemented in most standard statistical packages, thus making it simpler for any practitioner to implement our proposal to assess agreement.

Background

In an agreement assay, it is of interest to evaluate the degree of agreement between different methods (devices, instruments or observers) used to measure the same characteristic. Thus, the closeness between the measures of the methods must be evaluated. Different procedures for assessing agreement with continuous measurements have been proposed and these can be classified under two terms [1]: (1) unscaled summary indices based on absolute differences; and (2) scaled summary indices which translate absolute differences into more meaningful values ranging between -1 (perfectly reversed agreement) and 1 (perfect agreement), where 0 indicates no agreement.

Scaled indices have probably been the most widely used, especially the intraclass correlation coefficient [2-4] (ICC) and the concordance correlation coefficient [5] (CCC). Both ICC and CCC indices have recently been evaluated and compared in many studies [6-8], and have been shown to provide two different expressions of one common index. However, when conducting an agreement analysis it should be remembered that these scaled indices depend on the covariance between measurement devices [9], as the resulting estimates can vary depending on the possible range of values of the measurement instrument under consideration. Another consequence of this covariance dependency is that the indices might be overestimated if potential confounding variables are not taken into account [8].

Among unscaled procedures, the total deviation index (TDI) describes a boundary such that a majority percent of the differences in paired measurements are within the boundary [10,11], i.e. a probability interval. The advantage of the TDI against scaled measures such as CCC is that it does not depend on the data range and therefore it avoids the inconvenience of not taking into account potential covariates that explain between-subject variation. However it must be noted that as in the CCC case the TDI will depend on covariates explaining within-subject variation. A further advantage is that it has a straightforward interpretation since it results in the same measurement scale as that of the variable considered for agreement purposes.

Several methods for inference about the TDI estimate have been proposed. To calculate the index Lin [10] derived the cumulative probability function of the square of the paired-measures difference variable, which is assumed to follow a non-central chi-squared distribution. He argues that inference about the estimate of the resulting equation is cumbersome, and he thus derives a further approximation with more desirable properties based on the asymptotic theory of the mean squared deviation (MSD) [10]. Lin et al. [12] extended the method to deal with repeated measures. Due to the positive skewness of the resulting TDI estimates, when performing inferences the natural log transformation of the estimate is used. This approximation has been shown to conclude satisfactory agreement when mean differences between two measurement devices are small, but it can be conservative when the relative bias square value is unreasonably large and when the coverage probability is large (0.95). Choudhary and Nagaraja [13] proposed an upper bound for the estimate of Lin's resulting TDI equation for the case of no repeated measures derived from an exact test. As the exact test method needs to maximise an integrated equation with no closed form, numerical computations are required to implement it; as the authors acknowledge, these may not be readily available in practice, so they also propose a closed-form approximation.

Choudhary [14] subsequently extended the method based on the asymptotic distribution of the logarithm of Lin's TDI proposal to deal with repeated measures. He argues that this method performs well with large sample sizes and proposes a modified version for smaller sample sizes based on a bootstrap approach. Recently, Quiroz and Burdick [15] also derived a method for inference about the TDI estimate when dealing with repeated measures for the two methods that are paired over time, and fit the data using an ANOVA model. They then construct generalised confidence intervals about the TDI estimate that are based on replacing parameters involved in Lin's [10] TDI expression with generalised pivotal quantities. The generalised confidence intervals are constructed via Monte Carlo simulations and have been shown to perform well in a wide range of scenarios, including those with either small, moderate or large sample sizes. Here we propose a technical simplification for inference about the TDI estimate based on a closed approach. We first estimate the TDI by finding the appropriate probability interval of the distribution of the paired-measures difference variable. Therefore, a natural way of making inferences about this TDI estimate is to derive its tolerance interval (TI). This procedure offers a straightforward approach as the theory and methods about TI for normal populations are well established [16-18].

The article is structured as follows: in the methods section the TDI is defined and Lin's [10] first approach is described. A brief description of two current closed approaches for inference about the TDI estimate is subsequently given. Thereafter, a probability interval approach is defined to obtain an alternative expression of the TDI estimate. This approach is also used to derive estimates of the inter- and intra-method [12,19] measures of agreement when more than one reading per subject is available. Based on the probability interval approach a direct inference method about this estimate is derived via the TI. Lastly, in this section we also describe how one may utilize the TI approach to perform inference for the computation of the coverage probability, an agreement measure related to the computation of the TDI. In the results section we illustrate the methodology by using it to evaluate agreement between a manual and an automatic blood pressure device. In this example we point out the independence of the TI method from the effect of the between-subject variation, as compared with other scaled methods such as the CCC, whose covariate adjustment that explains between-subject variation modifies the resulting agreement value. We will also describe and report our simulation study procedure for evaluating the performance of the method and compare it to already established methods. A discussion and concluding remarks are given at the end of the manuscript.

Methods

Definition

Suppose a continuous variable is measured by two different devices m times each from n different subjects. Therefore, the data can be fit using the following mixed model [12,14]:

where yis the lth measurement from subject i by device j, with i = 1, ..., n, j = 1, 2, and l = 1, ..., m. δ is the vector of fixed effects parameters common to both devices and xis the corresponding row of the design matrix for covariates, βis the fixed device effect, αis the individual random effect assuming that α~ N(0, ), γis the individual-method interaction random effect with γ~ N(0, ) and eis a random error assuming that e~ N(0, ) and is independent of any other component of the mixed model. If the error variability differs across devices, then e~ N(0, ).

Lin [10] defined the TDI as a boundary, κ, which captures a large proportion, p, of paired-measurement differences from two devices or observers within the boundary, i.e., the value of κthat yields P(|D| <κ) = p, where D is the paired-differences variate. Under the assumption of the mixed model in (1), D is the paired-differences variate based on any one of the replicates, D = (y- y), and hence κbased on D is actually known as the total-TDI for evaluating total agreement [12]. It is shown that the distribution of D is then D ~ N(μ, σ) with μ= β- βand , or in the case of different error variances between devices .

When more than one reading per subject given by device j is available, one might be interested in measuring, in addition to the total agreement, the inter- and intra-method agreement [12,19]. Intra-method indices are used to measure the agreement among the multiple readings obtained from the same device [12]. This agreement measure is useful when ones wishes to evaluate the reproducibility or repeatability of a specific device. To evaluate intra-method agreement, differences between replications from the same individual given by the j - th device are used and, therefore, under the assumption of the mixed model in (1): (y- y) ~ N(0, ) with = 2. Inter-method agreement is used to measure the agreement among different devices based on the average of their multiple readings [12]. If we denote , under the assumption of the mixed model in (1), the inter-method agreement can be evaluated by the following distribution: (y- y) ~ N(μ, ), where μ= β- βand , or in the case of different error variances between devices, .

The first formal definition [10] of the TDI for the case of one single reading from each device for each subject, i.e when m = 1, was based on the distributional assumption of the square transformation of the paired-measurement difference variable:

where F-1 is the inverse of the cumulative probability function of |D|, and χ2(-1)(·) is the p-th percentile of a non-central chi-square distribution with 1 degree of freedom and non-centrality parameter . Even though equation (2) was defined for the case of one single reading, one can apply the mixed model (1) to accommodate replicated readings from each device for each subject and use the model parameter estimates to obtain estimates of μand and, furthermore, compute the TDI estimate by plugging in these estimates [14]. However, and as Lin [10] argued, inference about this κestimate is cumbersome, and he therefore proposed a further approximation based on the mean squared deviation (MSD):

where ε2 = E(y- y)2, z(1 + is the (1 + p)/2 - th percentile of the standard normal distribution and |·| is the absolute value.

Current approaches for inference about the TDI estimate

There are two already existing closed procedures for inference about the TDI estimate that consider repeated measures taken by each of the two devices with multiple readings being compared. The first approach was defined by Lin et al. [12] where the authors expressed the TDI approximation based on the MSD, as in (3), which under the assumption of the mixed model in (1) the MSD becomes , and therefore , where = (β- β)2/2 is defined as the variance between the two devices. Furthermore, the generalized estimating equations (GEE) approach [20] is used to obtain the model parameter estimates in (1). Since this TDI estimate is positively skewed [11,12] the authors use the log transformation to form inference and the delta method is applied to obtain the variance of the resulting TDI estimate.

The second approach was defined by Choudhary [14] where the author proposes to use the maximum likelihood estimation (MLE) procedure to obtain model parameter estimates in (1) and, furthermore, compute the TDI estimate by simply plugging the MLE estimates of μand in (2). The author argues that the distribution of this MLE estimate of the TDI approach normality more quickly on the log scale, especially when the sample size is small. Based on this assumption the delta method is used to obtain the variance of the log-transformed TDI estimate.

Both approaches for inference about the TDI estimate are based on the delta method, which means that one should first find the partial derivatives of the log transformed TDI with respect to the model parameters used to obtain the expression of the TDI and then find the inverse of the information matrix for the fitted model.

TDI as a probability interval

Consider the TDI definition, which sets a boundary such that a majority p percent of the differences in paired observations are within the boundary: P(|D| <κ) = p. This is the same as finding κ, such that P(-κ

where , s = 1, 2, are the p-th percentile of the standard normal distribution, such that = -2μ/σ- and p1 - p2 = p. Therefore, one can find p1 by using its link with p:

where Φ(·) is the cumulative standard normal distribution.

However, p1 cannot be found in a closed form using equation (5), so a recursive algorithm is required. We propose to use a modified version of the binary search algorithm [21] to find p1 and, furthermore, to compute κusing (4).

Typically, the binary search algorithm is used to search in an ordered array for a single element by repeatedly dividing the array in half. Here we translate the ordered array into the interval [low; high], which means that low(high) is the lowest(highest) value that p1 can take. Now, since equation (5) has a single solution for p1 in the interval [p; 1], one can repeatedly halve the interval in an adequate manner to find the optimum for p1. Therefore, the algorithm is implemented as follows:

1. begin with the interval [low = p; high = 1];

2. calculate the midpoint of the interval mid = (low + high)/2;

3. if the left-hand side of equation (5) for p= low is greater than p up to a tolerance bound δ (i.e., ), then recalculate the interval [low = mid + δ; high = 1]; if it is lower than p up to a tolerance bound δ (i.e. ), then recalculate the interval [low = p; high = mid - δ];

4. repeat steps 2-3 until convergence, i.e. until the solution for p1 in (5) is p - δ < Φ(z) - Φ(-2μ/σ- )