Unified Least Squares Methods for the Evaluation of Diagnostic Tests With the Gold Standard.

The article proposes a unified least squares method to estimate the receiver operating characteristic (ROC) parameters for continuous and ordinal diagnostic tests, such as cancer biomarkers. The method is based on a linear model framework using the empirically estimated sensitivities and specificities as input "data." It gives consistent estimates for regression and accuracy parameters when the underlying continuous test results are normally distributed after some monotonic transformation. The key difference between the proposed method and the method of Tang and Zhou lies in the response variable. The response variable in the latter is transformed empirical ROC curves at different thresholds. It takes on many values for continuous test results, but few values for ordinal test results. The limited number of values for the response variable makes it impractical for ordinal data. However, the response variable in the proposed method takes on many more distinct values so that the method yields valid estimates for ordinal data. Extensive simulation studies are conducted to investigate and compare the finite sample performance of the proposed method with an existing method, and the method is then used to analyze 2 real cancer diagnostic example as an illustration.

Introduction

In diagnostic test development, one is concerned about whether a newly developed test is more accurate than traditional ones to correctly discriminate a subject with a certain condition (“the case”) from a subject without the condition (“the control”).[1] Early diagnosis of serious diseases plays an important role because late detection can have serious consequences. For example, a patient with lung cancer might have a higher chance of surviving if detected early and the lesion is surgically removed. But the person will die if the diagnosis is incorrect and necessary surgery is not performed.[2]

For diagnostic tests that generate binary results, their accuracy can be summarized in terms of the sensitivity (ie, probability of identifying a case when the subject truly has the condition) and specificity (ie, probability of correctly identifying a control when the subject does not have the condition). The sensitivity is also called as the true-positive rate (TPR), and the false-positive rate (FPR) is 1 − specificity. For tests that generate continuous or ordinal results, the receiver operating characteristic (ROC) curve is a standard statistical tool to describe and compare the accuracy of diagnostic tests.[1] The ROC curve, commonly used in medical diagnostic studies, is a plot of TPR versus FPR at different possible thresholds. It is widely used in radiology, psychophysical, and medical imaging research for detection performance, military monitoring, and industrial quality control. It is used to examine the trade-off between the TPR and FPR under different thresholds and overcomes the limitation of having to dichotomize the test results to use isolated measurements of TPR and FPR. The ROC curve is plotted by connecting all the points generated by possible thresholds.[1] A test with 100% TPR and 0% FPR is a perfect predictor, ie, all the case patients have positive test results and all the control patients have negative test results.

Most ROC curves are concave and above the chance diagonal which is the line segment between (0,0) and (1,1). However, some of them are below the chance diagonal and are called improper curves.[3] The closer the curve is to the upper left corner, the larger the area under ROC curve (AUC) is and the better distinguishing ability the diagnostic test has. The perfect test has an AUC of 1. Figure 1 provides the illustration of the ROC curves for 3 biomarkers with different diagnostic accuracies. The ROC curve for biomarker 1 is uniformly above the other 2 ROC curves. This means that biomarker 1 has the best performance in detecting the case and control among the 3 biomarkers.

Figure 1.

ROC curves for 3 biomarkers: dotted curve—biomarker 1 (AUC = 0.9), dashed curve—biomarker 2 (AUC = 0.7), and solid curve—biomarker 3 (AUC = 0.5). AUC indicates area under ROC curve; FPR, false-positive rate; ROC, receiver operating characteristic; TPR, true-positive rate.

The ROC analysis of continuous data from a single test has been extensively investigated since the seminal work by Dorfman and Alf.[4] Diagnostic test studies generate correlated results when the same subject undergoes 2 or more different tests.[5] An important area in ROC research with multiple markers is the comparison of tests’ accuracy. Parametric and semiparametric methods have been proposed to estimate ROC curves from this type of correlated data in the literature. Parametric methods assume distributions for measurements,[1] but these methods may not perform well if the parametric assumptions are invalid. An intuitive parametric least squares (LS) ROC method proposed by Zhang and Pepe[6] requires no iteration and thus takes much less computation time than the ROC methods using iterations. The asymptotic covariance of their LS estimator is derived by Tang and Zhou.[7] An essential assumption of the procedure by Zhang and Pepe[6] is that the basis function of the ROC curve is known. A recent paper by Tang and Zhou[8] relaxes this assumption and estimates the basis function nonparametrically.

Besides continuous test data, ordinal data occur frequently in radiology when radiologists or computer algorithms are used to read subjects’ medical images and provide ordinal ratings regarding their belief in the severity of subjects’ disease status. Several methods for estimating a single ROC curve from ordinal data have been proposed by various authors.[4,9] Morris et al[10] provide a detailed summary of these methods for ordinal data. The maximum likelihood estimation (MLE) method by Dorfman and Alf[4] is the most widely used procedure for ordinal data. Metz and colleagues[11,12] consider 2 modalities. Hsieh and Turnbull[9] develop a generalized LS approach. As the number of markers becomes larger than 2, the MLE method by Metz et al[11] becomes inapplicable. It is also not trivial to extend the single ROC method by Hsieh and Turnbull[9] to multiple binormal ROC curves because the correlation structure among empirical ROC curves is unknown.

In this article, we propose a unified linear regression method to estimate the ROC curve from pairs of consistent sensitivity and specificity estimates. The proposed method estimates a pair of sensitivity and specificity for a given cutoff point. For a set of chosen cutoff points on the continuous data, a number of pairs can be obtained, and the estimates in the pairs can be values for the response variable and covariate in the linear regression setting. The method provides valid ROC parameter estimates for both continuous data and ordinal data.

Notations and Methods

Suppose that multiple tests are applied to a case sample with subjects and a control sample with subjects. For test , the test result for the case subject, , and test result for the control subject, , are available, where and for . At some given thresholds , let and be the sensitivity and specificity of the test, respectively. The observed results and may be continuous or ordinal. In the latter case, they are derived from some underlying variables and . At a threshold , the TPR or the sensitivity is given by and the specificity or (1 − FPR) is given by , where is the indicator for disease status with 1 being a case and 0 being a control. For the continuous diagnostic tests, the observed test results are identical to the underlying results.

For the ordinal diagnostic test, the observed ordinal ratings and are considered to be obtained by applying decision thresholds to the latent variables. For the ordinal test, and take on ordinal ratings, . These ratings are considered to be obtained by applying decision thresholds, , to the latent variables. Specifically, the rating is given to a case subject (or a control subject) if (or ) falls between and ().

The Hsieh method for one binormal ROC curve

We first consider one test with ordinal test results and for cancer and control subjects, respectively. Because the ROC curve is invariant to any monotonic transformation of the underlying test results, and can be considered to have already been transformed by some unknown monotone function so that and . Let be the probability of having the rating for the case subject, and let be the probability of having the rating for the control subject. The sensitivity and specificity at a threshold are and , respectively. From this, we may write the probabilities and as and , where and . The log-likelihood function is given as follows:

where and are the observed numbers of responses in the category from the cancer and control subjects, respectively. Dorfman and Alf[4] solve the score equation of the log-likelihood function and obtain the MLE estimators of , , and .

Hsieh and Turnbull[9] developed a generalized LS approach. They estimated the empirical ROC curve at a fixed number of FPRs and applied the generalized LS method to the transformed empirical ROC curve to obtain the parameter estimators. The regression method by Hsieh and Turnbull[9] for estimating 1 ROC curve is similar to the method of Dorfman and Alf[4]. The essential difference between them is that the former only requires the estimated sensitivities and specificities, whereas the latter requires the actual observations. For the result , we have and for . Hsieh and Turnbull[9] observed that

for. The equations above can be written as follows:

for . Thus, by assuming a perfect gold standard, the authors use and to estimate and , respectively, and obtained the following linear regression model with error terms:

for , where and are mean 0 random error vectors. These random vectors are independent, but the error terms within each vector are correlated. Based on the regression model, Hsieh and Turnbull[9] propose to obtain a generalized LS estimator for and .

The proposed method for multiple binormal ROC curves

The least squares method of Hsieh and Turnbull[9] only deals with 1 diagnostic test. It is possible to extend it to allow multiple diagnostic tests. Our extension still builds on the intrinsic property of the ROC curve that the ROC curve is invariant to any monotonic transformation of the test results. We assume that after some unknown transformation, the latent test results follow normal distributions for the case and control subjects. Suppose that for test 1, after some monotone transformation, in the cancer group and in the control group:

The equations above lead to . Let and . Denote and . We have the following equation:

The resulting ROC curve for test 1 can be written as . Consider the test , for . Suppose that after some monotone transformation, and . Note that the transformation may vary among modalities, but it should be the same for the cancer and control subjects for the same modality. For test , the following equations give the relationship between the rating categories and normal distribution parameters for the more general setting in which the nondiseased population can take on any normal distribution:

Let and . The relationship between the sensitivities and specificities at varying cutoff points for multiple tests can be expressed as follows:

Here, the ROC curve is given by .

Let be the FPR, which is 1 − specificity. The empirical functions of and are defined by DeLong et al[5] and Tang and Zhou[7] as follows:

We will substitute the estimated proportions in the above model. The regression equations with error terms can be written as follows:

Our parametric procedure is based on the model (equation (7)). We outline our parametric LS procedure as follows:

Step 1. Obtain the empirical functions and at the threshold , for and .

Step 2. Transform the sample proportions, , by , and define the following vectors:

Step 3. Combine to get the following linear regression equation:

where , is a vector, and the design matrix is as follows:

with its submatrices

Also, the error vector is given by .

Step 4. Replace using the in to obtain .

Step 5: Obtain the final ordinal LS estimator

The method by Tang and Zhou[7] also first creates the response variable based on the test results and constructs a linear model so that the parameters are estimated using the LS method. However, the key difference between the proposed method and the LS method in a study by Tang and Zhou[7] lies in the response variable. The response variable in the latter is transformed empirical ROC curves at different thresholds. It takes on many values for continuous test results, but few values for ordinal test results. The limited number of values for the response variable makes it impractical to use the method by Tang and Zhou[7] for ordinal data. However, the response variable in the proposed method takes on values for the specificities and sensitivities. The number of values is at least 2 times the number of distinct ordinal test results.

Asymptotic properties of the parameter estimates

We study the asymptotic properties of the LS parameter estimates in the context of 2 tests for simplicity. Denote , , and with components given by expression (6). Denote the true value of generating the observed data, and , with each being the empirical estimators of at the threshold . The formula is shown in equation (6). Let

Denote , E(Φ−1(Spl))2 = , lim R−1l , and :

and with . Let Ω be the matrix:

Recall and .

Denote for convergence almost surely and for convergence in distribution. The following condition will be used:

(C1). Ω is positive definite.

For fixed , let . Then, as ,

Assume (C1) and , for all , with L fixed. Let first and then , then,

Theorem 1

Simulation Studies

Estimates from normal test results

We conduct simulation studies to investigate the finite sample performance of the proposed method. Two simulation settings are used to simulate data sets. The first setting is under the normal distributions for the cancer and control populations, and the second is under the lognormal distributions. Given the same parameter values, the true ROC curve should be the same for normal or lognormal data sets due to the monotonic invariant property of the ROC curve. Because the proposed method only deals with the sensitivities and specificities, the estimated ROC curve should be valid for both distributions given the correctly specified link and baseline functions.

Under the normal setting, we simulate normal observations for 2 diagnostic tests. The bivariate normal data were simulated as outcomes from paired tests. Assume that the bivariate normal models had the forms and , where

with denoting the correlation in bivariate outcomes.

We simulate 1000 replications with all combinations of and under . For each replication, the threshold points are chosen to be normal quantiles of 100 equally spaced points ranging from 0.001 to 0.999. The threshold points are used to dichotomize the continuous observations. The dichotomized data are used to obtain empirical sensitivities and specificities which are the proportions of the test results greater than the threshold point for the cancer group and the proportions of the test results less than the threshold points for the noncancer group, respectively. Model (7) is then fit to the estimated sensitivities and specificities to obtain the estimates for , , , and . The LS method by Tang and Zhou[7] is also fit to the simulated data sets for comparison with the proposed method. The difference in the Tang and Zhou (TZ) LS method is that it estimates the empirical ROC curve at 100 equally spaced points ranging from 0.001 to 0.999. The transformed empirical ROC curves at these points are the observations for the response variable in the linear model. Table 1 presents the biases and root-mean-square errors (RMSEs) of these ROC parameter estimates by the proposed method and the TZ method. The biases are generally small for the proposed method. As the sample sizes become larger, the biases do not change much. The RMSEs tend to become smaller when both sample sizes for the cancer and noncancer groups become larger. We can also see that the correlation between the 2 tests does not affect the biases and RMSEs. Table 2 presents the biases and square RMSEs of these ROC parameter estimates by the TZ method. The biases and RMSEs are close to those by the proposed method.

Table 1.

Biases (in %) and RMSEs for normal data—proposed method.

		m = 40						m = 100						m = 200
ρ		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300
		Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE
0.10	α 1	−0.78	0.22	−1.67	0.22	−1.61	0.21	−2.20	0.15	−1.33	0.13	−0.98	0.13	−0.57	0.12	−0.95	0.10	−1.52	0.10
	β 1	−1.17	0.11	0.03	0.10	−0.70	0.09	−1.77	0.09	−1.20	0.07	−0.58	0.06	−1.48	0.07	−1.00	0.06	−1.11	0.05
	α_2	−2.59	0.33	−0.59	0.32	−2.90	0.31	−0.50	0.22	−0.50	0.19	−1.61	0.20	−1.27	0.16	−0.65	0.15	−0.27	0.14
	β_2	0.29	0.16	−0.22	0.13	0.58	0.13	0.23	0.12	0.43	0.09	−0.55	0.09	−0.24	0.10	−0.45	0.08	−0.12	0.07
0.20	α 1	−1.95	0.22	−0.94	0.20	−1.38	0.21	−0.77	0.15	−0.59	0.13	−1.39	0.13	−0.82	0.12	−0.82	0.10	−1.04	0.10
	β_1	−1.51	0.12	−0.65	0.10	−0.59	0.09	−1.78	0.08	−1.41	0.07	−0.88	0.06	−1.82	0.08	−1.42	0.06	−0.89	0.05
	α_2	−2.55	0.34	−2.45	0.33	−1.36	0.32	−3.04	0.23	−1.82	0.19	−0.74	0.19	−1.26	0.16	−0.56	0.14	−0.64	0.13
	β_2	0.71	0.17	0.03	0.14	0.67	0.13	−0.32	0.11	0.14	0.09	0.11	0.08	0.09	0.10	0.04	0.07	−0.41	0.06
0.40	α_1	−1.41	0.21	−1.05	0.22	−0.16	0.20	−1.44	0.15	0.25	0.14	−0.56	0.13	−1.26	0.12	−0.32	0.10	−1.06	0.10
	β_1	−0.36	0.11	−0.42	0.09	0.29	0.09	−1.60	0.08	−0.56	0.07	−0.54	0.06	−1.78	0.07	−1.32	0.06	−1.09	0.05
	α_2	−2.02	0.30	−2.02	0.28	−2.44	0.28	−1.17	0.19	−1.85	0.17	−1.64	0.17	−0.75	0.14	−1.25	0.13	−0.36	0.13
	β 2	−0.55	0.15	0.11	0.13	−0.71	0.12	−0.07	0.11	−0.48	0.09	−0.51	0.08	0.54	0.09	0.17	0.07	−0.04	0.07
0.50	α_1	−1.03	0.23	−1.09	0.20	−0.74	0.20	−1.93	0.15	−0.48	0.13	−0.76	0.13	−0.40	0.12	−0.77	0.10	−0.63	0.10
	β_1	−0.92	0.12	−0.32	0.09	−0.16	0.09	−1.99	0.09	−0.93	0.07	−0.78	0.06	−1.68	0.08	−1.49	0.06	−0.95	0.05
	α_2	−2.82	0.28	−3.50	0.27	−2.29	0.27	−0.47	0.18	−1.80	0.17	−0.79	0.16	0.12	0.13	−1.79	0.12	−1.14	0.12
	β 2	0.14	0.16	−0.72	0.13	0.25	0.13	0.33	0.11	−0.05	0.09	−0.11	0.08	0.43	0.09	0.10	0.07	−0.07	0.06

Abbreviation: RMSEs, root-mean-square errors.

Results are based on 1000 realizations of bivariate normal model.

Table 2.

Biases (in %) and RMSEs for normal data—TZ method.

		m = 40						m = 100						m = 200
ρ		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300
		Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE
0.10	α_1	−0.95	0.23	−1.73	0.22	−1.26	0.21	0.74	0.15	−0.58	0.14	−0.30	0.13	0.60	0.12	0.70	0.10	0.28	0.10
	β_1	−1.74	0.13	−0.86	0.11	−1.04	0.10	−0.19	0.09	−0.96	0.08	−0.87	0.07	−0.64	0.07	−0.51	0.06	−0.46	0.05
	α_1	−2.88	0.36	−1.93	0.34	−3.36	0.33	−1.73	0.22	−0.72	0.21	−1.00	0.21	−0.54	0.16	−1.62	0.15	−1.12	0.15
	β_1	1.10	0.19	−0.07	0.16	0.21	0.16	−0.99	0.12	−0.43	0.11	−0.14	0.10	0.00	0.10	−0.34	0.08	−0.44	0.08
0.20	α_1	−0.17	0.21	−1.57	0.22	−2.62	0.22	0.65	0.15	−0.92	0.13	−1.15	0.13	1.94	0.12	0.22	0.10	−0.30	0.10
	β_1	−1.41	0.12	−1.19	0.11	−1.17	0.11	−1.29	0.09	−1.45	0.08	−1.16	0.07	−0.37	0.07	−0.52	0.06	−0.81	0.05
	α_1	−3.71	0.34	−0.92	0.34	−2.01	0.32	−1.39	0.20	−0.76	0.20	−1.13	0.20	−1.30	0.16	−0.76	0.15	−0.61	0.14
	β_1	0.50	0.18	0.81	0.16	0.79	0.16	0.24	0.12	0.35	0.11	−0.15	0.10	−0.19	0.10	−0.10	0.08	0.08	0.08
0.40	α_1	0.21	0.24	−3.07	0.23	−1.23	0.21	1.22	0.15	0.48	0.14	−0.86	0.13	1.88	0.12	0.25	0.10	0.00	0.09
	β_1	−1.16	0.13	−1.43	0.11	−0.86	0.11	−0.84	0.09	−1.08	0.08	−1.06	0.07	−0.42	0.08	−0.43	0.06	−0.94	0.06
	α_1	−2.90	0.31	−1.42	0.32	−3.72	0.30	−2.20	0.19	−1.81	0.19	−1.15	0.19	−1.57	0.15	−0.82	0.13	−0.19	0.13
	β_1	0.72	0.18	1.08	0.16	−0.60	0.16	−0.11	0.12	0.10	0.11	−0.50	0.10	0.06	0.10	−0.23	0.08	0.35	0.07
0.50	α_1	−1.12	0.24	−0.83	0.22	−1.91	0.22	1.21	0.15	−0.15	0.14	−1.05	0.14	1.15	0.12	−0.21	0.10	−1.00	0.10
	β_1	−0.61	0.13	−0.81	0.12	−1.28	0.11	−0.87	0.09	−1.23	0.08	−1.28	0.07	−0.29	0.07	−0.71	0.06	−0.62	0.05
	α_1	−0.51	0.29	−2.26	0.30	−3.07	0.28	−1.70	0.19	−2.08	0.18	−1.05	0.17	−0.65	0.14	−0.50	0.12	−0.70	0.13
	β_1	0.53	0.17	−0.31	0.17	0.25	0.15	0.30	0.12	−0.10	0.10	0.34	0.10	−0.27	0.09	0.01	0.08	−0.07	0.07

Abbreviations: RMSEs, root-mean-square errors; TZ, Tang and Zhou.

Results are based on 1000 realizations of bivariate normal model.

Estimates from lognormal test results

We use the same setting as in the previous section to simulate the bivariate normal results first. We then take the exponential of the normal results to generate bivariate lognormal results. We again apply the proposed method and TZ method to the simulated data sets. Table 3 shows the biases and RMSEs of the ROC parameter estimates for the simulated lognormal data under all combinations of the sample sizes and correlation values for the proposed method. The simulation results show that the proposed approach has nice finite sample property as the biases and RMSEs are small even for small sample sizes. The results are similar as those for normal test results. This indicates that the proposed method is robust to monotonic transformation of the test results. As the sample sizes for both cancer and control groups become larger, the RMSEs tend to decrease. Table 4 shows the biases and RMSEs of the ROC parameter estimates for the simulated lognormal data under all the combinations of the sample sizes and correlation values for the TZ method. The biases and RMSEs are close to those by the proposed method.

Table 3.

Biases (in %) and RMSEs for lognormal data—proposed method.

		m = 40						m = 100						m = 200
ρ		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300
		Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE
0.10	α_1	−1.78	0.23	−1.48	0.20	−1.25	0.20	−1.21	0.14	−0.90	0.14	−0.92	0.13	−1.26	0.12	−1.20	0.10	−0.83	0.10
	β_1	−0.86	0.11	0.01	0.10	−0.24	0.09	−1.59	0.09	−0.90	0.07	−0.62	0.06	−1.62	0.08	−1.46	0.06	−1.04	0.05
	α_2	−2.01	0.36	−1.92	0.32	−2.52	0.31	−1.69	0.22	−1.29	0.21	−2.01	0.20	−0.57	0.17	−0.59	0.15	−0.91	0.14
	β_2	−0.57	0.16	−0.57	0.14	−0.08	0.13	−0.10	0.11	−0.18	0.09	−0.68	0.08	−0.06	0.10	0.46	0.08	−0.32	0.07
0.20	α_1	−2.38	0.21	−1.21	0.21	−1.30	0.21	−0.53	0.15	−1.21	0.13	−0.77	0.13	−0.38	0.12	−1.24	0.10	−0.90	0.10
	β_1	−0.76	0.11	−0.47	0.10	−0.14	0.09	−1.42	0.08	−1.28	0.07	−0.79	0.06	−1.44	0.07	−1.49	0.06	−1.06	0.05
	α_2	−1.99	0.31	−3.06	0.31	−3.79	0.32	−1.80	0.20	−1.44	0.19	−1.19	0.19	−0.50	0.15	−0.99	0.14	−0.89	0.13
	β_2	0.28	0.16	−0.35	0.14	−0.23	0.13	−0.32	0.11	0.12	0.09	−0.15	0.08	0.50	0.09	0.25	0.07	−0.40	0.07
0.40	α_1	−1.85	0.23	−1.11	0.21	−1.55	0.20	−1.07	0.15	−1.40	0.13	−1.14	0.13	−1.52	0.12	−0.75	0.10	−0.26	0.10
	β_1	−1.48	0.12	−0.44	0.10	0.22	0.09	−1.19	0.08	−1.19	0.07	−0.69	0.06	−1.59	0.07	−1.24	0.06	−1.06	0.05
	α_2	−1.85	0.30	−1.70	0.29	−0.61	0.28	−1.30	0.19	−1.50	0.18	−0.21	0.17	0.53	0.14	−0.62	0.13	−0.76	0.13
	β_2	1.03	0.16	0.38	0.13	0.16	0.13	−0.12	0.11	−0.32	0.09	−0.18	0.08	0.13	0.09	−0.12	0.07	0.15	0.06
0.50	α_1	−2.52	0.21	−0.35	0.19	−0.41	0.21	−1.91	0.15	−0.94	0.14	−0.53	0.13	−1.42	0.12	−1.19	0.10	−1.00	0.10
	β_1	−0.76	0.11	−0.20	0.09	0.22	0.09	−1.81	0.09	−1.21	0.07	−0.23	0.06	−1.83	0.07	−1.27	0.06	−0.90	0.05
	α_2	−2.40	0.30	−2.56	0.27	−3.71	0.28	−0.64	0.18	−0.71	0.17	−1.20	0.16	−0.32	0.14	−0.39	0.12	−0.83	0.12
	β_2	−0.10	0.16	−0.24	0.13	−0.95	0.13	0.41	0.11	0.16	0.09	−0.64	0.08	−0.02	0.09	−0.09	0.07	−0.19	0.066

Abbreviation: RMSEs, root-mean-square errors.

Results are based on 1000 realizations of bivariate lognormal model.

Table 4.

Biases (in %) and RMSEs for lognormal data—TZ method.

		m = 40						m = 100						m = 200
ρ		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300		n = 50		n = 150		n = 300
		Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE	Bias	RMSE
0.10	α_1	−2.08	0.23	−2.34	0.22	−1.18	0.22	0.20	0.15	−0.69	0.14	−0.31	0.13	1.37	0.12	−0.36	0.10	−0.77	0.10
	β_1	−1.10	0.13	−0.96	0.11	−0.84	0.11	−0.73	0.09	−1.03	0.08	−1.04	0.07	−0.80	0.07	−0.85	0.06	−1.08	0.05
	α_1	−2.97	0.37	−0.88	0.34	−3.00	0.33	−1.11	0.22	−1.48	0.20	−2.27	0.21	−1.62	0.17	0.41	0.15	−0.26	0.14
	β_1	−0.49	0.19	0.42	0.16	0.50	0.15	−0.43	0.13	−0.68	0.11	−0.02	0.11	0.11	0.10	0.47	0.08	0.39	0.07
0.20	α_1	−1.05	0.23	−1.12	0.21	−1.34	0.21	0.21	0.16	−0.53	0.14	−1.32	0.13	1.47	0.12	−0.02	0.10	−0.72	0.10
	β_1	−0.51	0.12	−0.53	0.11	−1.01	0.11	−1.22	0.10	−0.94	0.07	−1.13	0.07	−0.36	0.07	−0.74	0.06	−0.58	0.05
	α_1	−1.43	0.34	−4.06	0.32	−2.64	0.33	−1.41	0.22	−1.90	0.20	−0.90	0.19	−1.68	0.16	−0.82	0.14	−0.52	0.14
	β_1	−0.07	0.17	−1.24	0.16	0.25	0.16	0.23	0.13	−0.36	0.11	−0.13	0.10	−0.38	0.10	0.09	0.08	−0.38	0.08
0.40	α_1	0.44	0.22	−2.02	0.22	−2.80	0.22	0.40	0.15	−0.77	0.14	−1.45	0.14	1.25	0.12	0.75	0.10	−0.86	0.09
	β_1	−1.17	0.13	−0.96	0.11	−1.72	0.11	−0.86	0.09	−0.72	0.08	−1.42	0.07	−0.48	0.07	−0.33	0.06	−0.58	0.06
	α_1	−3.74	0.32	−3.45	0.32	−2.72	0.32	−0.91	0.19	−1.52	0.19	−0.87	0.18	−0.78	0.14	−1.16	0.13	−0.30	0.13
	β_1	−0.32	0.18	−0.12	0.17	0.35	0.16	−0.19	0.12	−0.46	0.11	0.23	0.10	0.00	0.10	−0.08	0.08	−0.09	0.08
0.50	α_1	−1.78	0.23	−2.79	0.22	−3.26	0.22	0.31	0.15	−1.46	0.14	−0.59	0.14	2.04	0.12	−0.18	0.10	−0.08	0.10
	β_1	−1.44	0.13	−1.40	0.11	−1.07	0.11	−0.98	0.09	−1.39	0.08	−1.00	0.07	−0.40	0.07	−0.54	0.06	−0.70	0.05
	α_1	−2.84	0.30	−3.24	0.27	−0.74	0.30	−1.84	0.19	−0.72	0.17	−1.22	0.17	−1.54	0.13	−0.61	0.12	−1.19	0.13
	β_1	−0.41	0.17	−0.12	0.15	0.42	0.16	0.12	0.11	0.05	0.10	0.02	0.10	0.13	0.09	−0.45	0.08	0.00	0.08

Abbreviations: RMSEs, root-mean-square errors; TZ, Tang and Zhou.

Results are based on 1000 realizations of bivariate lognormal model.

Applications to Cancer Diagnostic Biomarkers

We apply our method to 2 real data sets. The first example investigates the diagnostic accuracy of serum biomarkers on pancreatic cancer, and the second example investigates the accuracy of gene expression biomarkers on ovarian cancer.

Pancreatic cancer tests

We use the cancer diagnostic example in Wieand et al[13] to illustrate the proposed method. The example is popular for the illustration of methodologies on estimating ROC curves from correlated data. Two pancreatic cancer tests, CA 19-9 and CA 125, were measured on 51 patients with pancreatitis and 90 patients with pancreatic cancer. It is of interest to estimate the ROC curves for these 2 tests. The test results approximately follow normal distributions after some monotonical transformation. We can assume a bivariate binormal ROC model for these tests. The estimation procedure follows the steps in section “Simulation Studies.” We first define cutpoints for 2 tests. For each test, we take the minimum and maximum of the combined results from both cancer and control groups as the lower and upper bounds and then obtain 100 equally spaced points within the bounds. The sets of cutpoints, , are different for 2 tests. The sensitivity at a cutpoint for a test, , is calculated as the proportions that the test results are greater than the cutpoint, and the specificity, , is calculated as the proportion that the test results are less than or equal to the cutpoint. The pairs of sensitivities and specificities are then obtained for each test. The response vector in the linear model is given as follows:

and the design matrix is as follows:

The final LS estimates are obtain through the following equation:

The parameter estimates are (1.0550,. Based on these estimates, the estimated ROC curves are given by ROC1(u) = Φ(1.0550 + for CA19-9 and ROC2(u) = Φ(0.7298 + for CA 125. Figure 2 shows the fitted ROC curves for 2 tests. Both fitted curves are close to the empirical ROC curves. Unlike the rough empirical curves, the fitted curves are much smoother.

Figure 2.

ROC curves for CA 19-9 and CA 125: solid lines, the proposed method; dashed and dotted lines, empirical ROC curves. ROC indicates receiver operating characteristic.

Ovarian cancer tests

We also illustrate our method using a gene expression data set previously analyzed using a suite of traditional ROC methods in a study by Pepe et al.[14] Briefly, these data report messenger RNA (mRNA) expression levels for 1536 gene clones in 30 subjects with ovarian cancer and 23 without cancer. We will focus our example on 2 gene clones, SPINT2 and TACSTD1, among the top 10 ranking clones identified in this prior work. SPINT2 is associated with ovarian cancer[15] and fallopian tube carcinomas specifically.[16] Elevated expression levels of TACSTD1, also called EPCAM, have been associated with local and metastatic prostate cancer[17] and colorectal cancer[18] while potentially being protective against ovarian cancer.[19]

We follow the same approach from the first example to define cutpoints for 2 biomarkers and obtain 100 equally spaced points within the bounds. The parameter estimates based on the proposed method are . Based on these estimates, the estimated ROC curves are given by for SPINT2 and for TACSTD1. The TZ method[7] is also applied to the data set to show the difference in the fitted ROC curves. The parameter estimates based on the proposed method are (1.7078,0.3242,. Based on these estimates, the estimated ROC curves are given by for SPINT2 and for TACSTD1. Figure 3 shows the fitted ROC curves from the proposed method and the TZ method for 2 tests. The fitted curves are close to the empirical ROC curves. Unlike the rough empirical curves, the fitted curves are much smoother.

Figure 3.

ROC curves for SPINT2 and TACSTD1: solid lines, the fitted ROC curves with the proposed method in black and TZ method in blue; dashed lines, empirical ROC curves. ROC indicates receiver operating characteristic; TZ, Tang and Zhou.

Discussion

This article proposes an LS method to estimate the ROC parameters. The method builds on the estimated sensitivities and specificities. This method differs from that of Tang and Zhou[7] by handling the case of continuous response data and ordinal response data. The key difference between the 2 methods lies in the response variable. The response variable in the latter is transformed empirical ROC curves at different thresholds. It takes on many values for continuous test results, but few values for ordinal test results. The limited number of values for the response variable makes it impractical for ordinal data. However, the response variable in the proposed method takes on many more distinct values so that the method yields valid estimates for ordinal data. The simulation studies show that the proposed method has good finite sample performance for both simulated normal and lognormal data. The method also shows satisfactory results in cancer diagnostic examples.

As demonstrated by Hanley,[20] the binormal ROC curve tends to fit data to other distributions reasonably well. However, the assumption of the binormal ROC curve may seem quite strong because the data need to be normal after some unknown transformation. As a future research topic, more simulation studies need to be conducted for other distributions and for ordinal data to investigate the finite sample performance of the proposed method.

The method proposed here assumes that the gold standard is known. Future topics include the estimation of the ROC curves without the presence of the gold standard or when the gold standard is imperfect. It is challenging to do so because our method requires valid estimates for sensitivities and specificities. The method of Hui and Walter[21] may be applied, but 2 or more populations are required for the estimation. If 2 binary tests are to be evaluated from the samples in 1 population, the sensitivity and specificity cannot be estimated with the absence of a gold standard. Under this situation, the 5 parameters to be estimated involve the prevalence, sensitivities, and specificities for 2 tests. However, only 3 degrees of freedom are allowed with testing within 1 population and are not sufficient for estimating 5 parameters. Testing on the samples from 2 populations increases the degree of freedom to 6. Hui and Walter[21] consider the setting in which multiple tests are applied to several populations and discuss the approaches to estimate the sensitivities and specificities. The estimated sensitivities and specificities can potentially be used as response variables in our method to generate valid ROC curve estimators for continuous data.