Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision thresholds on bias, precision, size and power.

This paper concerns robustness of the binormal assumption for inferences that pertain to the area under an ROC curve. I applied the binormal model to rating method data sets sampled from bilogistic curves and observed small biases in area estimates. Bias increased as the range of decision thresholds decreased. The variance of area estimates also increased as the range of decision thresholds decreased. Together, minor bias and inflated variance substantially altered the size and power of statistical tests that compared areas under bilogistic ROC curves. I repeated the simulations by applying the binormal assumption to data sampled from binormal curves. Biases in area estimates were minimal for the binormal data, but the variance of area estimates was again higher when the range of decision thresholds was narrow. The size of tests that compared areas did not vary from the chosen significance level. Power fell, however, when the variance of area estimates was inflated. I conclude that inferences derived from the binormal assumption are sensitive to model misspecification and to the location of decision thresholds. A narrow span of decision thresholds increases the variability of area estimates and reduces the power of area comparisons. Model misspecification produces bias that alters test size and can exaggerate the loss of power that accompanies increased variability.