D D Dorfman1, K S Berbaum. 1. Department of Psychology, University of Iowa, Iowa City 52242, USA.
Abstract
RATIONALE AND OBJECTIVES: Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic. METHODS: Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies. RESULTS: RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased. CONCLUSION: RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.
RATIONALE AND OBJECTIVES: Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic. METHODS: Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies. RESULTS: RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased. CONCLUSION: RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.