Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity.

Paradoxical effects of bias and prevalence on the kappa coefficient are examined using the concepts of sensitivity and specificity. Results that appear paradoxical when viewed as a 2 x 2 table of frequencies do not appear paradoxical when viewed as a pair of sensitivity and specificity measures where each observer is treated as a predictor of the other observer. An adjusted kappa value can be obtained from these sensitivity/specificity measures but simulation studies indicate that it would result in substantial overestimation of reliability when bias or prevalence effects are observed. It is suggested that investigators concentrate on obtaining populations with trait prevalence near 50% rather than searching for statistical indices to rescue or excuse inefficient experiments.