Werner Vach1. 1. Department of Statistics, University of Southern Denmark, Campusvej 55, 5230 Odense M, Denmark. werner.v@stat.sdu.dk
Abstract
BACKGROUND AND OBJECTIVE: The dependence of Cohen's kappa on the prevalence has been a major concern in the literature. Indeed, it indicates a serious limitation with respect to comparing kappa-values among studies with varying prevalences. STUDY DESIGN AND SETTING: The basic arguments used by different authors are reviewed. RESULTS: Two types of dependence can be distinguished: a dependence on the observed marginal prevalences and a dependence on the prevalence of a latent binary variable, representing the true status. The first dependence is simply a consequence of the purpose of kappa, which is to improve the interpretation of agreement rates, and so does not constitute a real argument against kappa. The second occurs only if one can change the prevalence without changing sensitivity and specificity. Typically, in agreement studies a change in prevalence implies also a change in sensitivity and specificity, and we show that in such a framework the dependence on the prevalence becomes negligible. CONCLUSION: We should stop criticizing kappa for its dependence on the prevalence. Instead, we should focus on its dependence on the composition of the population with respect to subjects easy or difficult to agree on.
BACKGROUND AND OBJECTIVE: The dependence of Cohen's kappa on the prevalence has been a major concern in the literature. Indeed, it indicates a serious limitation with respect to comparing kappa-values among studies with varying prevalences. STUDY DESIGN AND SETTING: The basic arguments used by different authors are reviewed. RESULTS: Two types of dependence can be distinguished: a dependence on the observed marginal prevalences and a dependence on the prevalence of a latent binary variable, representing the true status. The first dependence is simply a consequence of the purpose of kappa, which is to improve the interpretation of agreement rates, and so does not constitute a real argument against kappa. The second occurs only if one can change the prevalence without changing sensitivity and specificity. Typically, in agreement studies a change in prevalence implies also a change in sensitivity and specificity, and we show that in such a framework the dependence on the prevalence becomes negligible. CONCLUSION: We should stop criticizing kappa for its dependence on the prevalence. Instead, we should focus on its dependence on the composition of the population with respect to subjects easy or difficult to agree on.
Authors: Sintayehu Gebresillasie; Zerihun Tadesse; Ayalew Shiferaw; Sun N Yu; Nicole E Stoller; Zhaoxia Zhou; Paul M Emerson; Bruce D Gaynor; Thomas M Lietman; Jeremy D Keenan Journal: Ophthalmic Epidemiol Date: 2015 Impact factor: 1.648
Authors: Molly McFatrich; Jennifer Brondon; Nicole R Lucas; Pamela S Hinds; Scott H Maurer; Jennifer W Mack; David R Freyer; Shana S Jacobs; Justin N Baker; Catriona Mowbray; Mian Wang; Sharon M Castellino; Allison Barz Leahy; Bryce B Reeve Journal: Cancer Date: 2019-09-25 Impact factor: 6.860
Authors: Maria M Wertli; Franziska Christina Buletti; Ulrike Held; Eva Rasmussen-Barr; Sherri Weiser; Jakob M Burgstaller; Johann Steurer Journal: Eur Spine J Date: 2016-05-13 Impact factor: 3.134
Authors: Lidwine B Mokkink; Caroline B Terwee; Elizabeth Gibbons; Paul W Stratford; Jordi Alonso; Donald L Patrick; Dirk L Knol; Lex M Bouter; Henrica C W de Vet Journal: BMC Med Res Methodol Date: 2010-09-22 Impact factor: 4.615