Viswanathan Shankar1, Shrikant I Bangdiwala. 1. Division of Biostatistics, Department of Epidemiology and Population Health, Albert Einstein College of Medicine, Bronx, NY 10461, USA. shankar.viswanathan@einstein.yu.edu.
Abstract
BACKGROUND: Various measures of observer agreement have been proposed for 2 x 2 tables. We examine the behavior of alternative measures of observer agreement for 2 x 2 tables. METHODS: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. RESULTS: Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. CONCLUSIONS: The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2 x 2 tables.
BACKGROUND: Various measures of observer agreement have been proposed for 2 x 2 tables. We examine the behavior of alternative measures of observer agreement for 2 x 2 tables. METHODS: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. RESULTS: Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. CONCLUSIONS: The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2 x 2 tables.
Authors: Andrew B Rosenkrantz; Luke A Ginocchio; Daniel Cornfeld; Adam T Froemming; Rajan T Gupta; Baris Turkbey; Antonio C Westphalen; James S Babb; Daniel J Margolis Journal: Radiology Date: 2016-04-01 Impact factor: 11.105
Authors: Johanna Boretzki; Eva Wolf; Carmen Wiese; Sebastian Noe; Annamaria Balogh; Anja Meurer; Ivanka Krznaric; Alexander Zink; Christian Lersch; Christoph D Spinner Journal: Patient Prefer Adherence Date: 2017-11-08 Impact factor: 2.711
Authors: Marieke J Schreuder; Robin N Groen; Johanna T W Wigman; Catharina A Hartman; Marieke Wichers Journal: BMC Psychiatry Date: 2020-07-06 Impact factor: 3.630
Authors: Christiane Muth; Sebastian Harder; Lorenz Uhlmann; Justine Rochon; Birgit Fullerton; Corina Güthlin; Antje Erler; Martin Beyer; Marjan van den Akker; Rafael Perera; André Knottnerus; Jose M Valderas; Ferdinand M Gerlach; Walter E Haefeli Journal: BMJ Open Date: 2016-07-25 Impact factor: 2.692
Authors: Kelly Samara Silva; Jaqueline Aragoni da Silva; Valter Cordeiro Barbosa Filho; Priscila Cristina Dos Santos; Pablo Magno da Silveira; Marcus V V Lopes; Jo Salmon Journal: Medicine (Baltimore) Date: 2020-07-31 Impact factor: 1.817