Joanna H Shih1, Matthew D Greer2, Baris Turkbey2. 1. Biometric Research Program, NCI, NIH, 9609 Medical Center Drive, Room 5W124, MSC 9735, Bethesda, MD 20892. Electronic address: jshih@mail.nih.gov. 2. Molecular Imaging Program, NCI, NIH, Bethesda, Maryland.
Abstract
RATIONALE AND OBJECTIVES: To point out the problems with Cohen kappa statistic and to explore alternative metrics to determine interobserver agreement on lesion detection when locations are not prespecified. MATERIALS AND METHODS: Use of kappa and two alternative methods, namely index of specific agreement (ISA) and modified kappa, for measuring interobserver agreement on the location of detected lesions are presented. These indices of agreement are illustrated by application to a retrospective multireader study in which nine readers detected and scored prostate cancer lesions in 163 consecutive patients (n = 110 cases, n = 53 controls) using the guideline of Prostate Imaging Reporting and Data System version 2 on multiparametric magnetic resonance imaging. RESULTS: The proposed modified kappa, which properly corrects for the amount of agreement by chance, is shown to be approximately equivalent to the ISA. In the prostate cancer data, average kappa, modified kappa, and ISA equaled 30%, 55%, and 57%, respectively, for all lesions and 20%, 87%, and 87%, respectively, for index lesions. CONCLUSIONS: The application of kappa could result in a substantial downward bias in reader agreement on lesion detection when locations are not prespecified. ISA is recommended for assessment of reader agreement on lesion detection. Published by Elsevier Inc.
RATIONALE AND OBJECTIVES: To point out the problems with Cohen kappa statistic and to explore alternative metrics to determine interobserver agreement on lesion detection when locations are not prespecified. MATERIALS AND METHODS: Use of kappa and two alternative methods, namely index of specific agreement (ISA) and modified kappa, for measuring interobserver agreement on the location of detected lesions are presented. These indices of agreement are illustrated by application to a retrospective multireader study in which nine readers detected and scored prostate cancer lesions in 163 consecutive patients (n = 110 cases, n = 53 controls) using the guideline of Prostate Imaging Reporting and Data System version 2 on multiparametric magnetic resonance imaging. RESULTS: The proposed modified kappa, which properly corrects for the amount of agreement by chance, is shown to be approximately equivalent to the ISA. In the prostate cancer data, average kappa, modified kappa, and ISA equaled 30%, 55%, and 57%, respectively, for all lesions and 20%, 87%, and 87%, respectively, for index lesions. CONCLUSIONS: The application of kappa could result in a substantial downward bias in reader agreement on lesion detection when locations are not prespecified. ISA is recommended for assessment of reader agreement on lesion detection. Published by Elsevier Inc.
Entities:
Keywords:
MRI; Prostate cancer; index of specific agreement; interobserver agreement; kappa; lesion detection
Authors: Matthew D Greer; Joanna H Shih; Nathan Lay; Tristan Barrett; Leonardo Bittencourt; Samuel Borofsky; Ismail Kabakus; Yan Mee Law; Jamie Marko; Haytham Shebel; Maria J Merino; Bradford J Wood; Peter A Pinto; Ronald M Summers; Peter L Choyke; Baris Turkbey Journal: AJR Am J Roentgenol Date: 2019-03-27 Impact factor: 3.959
Authors: Sherif Mehralivand; Stephanie A Harmon; Joanna H Shih; Clayton P Smith; Nathan Lay; Burak Argun; Sandra Bednarova; Ronaldo Hueb Baroni; Abdullah Erdem Canda; Karabekir Ercan; Rossano Girometti; Ercan Karaarslan; Ali Riza Kural; Andrei S Purysko; Soroush Rais-Bahrami; Victor Martins Tonso; Cristina Magi-Galluzzi; Jennifer B Gordetsky; Ricardo Silvestre E Silva Macarenco; Maria J Merino; Berrak Gumuskaya; Yesim Saglican; Stefano Sioletic; Anne Y Warren; Tristan Barrett; Leonardo Bittencourt; Mehmet Coskun; Chris Knauss; Yan Mee Law; Ashkan A Malayeri; Daniel J Margolis; Jamie Marko; Derya Yakar; Bradford J Wood; Peter A Pinto; Peter L Choyke; Ronald M Summers; Baris Turkbey Journal: AJR Am J Roentgenol Date: 2020-08-05 Impact factor: 3.959
Authors: Sonia Gaur; Nathan Lay; Stephanie A Harmon; Sreya Doddakashi; Sherif Mehralivand; Burak Argun; Tristan Barrett; Sandra Bednarova; Rossanno Girometti; Ercan Karaarslan; Ali Riza Kural; Aytekin Oto; Andrei S Purysko; Tatjana Antic; Cristina Magi-Galluzzi; Yesim Saglican; Stefano Sioletic; Anne Y Warren; Leonardo Bittencourt; Jurgen J Fütterer; Rajan T Gupta; Ismail Kabakus; Yan Mee Law; Daniel J Margolis; Haytham Shebel; Antonio C Westphalen; Bradford J Wood; Peter A Pinto; Joanna H Shih; Peter L Choyke; Ronald M Summers; Baris Turkbey Journal: Oncotarget Date: 2018-09-18