Henrica C W de Vet1, Rieky E Dikmans2, Iris Eekhout2. 1. Department of Epidemiology and Biostatistics, EMGO Institute for Health and Care Research, VU Medical Center, De Boelelaan 1089A, Amsterdam 1081HV, The Netherlands. Electronic address: hcw.devet@vumc.nl. 2. Department of Epidemiology and Biostatistics, EMGO Institute for Health and Care Research, VU Medical Center, De Boelelaan 1089A, Amsterdam 1081HV, The Netherlands.
Abstract
OBJECTIVE: For assessing interrater agreement, the concepts of observed agreement and specific agreement have been proposed. The situation of two raters and dichotomous outcomes has been described, whereas often, multiple raters are involved. We aim to extend it for more than two raters and examine how to calculate agreement estimates and 95% confidence intervals (CIs). STUDY DESIGN AND SETTING: As an illustration, we used a reliability study that includes the scores of four plastic surgeons classifying photographs of breasts of 50 women after breast reconstruction into "satisfied" or "not satisfied." In a simulation study, we checked the hypothesized sample size for calculation of 95% CIs. RESULTS: For m raters, all pairwise tables [ie, m (m - 1)/2] were summed. Then, the discordant cells were averaged before observed and specific agreements were calculated. The total number (N) in the summed table is m (m - 1)/2 times larger than the number of subjects (n), in the example, N = 300 compared to n = 50 subjects times m = 4 raters. A correction of n√(m - 1) was appropriate to find 95% CIs comparable to bootstrapped CIs. CONCLUSION: The concept of observed agreement and specific agreement can be extended to more than two raters with a valid estimation of the 95% CIs.
OBJECTIVE: For assessing interrater agreement, the concepts of observed agreement and specific agreement have been proposed. The situation of two raters and dichotomous outcomes has been described, whereas often, multiple raters are involved. We aim to extend it for more than two raters and examine how to calculate agreement estimates and 95% confidence intervals (CIs). STUDY DESIGN AND SETTING: As an illustration, we used a reliability study that includes the scores of four plastic surgeons classifying photographs of breasts of 50 women after breast reconstruction into "satisfied" or "not satisfied." In a simulation study, we checked the hypothesized sample size for calculation of 95% CIs. RESULTS: For m raters, all pairwise tables [ie, m (m - 1)/2] were summed. Then, the discordant cells were averaged before observed and specific agreements were calculated. The total number (N) in the summed table is m (m - 1)/2 times larger than the number of subjects (n), in the example, N = 300 compared to n = 50 subjects times m = 4 raters. A correction of n√(m - 1) was appropriate to find 95% CIs comparable to bootstrapped CIs. CONCLUSION: The concept of observed agreement and specific agreement can be extended to more than two raters with a valid estimation of the 95% CIs.
Authors: Alicia Muñoz-Gallego; José L Torres-Peña; Martín Rodríguez-Salgado; Ana Ortueta-Olartecoechea; Cristina López-López; Javier De la Cruz; Pilar Tejada-Palacios Journal: Graefes Arch Clin Exp Ophthalmol Date: 2020-08-29 Impact factor: 3.117
Authors: Amir H Zamanipoor Najafabadi; Marthe C M Peeters; Daniel J Lobatto; Marieke L D Broekman; Timothy R Smith; Nienke R Biermasz; Saskia M Peerdeman; Wilco C Peul; Martin J B Taphoorn; Wouter R van Furth; Linda Dirven Journal: Acta Neurochir (Wien) Date: 2017-09-27 Impact factor: 2.216
Authors: Beate Zoch-Lesniak; Robert S Ware; Keith Grimwood; Stephen B Lambert Journal: J Pediatric Infect Dis Soc Date: 2020-04-30 Impact factor: 3.164
Authors: Andrea Coleman; Amanda Wood; Seweryn Bialasiewicz; Robert S Ware; Robyn L Marsh; Anders Cervin Journal: Microbiome Date: 2018-11-05 Impact factor: 14.650
Authors: D J Kadouch; A S E van Haersma de With; Y S Elshot; M Peppelman; M W Bekkenk; A Wolkerstorfer; I Eekhout; C A C Prinsen; M A de Rie Journal: J Eur Acad Dermatol Venereol Date: 2018-01-23 Impact factor: 6.166
Authors: Alicia Muñoz-Gallego; Javier De la Cruz; Martín Rodríguez-Salgado; José L Torres-Peña; Javier Sambricio; Ana Ortueta-Olartecoechea; Pilar Tejada-Palacios Journal: BMC Ophthalmol Date: 2020-05-24 Impact factor: 2.209
Authors: Hedy A van Oers; Lorynn Teela; Sasja A Schepers; Martha A Grootenhuis; Lotte Haverman Journal: Qual Life Res Date: 2020-08-15 Impact factor: 4.147