Phillip R Hunt1, Melissa C Friesen2, Susan Sama3, Louise Ryan4, Donald Milton5. 1. 1.Retrospective Observation Studies, Evidera, 430 Bedford St, Suite 300, Lexington, MA 02420, USA phillip.hunt@evidera.com. 2. 2.Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Bethesda, MD 20892, USA. 3. 3.Work Environment Department, University of Massachusetts Lowell, One University Avenue, Lowell, MA 01854, USA. 4. 4.School of Mathematical Sciences, University of Technology Sidney, 15 Broadway, Ultimo, NSW 2007, Australia. 5. 5.Maryland Institute for Applied Environmental Health, University of Maryland School of Public Health, 2234V School of Public Health, College Park, Maryland 20742, USA.
Abstract
BACKGROUND: Evaluation of expert assessment of exposure depends, in the absence of a validation measurement, upon measures of agreement among the expert raters. Agreement is typically measured using Cohen's Kappa statistic, however, there are some well-known limitations to this approach. We demonstrate an alternate method that uses log-linear models designed to model agreement. These models contain parameters that distinguish between exact agreement (diagonals of agreement matrix) and non-exact associations (off-diagonals). In addition, they can incorporate covariates to examine whether agreement differs across strata. METHODS: We applied these models to evaluate agreement among expert ratings of exposure to sensitizers (none, likely, high) in a study of occupational asthma. RESULTS: Traditional analyses using weighted kappa suggested potential differences in agreement by blue/white collar jobs and office/non-office jobs, but not case/control status. However, the evaluation of the covariates and their interaction terms in log-linear models found no differences in agreement with these covariates and provided evidence that the differences observed using kappa were the result of marginal differences in the distribution of ratings rather than differences in agreement. Differences in agreement were predicted across the exposure scale, with the likely moderately exposed category more difficult for the experts to differentiate from the highly exposed category than from the unexposed category. CONCLUSIONS: The log-linear models provided valuable information about patterns of agreement and the structure of the data that were not revealed in analyses using kappa. The models' lack of dependence on marginal distributions and the ease of evaluating covariates allow reliable detection of observational bias in exposure data. Published by Oxford University Press on behalf of the British Occupational Hygiene Society 2015.
BACKGROUND: Evaluation of expert assessment of exposure depends, in the absence of a validation measurement, upon measures of agreement among the expert raters. Agreement is typically measured using Cohen's Kappa statistic, however, there are some well-known limitations to this approach. We demonstrate an alternate method that uses log-linear models designed to model agreement. These models contain parameters that distinguish between exact agreement (diagonals of agreement matrix) and non-exact associations (off-diagonals). In addition, they can incorporate covariates to examine whether agreement differs across strata. METHODS: We applied these models to evaluate agreement among expert ratings of exposure to sensitizers (none, likely, high) in a study of occupational asthma. RESULTS: Traditional analyses using weighted kappa suggested potential differences in agreement by blue/white collar jobs and office/non-office jobs, but not case/control status. However, the evaluation of the covariates and their interaction terms in log-linear models found no differences in agreement with these covariates and provided evidence that the differences observed using kappa were the result of marginal differences in the distribution of ratings rather than differences in agreement. Differences in agreement were predicted across the exposure scale, with the likely moderately exposed category more difficult for the experts to differentiate from the highly exposed category than from the unexposed category. CONCLUSIONS: The log-linear models provided valuable information about patterns of agreement and the structure of the data that were not revealed in analyses using kappa. The models' lack of dependence on marginal distributions and the ease of evaluating covariates allow reliable detection of observational bias in exposure data. Published by Oxford University Press on behalf of the British Occupational Hygiene Society 2015.
Authors: Lin Fritschi; Louise Nadon; Geza Benke; Ramzan Lakhani; Benoit Latreille; Marie-Elise Parent; Jack Siemiatycki Journal: Am J Ind Med Date: 2003-05 Impact factor: 2.214
Authors: Andrea 't Mannetje; Joelle Fevotte; Tony Fletcher; Paul Brennan; Joszef Legoza; Maria Szeremi; Ana Paldy; Slawomir Brzeznicki; Jan Gromiec; Carmen Ruxanda-Artenie; Rodica Stanescu-Dumitru; Nicolai Ivanov; Raphael Shterengorz; Lubica Hettychova; Daniela Krizanova; Adrian Cassidy; Martie van Tongeren; Paolo Boffetta Journal: Epidemiology Date: 2003-09 Impact factor: 4.822
Authors: Melissa C Friesen; Joseph B Coble; Hormuzd A Katki; Bu-Tian Ji; Shouzheng Xue; Wei Lu; Patricia A Stewart Journal: Ann Occup Hyg Date: 2011-04-21
Authors: Svetlana Solovieva; Irmeli Pehkonen; Johanna Kausto; Helena Miranda; Rahman Shiri; Timo Kauppinen; Markku Heliövaara; Alex Burdorf; Kirsti Husgafvel-Pursiainen; Eira Viikari-Juntura Journal: PLoS One Date: 2012-11-12 Impact factor: 3.240