Robert West1, Ann McIlwaine. 1. Psychology Department, St George's Hospital Medical School, London, UK. sgt600@sghms.ac.uk
Abstract
AIMS: This study investigated the value of citation counts as an index of quality in the field of addiction and examined factors that contribute to papers being cited more or less frequently. DESIGN: The number of times papers published by the journal Addiction in 1995-98 that had been cited up to May 2000 were counted using the Science and the Social Science Citation Indexes. Articles in nine of the monthly issues from 1997 were rated by two independent expert raters for quality. Factors related to citation counts were also examined including: country of origin of the paper, substance type, solicited versus unsolicited papers and methodology used. FINDINGS: A total of 417 unsolicited research reports were included in the citation analysis, of which 79 were also subjected to quality ratings. The experts showed a moderate level of agreement in their ratings (intraclass correlation = 0.39. p < 0.001). However, there was no correlation between number of citations and expert ratings of article quality (R < 0.1). Papers from developing countries received significantly fewer citations than papers from other countries but substance type (e.g. nicotine, opiate, alcohol) and methodology (e.g. survey, treatment trial) were not related to number of citations. CONCLUSIONS: This study involved just one journal but raised an important issue: the number of citations received by papers on addiction appears to reflect the geographical region of study rather than what experts would consider as 'quality'. If these findings are found to generalize they call into question the use of citation-related indices as measures of quality in this field and perhaps in others as well. To our knowledge our methodology has not been used before and could be adapted to study the value of citations more widely.
AIMS: This study investigated the value of citation counts as an index of quality in the field of addiction and examined factors that contribute to papers being cited more or less frequently. DESIGN: The number of times papers published by the journal Addiction in 1995-98 that had been cited up to May 2000 were counted using the Science and the Social Science Citation Indexes. Articles in nine of the monthly issues from 1997 were rated by two independent expert raters for quality. Factors related to citation counts were also examined including: country of origin of the paper, substance type, solicited versus unsolicited papers and methodology used. FINDINGS: A total of 417 unsolicited research reports were included in the citation analysis, of which 79 were also subjected to quality ratings. The experts showed a moderate level of agreement in their ratings (intraclass correlation = 0.39. p < 0.001). However, there was no correlation between number of citations and expert ratings of article quality (R < 0.1). Papers from developing countries received significantly fewer citations than papers from other countries but substance type (e.g. nicotine, opiate, alcohol) and methodology (e.g. survey, treatment trial) were not related to number of citations. CONCLUSIONS: This study involved just one journal but raised an important issue: the number of citations received by papers on addiction appears to reflect the geographical region of study rather than what experts would consider as 'quality'. If these findings are found to generalize they call into question the use of citation-related indices as measures of quality in this field and perhaps in others as well. To our knowledge our methodology has not been used before and could be adapted to study the value of citations more widely.
Authors: Mohit Bhandari; Jason Busse; P J Devereaux; Victor M Montori; Marc Swiontkowski; Paul Tornetta Iii; Thomas A Einhorn; Vikas Khera; Emil H Schemitsch Journal: Can J Surg Date: 2007-04 Impact factor: 2.089