STUDY OBJECTIVES: In 1997, Annals of Emergency Medicine initiated a protocol by which every original research article, in addition to each regular review, was concurrently evaluated by 1 of 2 methodology and statistical reviewers. We characterized and contrasted comments made by the methodology and regular peer reviewers. METHODS: After pilot testing, interrater reliability assessment, and revision, we finalized a 99-item taxonomy of reviewer comments organized in 8 categories. Two authors, uninvolved in the writing of reviews, classified each comment from a random sample of methodology reviews from 1999. For 30 of these reviews (15 for each methodology reviewer), the 2 authors also scored all (range 2 to 5) regular reviews. RESULTS: Sixty-five reviews by methodologist A, 60 by methodologist B, and 68 by regular reviewers were analyzed. Comments by methodologist A most frequently concerned the presentation of results (33% of all comments) and methods (17%). Methodologist B commented most frequently on presentation of results (28%) and statistical methods (16%). Regular reviewers most frequently made non-methodology/statistical comments (45%) and comments on presentation of results (18%). Of note, comments made by methodology and regular reviewers about methods issues were often contradictory. CONCLUSION: The distributions of comments made by the 2 methodology and statistical reviewers were similar, although reviewer A emphasized presentation and reviewer B stressed statistical issues. The regular reviewers (most of whom were unaware that a dedicated methodology and statistical reviewer would be reviewing the article) paid much less attention to methodology issues. The 2 dedicated methodology and statistical reviewers created reviews that were similarly focused and emphasized methodology issues that were distinct from the issues raised by regular reviewers.
STUDY OBJECTIVES: In 1997, Annals of Emergency Medicine initiated a protocol by which every original research article, in addition to each regular review, was concurrently evaluated by 1 of 2 methodology and statistical reviewers. We characterized and contrasted comments made by the methodology and regular peer reviewers. METHODS: After pilot testing, interrater reliability assessment, and revision, we finalized a 99-item taxonomy of reviewer comments organized in 8 categories. Two authors, uninvolved in the writing of reviews, classified each comment from a random sample of methodology reviews from 1999. For 30 of these reviews (15 for each methodology reviewer), the 2 authors also scored all (range 2 to 5) regular reviews. RESULTS: Sixty-five reviews by methodologist A, 60 by methodologist B, and 68 by regular reviewers were analyzed. Comments by methodologist A most frequently concerned the presentation of results (33% of all comments) and methods (17%). Methodologist B commented most frequently on presentation of results (28%) and statistical methods (16%). Regular reviewers most frequently made non-methodology/statistical comments (45%) and comments on presentation of results (18%). Of note, comments made by methodology and regular reviewers about methods issues were often contradictory. CONCLUSION: The distributions of comments made by the 2 methodology and statistical reviewers were similar, although reviewer A emphasized presentation and reviewer B stressed statistical issues. The regular reviewers (most of whom were unaware that a dedicated methodology and statistical reviewer would be reviewing the article) paid much less attention to methodology issues. The 2 dedicated methodology and statistical reviewers created reviews that were similarly focused and emphasized methodology issues that were distinct from the issues raised by regular reviewers.
Authors: Jeffrey L Jackson; Malathi Srinivasan; Joanna Rea; Kathlyn E Fletcher; Richard L Kravitz Journal: PLoS One Date: 2011-07-25 Impact factor: 3.240
Authors: Oliver Rudolf Herber; Caroline Bradbury-Jones; Susanna Böling; Sarah Combes; Julian Hirt; Yvonne Koop; Ragnhild Nyhagen; Jessica D Veldhuizen; Julie Taylor Journal: BMC Med Res Methodol Date: 2020-05-18 Impact factor: 4.615