Ira Todd Cohen1, Kantilal Patel1. 1. Associate Professor of Anesthesiology and Pediatrics Children's National Medical Center George Washington University, Washington, DC.
Abstract
BACKGROUND: Presentation of scientific abstracts is an important function of medical specialty and subspecialty societies. Selection is typically performed by the means of a peer review process. The validity and reliability of the peer review is under examination. The purpose of this study was to determine the interrater reliability of abstract assessment by a subspecialty organization at their Annual Winter meeting. The subspecialty society was selected on the basis of representing the median number for membership and abstracts. METHODS: After institutional review board approval, data collection included number of abstracts submitted, abstract groupings, number of reviewers, assessment criteria, and rating scales. Interrater reliability was defined as kappa = N (PMS-EMS)/{N.PMS + (k-1) RMS+(N-1)(k-1) EMS; in which PMS, RMS and EMS are the mean square values for abstracts, reviewers, and error, respectively, N is the number of abstracts, and k is the number of evaluators. Resulting values may range from 0.0 (no agreement) to 1.0 (perfect agreement). RESULTS: Eleven reviewers, blinded to authors and institutions, rated 87abstracts divided into two randomly assigned groups. Abstracts were judged on six criteria and assigned a numerical score of 1 to 4, using a nominal scale. The average abstract rating in Group A was 3.12 (± 0.47) and in Group B was 2.99 (± 0.63). The Kappa statistic for Group A was 0.21 and for Group B was 0.39. For categorical data, these scores denote a "fair" level of agreement. CONCLUSIONS: A low level of interrater reliability was found among reviewers of abstracts submitted for presentation at an anesthesiology subspecialty society's annual meeting. This lack of evaluator agreement is similar to that found for abstract scoring by other medical subspecialties. The low kappa statistic appears to be secondary to use of a narrowly defined nominal scale, which reduced accuracy and variability.
RCT Entities:
BACKGROUND: Presentation of scientific abstracts is an important function of medical specialty and subspecialty societies. Selection is typically performed by the means of a peer review process. The validity and reliability of the peer review is under examination. The purpose of this study was to determine the interrater reliability of abstract assessment by a subspecialty organization at their Annual Winter meeting. The subspecialty society was selected on the basis of representing the median number for membership and abstracts. METHODS: After institutional review board approval, data collection included number of abstracts submitted, abstract groupings, number of reviewers, assessment criteria, and rating scales. Interrater reliability was defined as kappa = N (PMS-EMS)/{N.PMS + (k-1) RMS+(N-1)(k-1) EMS; in which PMS, RMS and EMS are the mean square values for abstracts, reviewers, and error, respectively, N is the number of abstracts, and k is the number of evaluators. Resulting values may range from 0.0 (no agreement) to 1.0 (perfect agreement). RESULTS: Eleven reviewers, blinded to authors and institutions, rated 87abstracts divided into two randomly assigned groups. Abstracts were judged on six criteria and assigned a numerical score of 1 to 4, using a nominal scale. The average abstract rating in Group A was 3.12 (± 0.47) and in Group B was 2.99 (± 0.63). The Kappa statistic for Group A was 0.21 and for Group B was 0.39. For categorical data, these scores denote a "fair" level of agreement. CONCLUSIONS: A low level of interrater reliability was found among reviewers of abstracts submitted for presentation at an anesthesiology subspecialty society's annual meeting. This lack of evaluator agreement is similar to that found for abstract scoring by other medical subspecialties. The low kappa statistic appears to be secondary to use of a narrowly defined nominal scale, which reduced accuracy and variability.