Development of a scoring system to judge the scientific quality of information from case-control and cohort studies of nutrition and disease.

A scoring system was developed to help judge the scientific quality of observational epidemiologic studies linking diet with risk of cancer. The scoring system was developed from key headings used in developing research protocols and included questions under headings: three for case-control studies (dietary assessment, recruitment of subjects, and analysis) and four for cohort studies (dietary assessment, definition of cohort, ascertainment, and analysis). Points were awarded for questions in each section, and a total score was derived. Interobserver variation was assessed for five case-controls and five cohort studies for 13 observers: 1 observer repeated the assessment of each paper. Absolute scores and ranking within observer were assessed. There was good agreement between observers in the ranking of studies. Papers that scored higher presented sufficient detail to enable the questions in the scoring system to be answered more easily. For some studies, the information required was either not collected or, if it was collected, not presented. In either case, the frequent lack of information available to judge papers raises questions about the editorial policy and review process of journals publishing dietary studies as much as it does about the scoring system. Applying the scoring system to a review of meat and cancer risk suggested that, taking the score into account, from what seemed like a large literature, there were relatively few studies that scored well (defined as a score > 65%), but these studies tended to provide more consistent information.