The quality of alcohol treatment research: an examination of influential controlled trials and development of a quality rating system.

BACKGROUND: The importance of evidence-based practice has stimulated interest in the methodology of clinical trials. Various weaknesses of evaluation research in the alcohol field have been indicated previously. This study set out to develop a comprehensive system for the assessment of the methodological quality of outcome research for treatment of alcohol misuse and to apply the system to well-known trials in the area.
METHODOLOGY: A sample of the most highly cited controlled trials of interventions for alcohol misuse was selected using the Science Citation Index. Thirty methodological criteria were formulated and a scoring system devised. Two raters applied this system to the sample of trials. Reliability testing was performed and used to refine the criteria.
RESULTS: Inter-rater reliability of the overall quality score was initially 0.85 and 0.92 after review and re-rating. Internal consistency was also high (0.87). Quality score correlated with year of publication. Certain areas of methodology were poorly addressed in the sample, including specification of main outcomes, documentation of recruitment and selection procedures, testing of blinding, analysis of withdrawals and reporting of results. DISCUSSION: The methodology of this sample of trials was frequently deficient in ways which might bias results or compromise generalizability. It is recommended that the system of quality assessment described here is used to evaluate existing research and to inform the design of future studies.