Does the non-randomized controlled study have a place in the systematic review? A pilot study.

BACKGROUND: A major issue in any systematic review is deciding which trials or studies to include and which to exclude. The Cochrane Collaboration and similar respected organizations have traditionally viewed the randomized trial (RCT) as the only acceptable evidence on treatment outcome. However, many systematic reviews are indeterminate because they include insufficient RCTs whilst they reject large numbers of non-randomized controlled studies. This is particularly true in forensic mental health, a domain where RCT methodology can be problematic. Systematic reviews could become more informative if reviewers knew when, and under what circumstances, non-randomized designs are acceptable for inclusion alongside RCTs.
METHOD: This pilot study explores whether good-quality, controlled, non-randomized studies can be reliable surrogates for RCTs. We examined two published reviews from the Cochrane Schizophrenia Group. We compared outcomes between (a) randomized trials (that had been included) and non-randomized studies (that had been excluded), and (b) between high- and low-quality studies using an established quality checklist.
RESULTS: In the first review, effect scores were similar for randomized and non-randomized studies and relatively insensitive to study quality. In the second review, the treatment effect was considerably lower for the RCT group - here, however, studies of high-quality showed much smaller effect scores than those of low-quality on two separate outcomes.
CONCLUSIONS: Non-randomized controlled studies of high quality can produce outcomes that approximate to those found in RCTs. Trial quality may have a greater impact on treatment effect size than randomization alone, suggesting that randomization should not be seen as a reliable proxy for overall quality. The problems and issues still to be resolved are discussed with recommendations for future research.