CONTEXT: Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas. OBJECTIVE: To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs). DESIGN: Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale. MAIN OUTCOME MEASURES: Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies. RESULTS: Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent. CONCLUSIONS: Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.
CONTEXT: Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas. OBJECTIVE: To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs). DESIGN: Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale. MAIN OUTCOME MEASURES: Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies. RESULTS: Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent. CONCLUSIONS: Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.
Authors: Paul G Shekelle; Sally C Morton; Lara K Jungvig; Jay Udani; Myles Spar; Wenli Tu; Marika J Suttorp; Ian Coulter; Sydne J Newberry; Mary Hardy Journal: J Gen Intern Med Date: 2004-04 Impact factor: 5.128
Authors: Kevin R Kasten; Adam C Celio; Lauren Trakimas; Mark L Manwaring; Konstantinos Spaniolas Journal: Surg Endosc Date: 2017-07-19 Impact factor: 4.584
Authors: Joseph E Glass; Ashley M Hamilton; Byron J Powell; Brian E Perron; Randall T Brown; Mark A Ilgen Journal: Addiction Date: 2015-06-03 Impact factor: 6.526
Authors: Peter P Roy-Byrne; Michelle G Craske; Murray B Stein; Greer Sullivan; Alexander Bystritsky; Wayne Katon; Daniela Golinelli; Cathy D Sherbourne Journal: Arch Gen Psychiatry Date: 2005-03