OBJECTIVE: Despite the increasing dependence on systematic reviews to summarise the literature and to issue public health recommendations, the formal assessment of the reliability of conclusions emerging from systematic reviews has received little attention. The main goal of the present study was to evaluate whether two independent centres, in two continents, draw similar conclusions regarding the association of food, nutrition and physical activity and endometrial cancer, when provided with the same general instructions and with similar resources. DESIGN: The assessment of reproducibility concentrated on four main areas: (1) paper search and selection; (2) assignment of study design; (3) inclusion of papers; and (4) individual studies selected for meta-analysis and the summary risk estimate obtained. RESULTS: In total 310 relevant papers were identified, 166 (54 %) were included by both centres. Of the remaining 144 papers, 72 (50 %) were retrieved in the searches of one centre and not the other (54 in centre A, 18 in centre B) and 72 were retrieved in both searches but regarded as relevant by only one of the centres (52 in centre A, 20 in centre B). Of papers included by both centres, 80 % were allocated the same study design. Agreement for inclusion of cohort-type and case-control studies was about 63% compared with 50% or less for ecological and case series studies. The agreement for inclusion of 138 'key' papers was 87 %. Summary risk estimates from meta-analyses were similar. CONCLUSIONS: Transparency of process and explicit detailed procedures are necessary parts of a systematic review and crucial for the reader to interpret its findings.
OBJECTIVE: Despite the increasing dependence on systematic reviews to summarise the literature and to issue public health recommendations, the formal assessment of the reliability of conclusions emerging from systematic reviews has received little attention. The main goal of the present study was to evaluate whether two independent centres, in two continents, draw similar conclusions regarding the association of food, nutrition and physical activity and endometrial cancer, when provided with the same general instructions and with similar resources. DESIGN: The assessment of reproducibility concentrated on four main areas: (1) paper search and selection; (2) assignment of study design; (3) inclusion of papers; and (4) individual studies selected for meta-analysis and the summary risk estimate obtained. RESULTS: In total 310 relevant papers were identified, 166 (54 %) were included by both centres. Of the remaining 144 papers, 72 (50 %) were retrieved in the searches of one centre and not the other (54 in centre A, 18 in centre B) and 72 were retrieved in both searches but regarded as relevant by only one of the centres (52 in centre A, 20 in centre B). Of papers included by both centres, 80 % were allocated the same study design. Agreement for inclusion of cohort-type and case-control studies was about 63% compared with 50% or less for ecological and case series studies. The agreement for inclusion of 138 'key' papers was 87 %. Summary risk estimates from meta-analyses were similar. CONCLUSIONS: Transparency of process and explicit detailed procedures are necessary parts of a systematic review and crucial for the reader to interpret its findings.
Authors: Matthew J Page; David Moher; Fiona M Fidler; Julian P T Higgins; Sue E Brennan; Neal R Haddaway; Daniel G Hamilton; Raju Kanukula; Sathya Karunananthan; Lara J Maxwell; Steve McDonald; Shinichi Nakagawa; David Nunan; Peter Tugwell; Vivian A Welch; Joanne E McKenzie Journal: Syst Rev Date: 2021-04-16
Authors: Jeffrey Low; Joseph S Ross; Jessica D Ritchie; Cary P Gross; Richard Lehman; Haiqun Lin; Rongwei Fu; Lesley A Stewart; Harlan M Krumholz Journal: Syst Rev Date: 2017-02-15