Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias?

Although they usually claim a very thorough search to retrieve every pertinent work, most meta-analyses published in English restrict their search to papers which were also published in English. We reviewed all the meta-analyses printed from 1 January 1991 to 1 April 1993 in 8 medical journals published in English and selected those who stated linguistic restrictions for inclusion in the analysis. The computerized search methods used in these meta-analyses were then duplicated looking specifically for publications written in the excluded languages. Each meta-analysis was then redone with identical statistical tests to determine if its conclusions would have been different if the paper(s) absent only for linguistic reasons had been included. A total of 36 meta-analyses of which 28 had language restrictions were identified. The computer searches yielded 19 papers scientifically acceptable but excluded for linguistic reasons. Eleven of these articles were retained as having the potential to modify their corresponding 7 meta-analyses. One meta-analysis which concluded that selective decontamination of the digestive tract in intensive care units did not produce a significant change in mortality between treatment and control patients (OR 0.70; 95% CI 0.45-1.09) would have arrived at a different conclusion (OR 0.67; 95% CI 0.47-0.95) if a paper written in German in a Swiss journal had been included in the analysis. Our study demonstrates that, in at least one out of 36 consecutive meta-analyses the exclusion of papers for linguistic reasons produced results different from those which would have been obtained if this exclusion criteria had not been used.