Publication bias in medical informatics evaluation research: is it an issue or not?

The phenomenon of publication bias has probably existed since results of scientific research are being published. Positive and/or statistically significant results seem more likely to be published than negative and/or insignificant results. However, it is unclear if there is a remarkable impact of publication bias in medical informatics evaluation literature and how aware researchers are of its effect. We conducted a small-scale study in order to find out what the ratio of papers describing positive results vs. negative results is, tried to find enough studies to a certain subject to carry out a meta-analysis and assess publication bias by statistical methods, and finally examined reviews and meta-analyses for their results and their quality. A random sample of 86 studies showed a remarkably high percentage of descriptions of positive results (69.8%). 19 (36.6%) of the analyzed 54 reviews and meta-analyses came to a positive conclusion with regard to the overall effect of the analyzed system, 32 (62.5%) were inconclusive, and only one review came to a negative conclusion. Quantitative assessment of publication bias for health informatics studies was found difficult due to the low number of comparable studies. Although there is no clear evidence for a great impact of publication bias in medical informatics evaluation literature, further research should carried out.