Translational medicine and reliability of single-nucleotide polymorphism studies: can we believe in SNP reports or not?

BACKGROUND: The number of genetic association studies is increasing exponentially. Nonetheless, genetic association reports are prone to potential biases which may influence the reported outcome. AIM: We hypothesized that positive outcome for a determined polymorphism might be over-reported across genetic association studies analysing a small number of polymorphisms, when compared to studies analysing the same polymorphism together with a high number of other polymorphisms.
METHODS: We systematically reviewed published reports on the association of glutathione s-transferase (GST) single-nucleotide polymorphisms (SNPs) and cancer outcome. RESULT: We identified 79 eligible trials. Most of the studies examined the GSTM1, theGSTP1 Ile105Val mutation, and GSTT1polymorphisms (n = 54, 57 and 46, respectively). Studies analysing one to three polymorphisms (n = 39) were significantly more likely to present positive outcomes, compared to studies examining more than 3 polymorphisms (n=40) p = 0.004; this was particularly evident for studies analysing the GSTM1polymorphism (p =0.001). We found no significant associations between journal impact factor, number of citations, and probability of publishing positive studies or studies with 1-3 polymorphisms examined.
CONCLUSIONS: We propose a new subtype of publication bias in genetic association studies. Positive results for genetic association studies analysing a small number of polymorphisms (n = 1-3) should be evaluated extremely cautiously, because a very large number of such studies are inconclusive and statistically under-powered. Indeed, publication of misleading reports may affect harmfully medical decision-making and use of resources, both in clinical and pharmacological development setting.

Introduction

Genetic association studies investigate the relationship between gene polymorphisms and risk of disease or treatment outcome. Furthermore, due to advances in molecular targeted treatment technologies and the continual expansion of translational research, genetic association studies play a key role in the development of new therapeutical targets. For these reasons the number of genetic association studies is increasing rapidly and this trend is expected to accelerate due to the availability of mapped single-nucleotide polymorphisms (SNP) in the human genome and advances in genotyping technologies 1.

However, despite the number of genetic association studies being expected to grow exponentially over the next decade, no clear criteria are available to assess the credibility of these reports. Are all statistical significant medical reports, on SNP studies, reliable enough to drive firm conclusions and trig clinical/therapeutical applications?

Positive-outcome (also known as “publication”) bias refers to the greater likelihood of a study with positive results to be published compared with studies with negative results 2,3. Publication bias (“false-positive” reporting) is a particular threat to the credibility of the literature, including genetic association studies, since it may affect decision-making both in clinical and pharmacological development settings.

Biologists, researchers and physicians are actually called to deal with manuscripts of translational medicine research in their daily life. However, no parameters are actually available to orient them in a correct interpretation of potential misleading sources of literature-bias.

Based on the over mentioned reflections and considering the following three facts: 1) reviewer's and editor's decision about publication of manuscripts are influenced by positive findings 2,3,4; 2) positive studies are more possible to be published in journals with higher impact factor (IF) 3,5 and may be cited more often than negative studies 6,7; 3) null papers are typically given low publication priority scores and may not be accepted for publication 2, we hypothesized that the pressure for publication among authors and the fierce competition for acceptance in leading journals 3,4 may lead authors firstly, to perform studies with few polymorphisms, which are less expensive, need less time to complete and secondly, to submit for publication only those studies with positive outcome.

Is it the case? If yes, what about the impact of this phenomenon on medical literature? How positive compared to negative reports correlate with publication differences in impact factor journals or citation frequency?

In our study, we thereafter tested the hypothesis that a positive outcome for a determined polymorphism might be over reported across genetic association studies analysing a small number of polymorphisms, when compared to studies analysing the same polymorphism together with a high number of other polymorphisms. We also tried to assess any differences in journal impact factor or citation frequency among positive versus negative reports.

Due to the high number of published reports on the association of GST polymorphisms and cancer outcome (mainly colorectal, breast, and lung malignancy) SNPs for GST polymorphisms were used as a substrate for analyses.

Materials and Methods

Search strategy and eligibility criteria

We electronically searched the PubMed medical literature database and ISI Web of Science from inception to June 1, 2009, without language restrictions, using the following keyword combinations: (glutathione s-transferase OR GST OR GSTT1 OR GSTM1 OR GSTP1) AND (polymorphism OR polymorphisms OR SNP OR mutation) AND (cancer OR malign* OR carcinoma OR tumor OR tumour). The electronic search was supplemented by a manual review of the references of included studies.

The studies selected for our analysis had to meet the following criteria: (a) investigate the association between at least of one GST SNP: and cancer outcome; and (b) include only patients with solid tumours.

We excluded case-control studies that examined the role of GST polymorphisms on cancer risk, studies that included patients with hematologic malignancies and studies that investigated the role of GST polymorphisms on pharmacokinetics of specific drugs.

Two investigators independently reviewed all potentially relevant articles to determine whether an article met the inclusion criteria, and disagreement was resolved by discussion between the investigators.

Data extraction

We abstracted the following information from eligible trials: authors' name, year of publication, country of origin, type of cancer, sample size, number of polymorphisms tested and results of the study.

Studies were divided into two categories based on the results reported: positive or negative study. Since there is no standardized definition of positive results 8, the following definitions for positive and negative studies were used in our study:

A study was defined as “positive” if it reported any statistical significant difference for any of the GST polymorphisms for at least one of the following outcome measures: overall survival or disease recurrence or response to treatment. In the case of lack of a clear definition, or threshold, for statistical significant difference, we defined “significance” as the presence of a P-value of <0.05 or another effect metric with 95% confidence interval (C.I.) that fell entirely on one side of the null. A study was defined as negative if there was no statistically significant difference detected between a determined GST polymorphisms and any of these outcomes.

Regarding number of polymorphisms examined, eligible studies were divided into two groups: those which examined the association between 1-3 polymorphisms and cancer outcome and those which examined the association of more than 3 polymorphisms.

The Impact Factor (IF) of each journal was extracted from Journal Citation Reports (Institute for Scientific Information, JCR-ISI) 9. When a journal was not included in the citation index, we set 0 as IF.

The number of citations was obtained though the Science Citation Index 10. For each published article, all citations of that article from publication to the time of the search were identified. The number of citations per year from the year of publication to the study period was calculated for each article (citations per year).

Statistical analysis

Associations were tested using the chi-square statistic or Fisher's exact test with significance set at P < 0.05. The null hypothesis is that there is no difference in the proportions of positive and negative studies analysing a determined polymorphisms between the studies examining it within 1-3 polymorphisms versus studies examining the same polymorphism among more than 3 polymorphism.

Since the distribution of IF and citation frequency were not normal (Shapiro-Wilk test < 0.05) 11, we used nonparametric tests (Mann-Whitney tests) 12 to study the difference in IF and in frequency of citations per year between groups.

To better examine the possibility of a bias for positive results in studies examined 1-3 polymorphisms, logistic regression analysis, with adjustment for sample size, was used to calculate the odds ratio (OR) of reporting positive results in 2 study groups.

All statistical analyses were done using the SPSS software (SPSS Inc., Chicago, IL, USA, version 11.5). All tests were two-sided with a significance level of 0.05.

Results

Description of studies

A total of 4695 studies were identified from the combined searches. We scanned titles and abstracts for mention of GST polymorphisms associated with cancer outcome in either the title or the abstract. We retrieved 121 potentially eligible articles in full text [Figure 1].

Figure 1

Flowchart diagram of study selection.

A total of 79 articles that fulfilled the inclusion criteria were found 13-91. A total of 10 GST polymorphisms were analysed: GSTM1, GSTP1 Ile105Val, GSTP1 Ala114Val, GSTP1 Thr110Ser, GSTP1 Asp147Tyr, GSTT1, GSTM3, GSTA1, GSTO1, GSTO2. Of those, 39 examined 1-3 polymorphisms 13,14,16,19,20,23,24,26-28,31,36,37,39-41,45,46,48,50,53,56,60-62,64,65,68,69,72-74,76-79,81,89,91 and 40 more than 3 polymorphisms 15,17,18,21,22,25,29,30,32-35,38,42-44,47,49,51,52,54,55,57-59,63,66,67,70,71,75,80,82-88,90. Fifty four studies examined the GSTM1 polymorphism 13,14,16,17,19-27,30-33,35,37,38,43-46,48,49,51-63,65,68,69,70,72-75,77,78,82,83,85-87, 57 the GSTP1 Ile105Val polymorphism 13-16,18,21,22,26,28-30,32-49,51-55,57-61,64,66,67,71,74,76-85,87-90, 46 the GSTT1 polymorphism 13,14,16,17,19, 21-23,25,27,32,33,35,37,38,43-46,48-63,65,70,72,74,75,77,78,82,83,85,87. Only a small number of reports were available for other polymorphisms: GSTP1 Ala114Val (n=7) 15,32,40,42,66,80,85, GSTP1 Thr110Ser (n=1) 15, GSTP1 Asp147Tyr (n=1) 15, GSTM3 (n=2) 26,63, GSTA1 (n=1) 91, GSTO1 (n=1) 38, GSTO2 (n=1) 44.

Single studies characteristics for each of the 79 eligible studies are reported in the appendix Table, while general characteristics for the eligible studies are summarized in Table 1.

Table 1

Characteristics of eligible genetic association studies

Characteristic	No of studies (%)
Country of origin
USA	21 (26.5)
United Kingdom	10 (13)
Germany	6 (8)
India	5 (6)
South Korea	5 (6)
Other	32 (40.5)
Type of cancer
Breast	15 (19)
Colorectal	14 (18)
Lung	12 (15)
Ovarian	10 (13)
Other	28 (35)
Sample size
< 50 patients	6 (7.5)
50-150 patients	33 (42)
150-250 patients	18 (23)
250-500 patients	17 (21.5)
500-1000 patients	2 (2.5)
> 1000 patients	3 (3.5)
No of polymorphisms examined
1-3 polymorphisms	39 (49)
> 3 polymorphisms	40 (51)
Type of GST examined
GSTM1 present/null	54 (68)
GSTT1 present/null	46 (58)
GSTP1 Ile105Val	57 (72)
GSTP1 Ala114Val	7 (9)
GSTM3 A*/A* or A*/B* or B*/B*	2 (2.5)
GSTA1 A*/A* or A*/B* or B*/B*	1 (1)
GSTP1 Thr110Ser	1 (1)
GSTP1 Asp147Tyr	1 (1)
GSTO1 Ala140Asp	1 (1)
GSTO2 Asn142Asp	1 (1)

Association between the outcome of studies and number of polymorphisms tested

When a given polymorphism was analysed, studies reporting 1-3 polymorphisms were significantly more likely to present positive outcomes (n= 29; 74%) compared to studies evaluating the polymorphism across more than 3 polymorphisms (n= 17; 42.5%) (P-value = 0.004); this was particularly evident for studies analysing GSTM1 polymorphism (n= 13 vs. 2, P-value = 0.001), but it does not reach statistical significant differences for studies analysing GSTT1 and GSTP1 polymorphisms (P-value = 0.685 and 0.147 respectively) [Table 2].

Table 2

Outcome of eligible studies (positive-negative) according to number of polymorphisms tested

Polymorphisms	No of studies (%)		P-value
	Positive outcome (%)	Negative outcome (%)
any GST analysed
1-3 polymorphisms tested	29 (74)	10 (26)	0.004
> 3 polymorphisms tested	17 (42.5)	23 (57.5)
GSTM1 present/null
1-3 polymorphisms tested	13 (48)	14 (52)	0.001
> 3 polymorphisms tested	2 (7)	25 (93)
GSTT1 present/null
1-3 polymorphisms tested	4 (19)	17 (81)	0.685
> 3 polymorphisms tested	6 (24)	19 (76)
GSTP1 Ile105Val
1-3 polymorphisms tested> 3 polymorphisms tested	11 (46)9 (27)	13 (54)24 (73)	0.147
GSTP1 Ala114Val
1-3 polymorphisms tested	1 (100)	0 (0)	0.286
> 3 polymorphisms tested	1 (17)	5 (83)

Logistic regression analysis for studies examined any GST polymorphism revealed that the OR for positive outcome, when comparing studies with 1-3 polymorphisms tested to studies with more than 3 polymorphisms tested, was 3.906 (95% CI, 1.506 to 10.204, P-value = 0.005) after adjustment for sample size.

Association of outcome of studies, IF and citation frequency

There were no significant associations between the impact factor (range: 0.0 - 17.157) and positive studies or studies (P-value = 0.415) with 1-3 polymorphisms examined (P-value = 0.341) [Table 3].

Table 3

Impact factor and Citations per Year in studies regarding outcome and number of polymorphisms

	Impact Factor			Citations Per Year
	Mean (+/- SD)	Median (range)	P-value	Mean +/- SD	Median (range)	P-value
Any GST tested
Positive outcome	4.848 (4.170)	4.154 (0.0 - 17.157)		3.62 (4.24)	2.00 (0 - 17.33)
Negative outcome	5.099 (5.431)	3.508 (0.0 - 17.157)	0.415	2.40 (3.14)	1.17 (0 - 11.67)	0.185
1-3 polymorphisms tested	3.855 (2.743)	3.551 (0.0 - 14.933)		2.66 (3.22)	1.67 (0 -16.375)
> 3 polymorphisms tested	6.023 (5.879)	3.508 (0.0 - 17.157)	0.341	3.50 (4.31)	1.70 (0 - 17.33)	0.986
GSTM1 present/null
Positive outcome	3.260 (2.421)	2.919 (0.0 - 7.514)		2.89 (2.65)	2.00 (0 - 8.33)
Negative outcome	5.212 (5.335)	3.508 (0.0 - 17.157)	0.369	3.32 (4.42)	1.67 (0 - 17.33)	0.736
1-3 polymorphisms tested	3.777 (3.138)	2.970 (0.0 - 14.933)		2.87 (3.61)	1.67 (0 - 16.375)
> 3 polymorphisms tested	5.563 (5.889)	2.970 (0.0 - 17.157)	0.522	3.51 (4.37)	2.00 (0 - 17.33)	0.893
GSTT1 present/null
Positive outcome	5.224 (4.816)	3.883 (0.0 - 17.157)		2.39 (2.85)	1.14 (0 - 8.33)
Negative outcome	4.906 (5.138)	3.289 (0.0 - 17.157)	0.454	3.80 (4.54)	1.95 (0 - 17.33)	0.343
1-3 polymorphisms tested	3.876 (3.368)	3.069 (0.0 - 14.933)		3.17 (3.99)	1.67 (0 - 16.375)
> 3 polymorphisms tested	5.899 (5.986)	3.508 (0.0 - 17.157)	0.420	3.74 (4.48)	2.40 (0 - 17.33)	0.854
GSTP1 Ile105Val
Positive outcome	5.472 (3.966)	4.846 (1.932 - 17.157)		4.52 (5.44)	2.04 (0 - 17.33)
Negative outcome	5.107 (5.117)	3.508 (0.0 - 17.157)	0.162	2.71 (3.43)	1.38 (0 - 11.67)	0.238
1-3 polymorphisms tested	4.078 (3.010)	3.738 (0.0 -14.933)		3.05 (3.69)	1.78 (0 - 16.375)
> 3 polymorphisms tested	6.077 (5.532)	3.551 (0,843 - 17.157)	0.352	3.51 (4.64)	1.25 (0 - 17.33)	0.461

We failed to retrieve information about citation frequency from 8 studies 59,61,62,63,73,74,79,89 . The median citations per year for the remaining 71 studies was 1.67 (range: 0 - 17.33). Citations per year were not significantly associated with either the study outcome (P-value = 0.185) or the number of polymorphisms tested (P-value = 0.986) [Table 3].

Discussion

To our knowledge, this is the first study examined the potential role of number of polymorphisms tested on publication bias. We found that the positive outcome for a given polymorphism might be over reported across genetic association studies analysing a small number of polymorphisms (n = 1-3) when compared to studies analysing the same polymorphism within a higher number of polymorphisms. This was particularly evident for GSTM1 polymorphism. We, therefore, propose a new subtype of publication bias in genetic association studies regarding the number of polymorphisms tested. Thereafter positive results for genetic association studies analysing a small number of polymorphisms (n = 1-3) should be evaluated cautiously and considered at a lower level of evidence.

There are several possible explanations for this finding. First, the pressure for publication among authors and the competition for acceptance in journals are fierce 3,4 and may lead the authors to perform studies with few polymorphisms, which need less time to complete, and to submit for publication only those with positive outcome. Researchers are generally more enthusiastic about projects that have positive results and are more likely to complete them and submit them for publication 4. On the other hand, authors of studies with negative results are disappointed due to the feeling that null papers are typically given low publication priority scores and may not be accepted for publication 92. Moreover, it is possible that reviewer's and editor's decision about publication are influenced by positive findings 2,3,4.

An additional potential explanation for publication bias is that positive studies are more possible to be published in journals with higher IF 3,5. Nonetheless , when we investigated this theory by comparing the IF from journals published positive studies versus those published negative we found no significant association between IF and positive studies. We further tried to investigate the magnitude of the reported publication bias on scientific knowledge. Since there is no way to measure the impact of published articles on medical knowledge we may estimate their impact indirectly by calculating how frequently other authors cite them. Our results are encouraging since we found that studies with positive results or with 1-3 polymorphisms were not cited more frequently compared with studies with negative results or with > 3 polymorphisms. In literature, there are controversies concerning the citation frequency among positive or negative studies. Previous studies in other medical fields demonstrated that trials with a positive outcome were cited significantly more often than trials with a negative outcome 6,7 while other studies find no association 93.

We further documented that studies analysing a low number or a higher number of polymorphisms has the same probability to be published in impact factor journals.

Biologists, researchers and physicians are actually called to deal with translational medicine research manuscripts in their daily life. However, no parameters are actually available to orient them in a correct interpretation of potential misleading sources of literature-bias. The existence of such type of bias in genetic association studies might lead to incorrect conclusions about the usefulness of certain polymorphisms as prognostic genetic markers. It may also have direct impact in medical research by guiding researchers and funding sources in investigating insignificant genes. It is, therefore, extremely important to minimize this bias in medical literature since it may lead to severe decision-making consequences both in clinical and pharmacological development settings. We, therefore, propose that researchers should perform studies that examine many polymorphisms; however, for researchers who investigate one or a few SNPs, they should publish their study regardless of the outcome. It should be emphasized, however, that the transition from single SNP studies to genome-wide association studies (GWAS)--of cohorts sufficiently large in size so as to guarantee ample statistical power--represents a new era in human genetics, which has now arrived and offers an opportunity to overcome biases related to under-powered SNP studies on cohorts that are too small. There are several drawbacks in the study which should be discussed. First, one can oppose that there is problem in generalizing these data to all genetic association studies. This is correct. Anyhow, since our study outcomes were pre-specified and include all available evidence (79 studies) in this topic, our data are extremely possible to be solid enough to support our hypotheses. Confirmatory studies on polymorphisms of other genes are anyhow needed to depict the strength of our hypothesis. Second, one could argue that the observed higher rate of positive outcome in studies with small numbers of polymorphisms might reflect the investigation of SNPs that have been found to be “positive” in previous studies. Moreover, because several thresholds for the definition of statistical significance on genetic association studies have been proposed 94 whereas no consensus has been reached, there is the risk that some of these studies could be misclassified as “positive.

Conclusion: publication bias due to the number of polymorphism tested is a potential threat in medical literature. Positive results for genetic association studies analysing a small number of polymorphisms (n = 1-3) should be evaluated cautiously and considered at a lower level of evidence. Biologists, researchers and physicians dealing with translational medicine research should be aware of this potential threat for “false-positive” reports.