Detecting loci under selection in a hierarchically structured population.

Patterns of genetic diversity between populations are often used to detect loci under selection in genome scans. Indeed, loci involved in local adaptations should show high F(ST) values, whereas loci under balancing selection should rather show low F(ST) values. Most tests of selection based on F(ST) use a null distribution generated under a simple island model of population differentiation. Although this model has been shown to be robust, many species have a more complex genetic structure, with some populations sharing a recent ancestry or due to the presence of barriers to gene flow between different parts of a species range. In this paper, we propose the use of a hierarchical island model, in which demes exchange more migrants within groups than between groups, to generate the joint distribution of genetic diversity within and between populations. We show that tests not accounting for a hierarchical structure, when it exists, do generate a large excess of false positive loci, whereas the hierarchical island model is robust to uncertainties about the exact number of groups and demes per group in the system. Our approach also explicitly takes into account the mutational process, and does not just rely on allele frequencies, which is important for short tandem repeat (STR) data. An application to human and stickleback STR data sets reveals a much lower number of significant loci than previously obtained under a non-hierarchical model. The elimination of false positive loci from genome scans should allow us to better determine on which specific class of genes selection is operating.