Determining epistatic selection in admixed populations.

When two diverging species begin hybridizing, selection against hybridization is likely driven not by single substitutions, but by interactions between incompatible mutations. To identify these incompatibilities in natural populations, researchers examine the extent of nonrandom associations between ancestry at physically unlinked loci in admixed populations. In this approach, which we call 'AD scans', locus pairs with significantly positive 'ancestry disequilibrium' (AD, i.e. locus pairs that positively covary by ancestry) represent incompatible alleles. Past research has uniformly revealed an excess of locus pairs with significantly positive AD, suggesting that dozens to hundreds of incompatibilities separate species. With forward simulations, we show that many realistic demographic scenarios, including recent and/or ongoing hybridization, generate a bias towards positive ancestry disequilibrium. We suggest steps that researchers can take to avoid pitfalls in interpreting AD scans, and present a novel measure of AD, which minimizes but does not fully eliminate bias in the AD distribution. We also show, by simulation, that the tail of the AD distribution is enriched for true incompatibilities. To illustrate the potential power and appropriate caution in interpretation of AD scans, we reanalyse previously published data from two admixed populations of Xiphophorus fishes. Our results imply that the prevalence of positive LD in admixed populations does not in itself support the idea that two-locus incompatibilities are widespread, but the co-enrichment of top AD hits across the two Xiphophorus populations supports the idea that AD scans can identify candidate interspecific incompatibilities.