Homozygosity in a population of variable size and mutation rate.

A formula is obtained for the probability that two genes at a single locus, sampled at random from a population at time t, are of particular types. The model assumed is a diffusion approximation to a neutral Wright-Fisher model in which mutation is not necessarily symmetric and the population size is a function of time. It is shown that for symmetric mutation in a population undergoing a step-function type bottleneck, homozygosity increases with decreasing population size. A formula is given for the distribution of the number of segregating sites occurring in two randomly sampled sequences of completely linked sites, with general mutation at a site and identical mutation structure between sites. We give similar results for a population of fixed size but for which the mutation rate is a function of time, and not necessarily symmetric. We confirm the intuitively clear effect that increasing the mutation rate decreases homozygosity.