Estimating effective population size or mutation rate using the frequencies of mutations of various classes in a sample of DNA sequences.

Mutations resulting in segregating sites of a sample of DNA sequences can be classified by size and type and the frequencies of mutations of different sizes and types can be inferred from the sample. A framework for estimating the essential parameter theta = 4Nu utilizing the frequencies of mutations of various sizes and types is developed in this paper, where N is the effective size of a population and mu is mutation rate per sequence per generation. The framework is a combination of coalescent theory, general linear model and Monte-Carlo integration, which leads to two new estimators theta xi and theta eta as well as a general Watterson's estimator theta K and a general Tajima's estimator theta tau. The greatest strength of the framework is that it can be used under a variety of population models. The properties of the framework and the four estimators theta K, theta tau, theta xi and theta eta are investigated under three important population models: the neutral Wright-Fisher model, the neutral model with recombination and the neutral Wright's finite-islands model. Under all these models, it is shown that theta xi is the best estimator among the four even when recombination rate or migration rate has to be estimated. Under the neutral Wright-Fisher model, it is shown that the new estimator theta xi has a variance close to a lower bound of variances of all unbiased estimators of theta which suggests that theta xi is a very efficient estimator.