Phylogenetic analysis under reticulate evolution.

The usual assumption that species have evolved from a common ancestor by a simple branching process--where each branch is genetically isolated--has been challenged by the observation of frequent hybridization between species in natural populations. In fact, most plant species are thought to have hybrid origins. This reticulate pattern of species evolution has posed problems in the definition of speciation and in phylogenetic reconstruction, especially when molecular data are used. As a result, hybridization has been largely treated as an evolutionary accident or statistical error in phylogenetic analysis. In this paper, I explicitly incorporate hybridization as an evolutionary occurrence and then conduct phylogenetic reconstruction. I first examine the reticulate evolution under a pure drift model, and then extend the theory to fit a mutation model. A least-squares method is developed for reconstructing a reticulate phylogeny using gene frequency data. The efficacy of the method under the pure drift model is verified via Monte Carlo simulations.