Is homoplasy or lineage sorting the source of incongruent mtdna and nuclear gene trees in the stiff-tailed ducks (Nomonyx-Oxyura)?

We evaluated the potential effects of homoplasy, ancestral polymorphism, and hybridization as obstacles to resolving phylogenetic relationships within Nomonyx-Oxyura stiff-tailed ducks (Oxyurinae; subtribe Oxyurina). Mitochondrial DNA (mtDNA) control region sequences from 94 individuals supported monophyly of mtDNA haplotypes for each of the six species and provided no evidence of extant incomplete lineage sorting or inter-specific hybridization. The ruddy ducks (O. j. jamaicensis,O. j. andina, O. j. ferruginea) are each others' closest relatives, but the lack of shared haplotypes between O. j. jamaicensis and O. j. ferruginea suggests long-standing historical isolation. In contrast, O. j. andina shares haplotypes with O. j. jamaicensis and O. j. ferruginea, which supports Todd's (1979) and Fjeldså's (1986) hypothesis that O. j. andina is an intergrade or hybrid subspecies of O. j. jamaicensis and O. j. ferruginea. Control region data and a much larger data set composed of approximately 8800 base pairs of mitochondrial and nuclear sequence for each species indicate that the two New World species, O. vittata and O. jamaicensis, branch basally within Oxyura. A clade of three Old World species (O. australis, O. maccoa, O. leucocephala) is well supported, but different loci and also different characters within the mtDNA data support three different resolutions of the Old World clade, yielding an essentially unresolved trichotomy. Fundamentally different factors limited the resolution of the mtDNA and nuclear gene trees. Gene trees for most nuclear loci were unresolved due to slow rates of mutation and a lack of informative variation, whereas uncertain resolution of the mtDNA gene tree was due to homoplasy. Within the mtDNA, approximately equal numbers of characters supported each of three possible resolutions. Parametric and nonparametric bootstrap analyses suggest that resolution of the mtDNA tree based on ~4300 bp per taxon is uncertain but that complete mtDNA sequences would yield a fully resolved gene tree. A short internode separating O. leucocephala from (O. australis, O. maccoa) in the best mtDNA tree combined with long terminal branches and substantial rate variation among nucleotide sites allowed the small number of changes occurring on the internode to be obscured by homoplasy in a significant portion of simulated data sets. Although most nuclear loci were uninformative, two loci supported a resolution of the Old World clade (O. maccoa, O. leucocephala) that is incongruent with the best mtDNA tree. Thus, incongruence between nuclear and mtDNA trees may be due to random sorting of ancestral lineages during the short internode, homoplasy in the mtDNA data, or both. The Oxyura trichotomy represents a difficult though likely common problem in molecular systematics. Given a short internode, the mtDNA tree has a greater chance of being congruent with the history of speciation because its effective population size (N(e)) is one-quarter that of any nuclear locus, but its resolution is more likely to be obscured by homoplasy. In contrast, gene trees for more slowly evolving nuclear loci will be difficult to resolve due to a lack of substitutions during the internode, and when resolved are more likely to be incongruent with the species history due to the stochastic effects of lineage sorting. We suggest that researchers consider first whether independent gene trees are adequately resolved and then whether those trees are congruent with the species history. In the case of Oxyura, the answer to both questions may be no. Complete mtDNA sequences combined with data from a very large number of nuclear loci may be the only way to resolve such trichotomies.