Mott physics and topological phase transition in correlated dirac fermions.

We investigate the interplay between the strong correlation and the spin-orbit coupling in the Kane-Mele-Hubbard model and obtain the qualitative phase diagram via the variational cluster approach. We identify, through an increase of the Hubbard U, the transition from the topological band insulator to either the spin liquid phase or the easy-plane antiferromagnetic insulating phase, depending on the strength of the spin-orbit coupling. A nontrivial evolution of the bulk bands in the topological quantum phase transition is also demonstrated.