Topological Mott insulators.

We consider extended Hubbard models with repulsive interactions on a honeycomb lattice, and the transitions from the semimetal to Mott insulating phases at half-filling. Because of the frustrated nature of the second-neighbor interactions, topological Mott phases displaying the quantum Hall and the quantum spin Hall effects are found for spinless and spin fermion models, respectively. The mean-field phase diagram is presented and the fluctuations are treated within the random phase approximation. Renormalization group analysis shows that these states can be favored over the topologically trivial Mott insulating states.