Bose-Einstein condensation in a frustrated triangular optical lattice

The recent experimental condensation of ultracold atoms in a triangular optical lattice with negative effective tunneling energies paves the way to study frustrated systems in a controlled environment. Here, we explore the critical behavior of the chiral phase transition in such a frustrated lattice in three dimensions. We represent the low-energy action of the lattice system as a two-component Bose gas corresponding to the two minima of the dispersion. The contact repulsion between the bosons separates into intra- and inter-component interactions, referred to as V₀ and V₁₂, respectively. We first employ a Huang-Yang-Luttinger approximation of the free energy. For V₁₂/V₀ = 2, which corresponds to the bare interaction, this approach suggests a first order phase transition, at which both the U(1) symmetry of condensation and the mathbb{Z}₂ symmetry of the emergent chiral order are broken simultaneously. Furthermore, we perform a renormalization group calculation at one-loop order. We demonstrate that the coupling regime 0<V₁₂/V_0leq1 shares the critical behavior of the Heisenberg fixed point at V₁₂/V₀=1. For V₁₂/V₀>1 we show that V₀ flows to a negative value, while V₁₂ increases and remains positive. This results in a breakdown of the effective quartic field theory due to a cubic anisotropy, and again suggests a discontinuous phase transition.