Antiferromagnetic quantum spin systems can exhibit a transition between collinear and spiral ground states,
driven by frustration. Classically this is a smooth crossover and the crossover point is termed a Lifshitz point.
Quantum fluctuations change the nature of the transition. In particular, it has been argued previously that in the two-dimensional (2D) case a spin liquid (SL) state is developed in the vicinity of the Lifshitz point, termed a Lifshitz SL. In the present work, using a field theory approach, we solve the Lifshitz quantum phase transition problem for the 2D frustrated XY model. Specifically, we show that, unlike the SU (2) symmetric Lifshitz case, in the XY model, the SL exists only at the critical point. At zero temperature we calculate nonuniversal critical exponents in the Néel and in the spin spiral state and relate these to properties of the SL. We also solve the transition problem at a finite temperature and discuss the role of topological excitations.
