We investigate the XY model with ferromagnetic nearest and antiferroma
gnetic next-to-nearest neighbours couplings (ANNNXY model). We formula
te the study of both phases of the model in terms of different Coulomb
gas models. The non-frustrated ferromagnetic phase is thus transforme
d into a usual Coulomb gas with one species, whereas the frustrated ph
ase is translated into a Coulomb gas with three species interacting wi
th each other in an anisotropic lattice. We then generalize the Koster
litz-Thouless renormalization group equations for both phases by treat
ing the Ising and Kosterlitz-Thouless order parameters in an independe
nt way. This enables us to discuss the nature of the transition (Ising
and (or) Kosterlitz-Thouless) in the frustrated phase.