We develop a model to treat coupling between guided modes in planar dielect
ric waveguides that have been textured in two dimensions with a thin surfac
e grating. The formulation is based on a general Green's-function technique
that self-consistently determines the field in the surface grating due to
the polarization there. With simplifying approximations, this formalism is
cast into a two-dimensional (2D) vector-coupled-mode theory that is more co
mputationally efficient, and that gives considerable insight into the natur
e of mode coupling in 2D textured structures. These models are applied, by
way of example, to illustrate some interesting properties of leaky and boun
d modes that are coupled together by 2D periodic texture. In particular we
discuss the complex photonic band structure describing the dispersion, life
times, and polarization properties of the resonant states associated with t
he textured waveguide. In our analysis we emphasize the fundamental differe
nces between coupling in 2D textured waveguides and infinite 2D photonic cr
ystals. We also show that the vector-coupled-mode theory agrees well with t
he self-consistent formulation.