Experiments demonstrate a dramatic decrease in polarization-instability thr
eshold as an optical pulse is tuned near the short-wavelength edge of the p
hotonic bandgap formed by a fiber Bragg grating. These enhanced nonlinear i
nteractions and birefringent effects are modeled with coupled-mode numerica
l simulations. Nonlinearities are shown to increase much more rapidly than
the effective birefringence as the pulse wavelength approaches the bandgap
edge. (C) 2000 Optical Society of America.