The inconsistency of the progressive wave boundary conditions for the
quasi-bound modes in open leaky structures is shown and a new boundary
value formulation is developed. The results obtained are tested for l
eaky waveguides with delta function barriers for which a simple analyt
ical characteristic equation is available. The method is applied for t
he determination of eigenvalues of prism-loaded multilayer waveguides.
The influence of the prism coupler on the waveguide modes is investig
ated in detail for slab and four-layer waveguides. The method is compa
red with several approximate methods.