Diocotron instabilities form an important class of E X B shear flow instabi
lities which occur in nonneutral plasmas. The case of a single-species plas
ma confined in a cylindrical Penning trap, with an axisymmetric, hollow (no
nmonotonic) density profile is studied. According to the standard linear th
eory, the m = 1, k(z) = 0 diocotron mode is always stable. On the other han
d, experiments by Driscoll [Phys. Rev. Lett. 64, 645 (1990)] show a robust
exponential growth of m = 1 diocotron perturbations in hollow density profi
les. The apparent contradiction between these experimental results and line
ar theory has been an outstanding problem in the theory of nonneutral plasm
as. A new instability mechanism due to the radial variation of the equilibr
ium plasma length is proposed in this paper. This mechanism involves the co
mpression of the plasma parallel to the magnetic field and implies the cons
ervation of the line integrated density. The predicted growth rate, frequen
cy, and mode structure are in reasonable agreement with the experiment. The
effect of a linear perturbation of the plasma length is also shown to give
instability with a comparable growth rate. The conservation of the line in
tegrated density in the plasma is analogous to the conservation of the pote
ntial vorticity in the shallow water equations used in geophysical fluid dy
namics. In particular, there is an analog of Rossby waves in nonneutral pla
smas. (C) 1999 American Institute of Physics. [S1070-664X(99)01310-5].