ON THE CONVEXITY OF THE MOMENT MAPPING FOR UNITARY HIGHEST WEIGHT REPRESENTATIONS

To each continuous unitary representation of a Lie group G on a Hilber t space H we associate a moment map from the projective space of smoot h vectors to the dual g of the Lie algebra of G. For unitary highest weight representations we obtain a characterization of those highest w eights for which the closure of the image of this map is convex, i.e., equal to the convex hull of the highest weight orbit. This result gen eralizes the corresponding result for representations of compact group s and holomorphic discrete series representations. (C) 1995 Academic P ress, Inc.