Mixtures of probabilistic principal component analyzers

Principal component analysis (PCA) is one of the most popular techniques fo r processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have b een proposed, an alternative paradigm is to capture data complexity by a co mbination of local linear PCA projections. However, conventional PCA does n ot correspond to a probability density, and so there is no unique way to co mbine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussia n latent variable model. This leads to a well-defined mixture model for pro babilistic principal component analyzers, whose parameters can be determine d using an expectation-maximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimen sionality reduction, and we demonstrate its application to image compressio n and handwritten digit recognition.