FACTORIAL HIDDEN MARKOV-MODELS

Hidden Markov models (HMMs) have proven to be one of the most widely u sed tools for learning probabilistic models of time series data. In an HMM, information about the past is conveyed through a single discrete variable-the hidden state. We discuss a generalization of HMMs in whi ch this state is factor-ed into multiple state variables and is theref ore represented in a distributed manner. We describe an exact algorith m far inferring the posterior probabilities of the hidden state variab les given the observations, and relate it to the forward-backward algo rithm for HMMs and to algorithms for more general graphical models. Do e to the combinatorial nature of the hidden state representation, this exact algorithm is intractable. As in other intractable systems, appr oximate inference can be carried out using Gibbs sampling or Variation al methods. Within the variational framework, we present a structured approximation in which the the state variables are decoupled, yielding a tractable algorithm for learning the parameters of the model. Empir ical comparisons suggest that these approximations are efficient and p rovide accurate alternatives to the exact methods. Finally, we use the structured approximation to model Bach's chorales and show that facto rial HMMs can capture statistical structure in this data set which an unconstrained HMM cannot.