SPEECHREADING USING PROBABILISTIC MODELS

We describe a robust method for locating and tracking lips in gray-lev el image sequences. Our approach learns patterns of shape variability from a training set which constrains the model during image search to only deform in ways similar to the training examples, Image search is guided by a learned gray-level model which is used to describe the lar ge appearance variability of lips, Such variability might be due to di fferent individuals, illumination, mouth opening, specularity, or visi bility of teeth and tongue, Visual speech features are recovered from the tracking results and represent both shape and intensity informatio n, We describe a speechreading (lip-reading) system, where the extract ed features are modeled by Gaussian distributions and their temporal d ependencies by hidden Markov models. Experimental results are presente d for locating lips, tracking lips, and speechreading. The database us ed consists of a broad variety of speakers and was recorded in a natur al environment with no special lighting or lip markers used, For a spe aker independent digit recognition task using visual information only, the system achieved an accuracy about equivalent to that of untrained humans. (C) 1997 Academic Press.