Hidden markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system

MPEG-4 standard allows composition of natural or synthetic video with facia l animation. Based on this standard, an animated face can be inserted into natural or synthetic video to create new virtual working environments such as virtual meetings or virtual collaborative environments. For these applic ations, audio-to-visual conversion techniques can be used to generate a tal king face that is synchronized with the voice. In this paper, we address au dio-to-visual conversion problems by introducing a novel Hidden Markov Mode l Inversion (HMMI) method. In training audio-visual HMMs, the model paramet ers {lambda (av)} can be chosen to optimize some criterion such as maximum likelihood. In inversion of audio-visual HMMs, visual parameters that optim ize some criterion can be found based on given speech and model parameters {lambda (av)}. By using the proposed HMMI technique, an animated talking fa ce can be synchronized with audio and can be driven realistically. The HMMI technique combined with MPEG-4 standard to create a virtual conference sys tem, named VIRTUAL-FACE, is introduced to show the role of HMMI for applica tions of MPEG-4 facial animation.