Optical flow constraints on deformable models with applications to face tracking

Optical flow provides a constraint on the motion of a deformable model. We derive and solve a dynamic system incorporating flow as a hard constraint, producing a model-based least-squares optical flow solution. Our solution a lso ensures the constraint remains satisfied when combined with edge inform ation, which helps combat tracking error accumulation. Constraint enforceme nt can be relaxed using a Kalman filter, which permits controlled constrain t violations based on the noise present in the optical flow information, an d enables optical flow and edge information to be combined more robustly an d efficiently. We apply this framework to the estimation of face shape and motion using a 3D deformable face model. This model uses a small number of parameters to describe a rich variety of face shapes and facial expressions . We present experiments in extracting the shape and motion of a face from image sequences which validate the accuracy of the method. They also demons trate that our treatment of optical flow as a hard constraint, as well as o ur use of a Kalman filter to reconcile these constraints with the uncertain ty in the optical flow, are vital for improving the performance of our syst em.