This work presents the acoustic and visual-based tracking system functionin
g at the Harvard Intelligent Multi-Media Environments Laboratory (HIMMEL).
The environment is populated with a number of microphones and steerable vid
eo cameras. Acoustic source localization, video-based face tracking and pos
e estimation, and multi-channel speech enhancement methods are applied in c
ombination to detect and track individuals in a practical environment while
also providing an improved audio signal to accompany the video stream. The
video portion of the system tracks talkers by utilizing source motion, con
tour geometry, color data, and simple facial features. Decisions involving
which camera to use are based on an estimate of the head's gazing angle. Th
is head pose estimation is achieved using a very general head model which e
mploys hairline features and a learned network classification procedure. Fi
nally, a beamforming and postfiltering microphone array technique is used t
o create an enhanced speech waveform to accompany the recorded video signal
. The system presented in this paper is robust to both visual clutter (e.g.
ovals in the scene of interest which are not faces) and audible noise (e.g
. reverberations and background noise).