MOTION-BASED ANALYSIS AND SEGMENTATION OF IMAGE SEQUENCES USING 3-D SCENE MODELS

In this paper we present an algorithm for automatic extraction and tra cking of multiple objects from a video sequence. Our approach is model -based in the sense that we first use a robust structure-from-motion a lgorithm to identify multiple objects and to recover initial 3-D-shape models. Then, these models are used to identify and track the objects over multiple frames of the video sequence. The procedure starts with recovering a dense depth map of the scene using two frames at the beg inning of the sequence, and representing the scene as a 3-D wire frame computed from the depth map. Texture extracted from the video frames is mapped onto the model. Once the initial models are available we use a linear and low-complexity algorithm to recover the motion parameter s and scene structure of the objects for the subsequent frames. Combin ing the new estimates of depth and the initially computed 3-D models i nto an unstructured set of 3-D points with associated color informatio n, we obtain updates of the 3-D scene description for each additional frame. We show that the usage of a 3-D scene model is suitable to anal yze complex scenes with several objects. In our experimental results, we apply the approach presented in this paper to the problem of video sequence segmentation, object tracking, and video object plane (VOP) g eneration. We separate the video sequences into different layers of de pth and combine the information from multiple frames to a compact and complete description of these layers. (C) 1998 Elsevier Science B.V. A ll rights reserved.