Eigen-texture method: Appearance compression and synthesis based on a 3D model

Image-based and model-based methods are two representative rendering method s for generating virtual images of objects from their real images. However, both methods still have several drawbacks when we attempt to apply them to mixed reality where we integrate virtual images with real background image s. To overcome these difficulties, we propose a new method, which we refer to as the Eigen-Texture method. The proposed method samples appearances of a real object under various illumination and viewing conditions, and compre sses them in the 2D coordinate system defined on the 3D model surface gener ated from a sequence of range images. The Eigen-Texture method is an exampl e of a view-dependent texturing approach which combines the advantages of i mage-based and model-based approaches: No reflectance analysis of the objec t surface is needed, while an accurate 3D geometric model facilitates integ ration with other scenes. This paper describes the method and reports on it s implementation.