Mixing synthetic and video images of an outdoor urban environment

Mixing video and computer-generated images is a new and promising area of r esearch for enhancing reality. It can be used in all the situations when a complete simulation would not be easy to implement. Past work on the subjec t has relied for a large part on human intervention at key moments of the c omposition. In this paper, we show that if enough geometric information abo ut the environment is available, then efficient tools developed in the comp uter vision literature can be used to build a highly automated augmented re ality loop. We focus on outdoor urban environments and present an applicati on for the visual assessment of a new lighting project of the bridges of Pa ris. We present a fully augmented 300-image sequence of a specific bridge, the Pont Neuf Emphasis is put on the robust calculation of the camera posit ion. We also detail the techniques used for matching 2D and 3D primitives a nd for tracking features over the sequence. Our system overcomes two major difficulties. First, it is capable of handling poor-quality images, resulti ng from the fact that images were shot at night since the goal was to simul ate a new lighting system. Second, it can deal with important changes in vi ewpoint position and in appearance along the sequence. Throughout the paper , many results are shown to illustrate the different steps and difficulties encountered.