AVIS: a connectionist-based framework for integrated auditory and visual information processing

This paper presents a general framework that facilitates the exploration of a single information-processing system in which auditory and visual inform ation are integrated. The framework allows for learning, adaptation, knowle dge discovery, and decision making. An application of the framework is a pe rson-identification task in which face and voice recognition are combined i n one system. Experiments are performed using visual and auditory dynamic f eatures which are synchronously extracted from visual and auditory informat ion flows, The experimental results support the hypothesis that the recogni tion rate is considerably enhanced by combining visual and auditory dynamic features. (C) 2000 Elsevier Science Inc. All rights reserved.