USE OF MULTIMODAL INFORMATION IN FACIAL EMOTION RECOGNITION

Detection of facial emotions are mainly addressed by computer vision r esearchers based on facial display. Also detection of vocal expression s of emotions is found in research work done by acoustic researchers. Most of these research paradigms are devoted purely to visual or purel y to auditory human emotion detection. However we found that it is ver y interesting to consider both of these auditory and visual informatio ns together, for processing, since we hope this kind of multimodal inf ormation processing will become a datum of information processing in f uture multimedia era. By several intensive subjective evaluation studi es we found that human beings recognize Anger, happiness, Surprise and Dislike by their visual appearance, compared to voice only detection. When the audio track of each emotion clip is dubbed with a different type of auditory emotional expression, still Anger, Happiness and Surp rise were video dominant. However Dislike emotion gave mixed responses to different speakers. In both studies we found that Sadness and Fear emotions were audio dominant. As a conclusion to the paper we propose a method of facial emotion detection by using a hybrid approach, whic h uses multimodal informations for facial emotion recognition.