MUSIC AND MULTIMEDIA KNOWLEDGE REPRESENTATION AND REASONING - THE HARP SYSTEM

This paper introduces Hybrid Action Representation and Planning, or HA RP, a hybrid system for the representation and real-time processing of music and multimedia data, based on the use of both artificial intell igence (AI) and traditional techniques. HARP is ''hybrid'' because it is based on the integration of different formalisms, able to manage th e different nature and levels of music and multimedia data. A symbolic component deals with the multi-level, abstract representations and re asoning mechanisms. A sub-symbolic component manages continuous proces ses and physical signals. The symbolic component integrates a semantic network language of the family of KL-ONE with a temporal logic and pr oduction rules. The sub-symbolic component is modeled as a network of cooperative agents, mainly based on reasoning by metaphors and dynamic systems. Both components are deeply integrated, e.g., the behavior an d the communication links of agents depend on the structure of the sym bolic knowledge base. In general, changes in one of the two components affect the behavior of the other. The HARP system has been designed f or computer-assisted composition, performance, and analysis, and for t wo particular multimedia domains: a theatrical automation project, whe re the system is delegated to integrate and control sound, music, and three-dimensional computer animation of humanoid figures interacting w ith real actors on stage; and a museum application, in which the syste m controls (in real time) the behavior of a sensor-based, autonomous m obile system in an exposition area, able to welcome, entertain, guide, and instruct visitors. The HARP symbolic kernel is written in Prolog and is integrated to the sub-symbolic system and the user interface wr itten in C++.