An expectation-based framework of object schemas and port-based agents fordisparate feedback assimilation

Robotic manipulation systems that operate in unstructured environments must be responsive to feedback from sensors that are disparate in both location and modality. This paper describes a distributed framework for assimilatin g the disparate feedback provided by force and vision sensors, including ac tive vision sensors, for robotic manipulation systems. The main components of the expectation-based framework include object schemas and port-based ag ents. Object schemas represent the manipulation task internally in terms of geometric models with attached sensor mappings. Object schemas are dynamic ally updated by sensor feedback, and thus provide an ability to perform thr ee dimensional spatial reasoning during task execution. Because object sche mas possess knowledge of sensor mappings, they are able to both select appr opriate sensors and guide active sensors based on task characteristics. Por t-based agents are the executors of reference inputs provided by object sch emas and are defined in terms of encapsulated control strategies. Experimen tal results demonstrate the capabilities of the framework in two ways: the performance of manipulation tasks with active camera-lens systems, and the assimilation of force and vision sensory feedback.