Assimilating disparate sensory feedback within virtual environments for telerobotic systems

Traditional telerobotic systems often provide feedback to the user through a variety of sensing modalities, for example through live video imagery, fo rce reflection, or acoustic signals. When a teleoperator guides a robotic t ask while immersed in a virtual environment, geometric representations of t he world are provided as feedback. In this case, the teleoperator interacts with the environment by visually observing these virtual objects and direc ting their motion. At issue is how to appropriately use various sensing mod alities provided by disparate sensors in a system of this configuration. Th is paper focuses on assimilating the disparate feedback provided by force a nd vision sensors for telerobotic systems guided from within virtual enviro nments. A framework for feedback assimilation is described based on the con cept of sensor resolvability. Sensor resolvability is used in two ways: (1) to update the virtual environment; (2) to guide the desired task in the re al world. Resolvability selects the appropriate sensing modality to use in updating the virtual environment within which the teleoperator is immersed. It is also used to direct semi-autonomous agents that interact directly wi th the real world to perform the desired task. Experimental results demonst rate the significant advantages of assimilating disparate sensory feedback throughout a telerobotic task using the concept of sensor resolvability. (C ) 2001 Elsevier Science B.V. All rights reserved.