Flexible operation of a robotic agent in an uncalibrated environment r
equires the ability to recover unknown or partially known parameters o
f the workspace through sensing. Of the sensors available to a robotic
agent, visual sensors provide information that is richer and more com
plete than other sensors. In this paper we present robust techniques f
or the derivation of depth from feature points on a target's surface a
nd for the accurate and high-speed tracking of moving targets. We use
these techniques in a system that operates with little or no a priori
knowledge of object-and camera-related parameters to robustly determin
e such object-related parameters as velocity and depth. Such determina
tion of extrinsic environmental parameters is essential for performing
higher level tasks such as inspection, exploration, tracking, graspin
g, and collision-free motion planning. For both applications, we use t
he Minnesota robotic visual tracker (MRVT) (a single visual sensor mou
nted on the end-effector of a robotic manipulator combined with a real
-time vision system) to automatically select feature points on surface
s, to derive an estimate of the environmental parameter in question, a
nd to supply a control vector based upon these estimates to guide the
manipulator.