This article describes a novel approach to sensor-based decision makin
g based on formulating and solving large systems of parametric constra
ints. The constraints describe both a model for sensor data and the cr
iteria for correct decisions about the data. An incremental constraint
solving technique that performs decision-directed model recovery is d
eveloped. This method is straightforward to apply, is easily paralleli
zed, and convergence can be demonstrated under very reasonable structu
ral and statistical assumptions. This approach is demonstrated on seve
ral different decision-making problems involving manipulation and cate
gorization of objects observed with a range scanner. The experiments i
ndicate that simultaneous solution of both model constraints and decis
ion criteria can lead to efficient and effective decision making, even
when the observed data does not strongly determine a data model.