Building cost functions minimizing to some summary statistics

A learning machine-or a model-is usually trained by minimizing a given crit erion (the expectation of the cost function), measuring the discrepancy bet ween the model output and the desired output. As is already well known, the choice of the cost function has a profound impact on the probabilistic int erpretation of the output of the model, after training. In this work, we us e the calculus of variations in order to tackle this problem, In particular , we derive necessary and sufficient conditions on the cost function ensuri ng that the output of the trained model approximates 1) the conditional exp ectation of the desired output given the explanatory variables; 2) the cond itional median land, more generally, the q-quantile); 3) the conditional ge ometric mean; and 4) the conditional variance, The same method could be app lied to the estimation of other summary statistics as well, We also argue t hat the least absolute deviations criterion could, in some cases, act as an alternative to the ordinary least squares criterion for nonlinear regressi on, In the same vein, the concept of "regression quantile" is briefly discu ssed.