A STATISTICAL-MECHANICAL ANALYSIS OF A BAYESIAN-INFERENCE SCHEME FOR AN UNREALIZABLE RULE

Within a Bayesian framework we consider a system that learns from exam ples. In particular, using a statistical mechanical formalism, we calc ulate the evidence and two performance measures, namely the generaliza tion error and the consistency measure, for a linear perceptron traine d and tested on a set of examples generated by a nonlinear teacher. Th e teacher is said to be unrealizable because the student can never mod el it without error. In fact, our model allows us to interpolate betwe en the known linear case and an unrealizable, nonlinear, case. A compa rison of the hyperparameters which maximize the evidence with those th at optimize the performance measures reveals that, when the student an d teacher are fundamentally mismatched, the evidence procedure is a mi sleading guide to optimizing the performance measures considered.