FINITE-SIZE EFFECTS IN BAYESIAN MODEL SELECTION AND GENERALIZATION

We show that in supervised learning from a supplied data set Bayesian model selection, based on the evidence, does not optimize generalizati on performance even for a learnable linear problem. This is demonstrat ed by examining the finite size effects in hyperparameter assignment f rom the evidence procedure and the resultant generalization performanc e. Our approach demonstrates the weakness of average case and asymptot ic analyses. Using simulations we corroborate our analytic results and examine an alternative model selection criterion, namely cross-valida tion. This numerical study shows that the cross-validation hyperparame ter estimates correlate more strongly than those of the evidence with optimal performance. However, we show that for a sufficiently large in put dimension the evidence procedure could provide a reliable alternat ive to the more computationally expensive cross-validation.