A Bayesian committee machine

The Bayesian committee machine (BCM) is a novel approach to combining estim ators that were trained on different data sets. Although the BCM can be app lied to the combination of any kind of estimators, the main foci are gaussi an process regression and related systems such as regularization networks a nd smoothing splines for which the degrees of freedom increase with the num ber of training data. Somewhat surprisingly, we find that the performance o f the BCM improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator. The BCM also provides a new solution for on-l ine learning with potential applications to data mining. We apply the BCM t o systems with fixed basis functions and discuss its relationship to gaussi an process regression. Finally, we show how the ideas behind the BCM can be applied in a non-Bayesian setting to extend the input-dependent combinatio n of estimators.