Linearly combining density estimators via stacking

This paper presents experimental results with both real and artificial data combining unsupervised learning algorithms using stacking. Specifically, s tacking is used to form a linear combination of finite mixture model and ke rnel density estimators for non-parametric multivariate density estimation. The method outperforms other strategies such as choosing the single best m odel based on cross-validation, combining with uniform weights, and even us ing the single best model chosen by "Cheating" and examining the test set. We also investigate (1) how the utility of stacking changes when one of the models being combined is the model that generated the data, (2) how the st acking coefficients of the models compare to the relative frequencies with which cross-validation chooses among the models, (3) visualization of combi ned "effective" kernels, and (4) the sensitivity of stacking to overfitting as model complexity increases.