A principal components approach to combining regression estimates

The goal of combining the predictions of multiple learned models is to form an improved estimator. A combining strategy must be able to robustly handl e the inherent correlation, or multicollinearity, of the learned models whi le identifying the unique contributions of each. A progression of existing approaches and their limitations with respect to these two issues are discu ssed. A new approach, PCR*, based on principal components regression is pro posed to address these limitations. An evaluation of the new approach on a collection of domains reveals that (1) PCR* was the most robust combining m ethod, (2) correlation could be handled without eliminating any of the lear ned models, and (3) the principal components of the learned models provided a continuum of "regularized" weights from which PCR* could choose.