Discrimination with many variables

Many statistical methods for discriminant analysis do not adapt well or eas ily to situations where the number of variables is large, possibly even exc eeding the number of cases in the training set. We explore a variety of met hods for providing robust identification of future samples in this situatio n. We develop a range of flexible Bayesian methods, and primarily a new hie rarchical covariance compromise method, akin to regularized discriminant an alysis. Although the methods are much more widely applicable, the motivatin g problem was that of discriminating between groups of samples on the basis of their near-infrared spectra. Here the ability of the Bayesian methods t o rake account of continuity of the spectra may be beneficial. The spectra may consist of absorbances or reflectances at as many as 1,000 wavelengths, and yet there may be only tens or hundreds of training samples in which bo th sample spectrum and group identity are known. Such problems arise in the food and pharmaceutical industries; for example, authentication of foods ( e.g., detecting the adulteration of orange juice) and identification of pha rmaceutical ingredients. Our illustrating example concerns the discriminati on of 39 microbiological taxa and 8 aggregate genera. Simulations also illu strate the effectiveness of the hierarchical Bayes covariance method. We di scuss a number of scoring rules, both local and global, for judging the fit of data to the Bayesian models, and adopt a cross-classificatory approach for estimating hyperparameters.