EXTREME EIGENVALUES OF LARGE-DIMENSIONAL SPIKED FISHER MATRICES WITH APPLICATION

Consider two p-variate populations, not necessarily Gaussian, with covariance matrices .. and .., respectively. Let S. and S. be the corresponding sample covariance matrices with degrees of freedom m and n. When the difference . between .. and .. is of small rank compared to p, m and n, the Fisher matrix $S: = S_2^{ - 1}{S_1}$ is called a spiked Fisher matrix. When p, m and n grow to infinity proportionally, we establish a phase transition for the extreme eigenvalues of the Fisher matrix: a displacement formula showing that when the eigenvalues of . (spikes) are above (or under) a critical value, the associated extreme eigenvalues of S will converge to some point outside the support of the global limit (LSD) of other eigenvalues (become outliers); otherwise, they will converge to the edge points of the LSD. Furthermore, we derive central limit theorems for those outlier eigenvalues of S. The limiting distributions are found to be Gaussian if and only if the corresponding population spike eigenvalues in . are simple. Two applications are introduced. The first application uses the largest eigenvalue of the Fisher matrix to test the equality between two high-dimensional covariance matrices, and explicit power function is found under the spiked alternative. The second application is in the field of signal detection, where an estimator for the number of signals is proposed while the covariance structure of the noise is arbitrary.