MSE-based regularization approach to direction estimation of coherent narrowband signals using linear prediction

This paper addresses the problem of directions-of-arrival (DOAs) estimation of coherent narrowband signals impinging on a uniform linear array (ULA) w hen the number of signals is unknown. By using an overdetermined linear pre diction (LP) model with a subarray scheme, the DOAs of coherent signals can be estimated from the zeros of the corresponding prediction polynomial. Al though the corrected least squares (CLS) technique can be used to improve t he accuracy of the LP parameters estimated from the noisy array data, the i nversion of the resulting matrix in the CLS estimation is ill-conditioned, and then, the CLS estimation becomes unstable. To combat this numerical ins tability, we introduce multiple regularization parameters into the CLS esti mation and show that determining the number of coherent signals is closely related to the truncation of the eigenvalues. An analytical expression of t he mean square error (MSE) of the estimated LP parameters is derived, and i t is clarified that the number of signals can be determined by comparing th e optimal regularization parameters with the corresponding eigenvalues. An iterative regularization algorithm is developed for estimating directions w ithout any a priori knowledge, where the number of coherent signals and the noise variance are estimated from the noise-corrupted received data simult aneously.