Research in the area of signal detection in the presence of unknown interfe
rence has resulted in a number of adaptive detection algorithms. Examples o
f such algorithms include the adaptive matched filter (AMF), the generalize
d likelihood ratio test (GLRT), and the adaptive coherence estimator (ACE),
Each of these algorithms results in a tradeoff between detection performan
ce for matched signals and rejection performance for mismatch signals. This
paper introduces a new detection algorithm we call the adaptive beamformer
orthogonal rejection test (ABORT). Our test decides if an observation cont
ains a multidimensional signal belonging to one subspace or if it contains
a multidimensional signal belonging to an orthogonal subspace when unknown
complex Gaussian noise is present. In our analysis, we use a statistical hy
pothesis testing framework to develop a generalized likelihood ratio decisi
on rule, We evaluate the performance of this decision rule in both the matc
hed and mismatched signal cases. Our results show that for constant power c
omplex Gaussian noise, if the signal is matched to the steering vector ABOR
T, GLRT, and AMF give approximately equivalent probability of detection, hi
gher than that of ACE, which trades detection probability for an extra inva
riance to scale mismatch between training and test data, Of these four test
s, ACE is most selective and, therefore, least tolerant of mismatch, wherea
s AMF is most tolerant of mismatch and, therefore, least selective. ABORT a
nd GLRT offer compromises between these extremes, with ABORT more like ACE
and with GLRT more like AR IF.