System comparison procedures for automatic target recognition systems

Estimating the performance of an automatic target recognition (ATR) system in terms of its probability of successfully identifying a target involves e xtensive image collection and processing, which can be very time-consuming and expensive. Therefore, we investigate the Wald sequential test for the d ifference in two proportions as a sample size-reducing alternative to ranki ng and selection procedures and confidence intervals. An analysis of the te st parameters leads to a practical methodology for implementing the Wald te st for fairly comparing two systems, based on specific experimental goals. The test is also extended with the modified, sequentially rejective Bonferr oni procedure for the multiple pairwise comparison of more than two systems . Two sampling schemes for different experimental goals are also discussed. The test methodology is applied to actual data to compare different config urations of a specific ATR system, with the results demonstrating that the modified Wald sequential procedure is a useful alternative to comparing pro portions with confidence intervals, particularly when data are expensive. ( C) 1999 John Wiley & Sons, Inc.