Coefficient of determination in nonlinear signal processing

For statistical design of an optimal filter, it is probabilistically advant ageous to employ a large number of observation random variables; however, e stimation error increases with the number of variables, so that variables n ot contributing to the determination of the target variable can have a detr imental effect. In linear filtering, determination involves the correlation coefficients among the input and target variables. This paper discusses us e of the more general coefficient of determination in nonlinear filtering. The determination coefficient is defined in accordance with the degree to w hich a filter estimates a target variable beyond the degree to which the ta rget variable is estimated by its mean. Filter constraint decreases the coe fficient, but it also decreases estimation error in filter design. Because situations in which the sample is relatively small in comparison with the n umber of observation variables are of salient interest, estimation of the d etermination coefficient is considered in detail. One may be unable to obta in a good estimate of an optimal filter, but can nonetheless use rough esti mates of the coefficient to find useful sets of observation variables. Sinc e minimal-error estimation underlies determination, this material is at the interface of signal processing, computational learning, and pattern recogn ition. Several signal-processing factors impact application: the signal mod el, morphological operator representation, and desirable operator propertie s. In particular, the paper addresses the VC dimension of increasing operat ors in terms of their morphological kernel/basis representations. Two appli cations are considered: window size for restoring degraded binary images; f inding sets of genes that have significant predictive capability relative t o target genes in genomic regulation. (C) 2000 Elsevier Science B.V. All ri ghts reserved.