SELF-CONSISTENT SIGNAL-TO-NOISE ANALYSIS OF THE STATISTICAL BEHAVIOR OF ANALOG NEURAL NETWORKS AND ENHANCEMENT OF THE STORAGE CAPACITY

Based on the self-consistent signal-to-noise analysis (SCSNA) capable of dealing with analog neural networks with a wide class of transfer f unctions, enhancement of the storage capacity of associative memory an d the related statistical properties of neural networks are studied fo r random memory patterns. Two types of transfer functions with the thr eshold parameter theta are considered, which are derived from the sigm oidal one to represent the output of three-state neurons. Neural netwo rks having a monotonically increasing transfer function F(M), F(M)(u) = sgnu (Absolute value of u > theta), F(M)(u) = 0 (Absolute value of u less-than-or-equal-to theta), are shown to make it impossible for the spin-glass state to coexist with retrieval states in a certain parame ter region of theta and alpha (loading rate of memory patterns), imply ing the reduction of the number of spurious states. The behavior of th e storage capacity with changing 0 is qualitatively the same as that o f the Ising spin neural networks with varying temperature. On the othe r hand, the nonmonotonic transfer function F(NM), F(NM)(u) = sgnu (Abs olute value of u < theta), F(NM)(u) = 0 (Absolute value of u greater-t han-or-equal-to theta) gives rise to remarkable features in several re spects. First, it yields a large enhancement of the storage capacity c ompared with the Amit-Gutfreund-Sompolinsky (AGS) value: with decreasi ng theta from theta = infinity, the storage capacity alpha(c) of such a network is increased from the AGS value (almost-equal-to 0.14) to at tain its maximum value of almost-equal-to 0.42 at theta congruent-to 0 .7 and afterwards is decreased to vanish at theta = 0. Whereas for the ta greater than or similar to 1 the storage capacity alpha(c) coincide s with the value alpha(c) determined by the SCSNA as the upper bound o f a ensuring the existence of retrieval solutions, for theta less than or similar to 1 the alpha(c) is shown to differ from the alpha(c) wit h the result that the retrieval solutions claimed by the SCSNA are uns table for alpha(c) < alpha < alpha(c). Second, in the case of theta < 1 the network can exhibit a new type of phase which appears as a resul t of a phase transition with respect to the non-Gaussian distribution of the local fields of neurons: the standard type of retrieval state w ith r not-equal 0 (i.e., finite width of the local field distribution) , which is implied by the order-parameter equations of the SCSNA, disa ppears at a certain critical loading rate alpha0, and for alpha less-t han-or-equal-to 0 a qualitatively different type of retrieval state co mes into existence in which the width of the local field distribution vanishes (i.e., r = 0+). As a consequence, memory retrieval without er rors becomes possible even in the saturation limit alpha not-equal 0. Results of the computer simulations on the statistical properties of t he novel phase with alpha less-than-or-equal-to alpha0 are shown to be in satisfactory agreement with the theoretical results. The effect of introducing self-couplings on the storage capacity is also analyzed f or the two types of networks. It is conspicuous for the networks with F(NM), where the self-couplings increase the stability of the retrieva l solutions of the SCSNA with small values of theta, leading to a rema rkable enhancement of the storage capacity.