M. Shiino et T. Fukai, SELF-CONSISTENT SIGNAL-TO-NOISE ANALYSIS OF THE STATISTICAL BEHAVIOR OF ANALOG NEURAL NETWORKS AND ENHANCEMENT OF THE STORAGE CAPACITY, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics, 48(2), 1993, pp. 867-897
Based on the self-consistent signal-to-noise analysis (SCSNA) capable
of dealing with analog neural networks with a wide class of transfer f
unctions, enhancement of the storage capacity of associative memory an
d the related statistical properties of neural networks are studied fo
r random memory patterns. Two types of transfer functions with the thr
eshold parameter theta are considered, which are derived from the sigm
oidal one to represent the output of three-state neurons. Neural netwo
rks having a monotonically increasing transfer function F(M), F(M)(u)
= sgnu (Absolute value of u > theta), F(M)(u) = 0 (Absolute value of u
less-than-or-equal-to theta), are shown to make it impossible for the
spin-glass state to coexist with retrieval states in a certain parame
ter region of theta and alpha (loading rate of memory patterns), imply
ing the reduction of the number of spurious states. The behavior of th
e storage capacity with changing 0 is qualitatively the same as that o
f the Ising spin neural networks with varying temperature. On the othe
r hand, the nonmonotonic transfer function F(NM), F(NM)(u) = sgnu (Abs
olute value of u < theta), F(NM)(u) = 0 (Absolute value of u greater-t
han-or-equal-to theta) gives rise to remarkable features in several re
spects. First, it yields a large enhancement of the storage capacity c
ompared with the Amit-Gutfreund-Sompolinsky (AGS) value: with decreasi
ng theta from theta = infinity, the storage capacity alpha(c) of such
a network is increased from the AGS value (almost-equal-to 0.14) to at
tain its maximum value of almost-equal-to 0.42 at theta congruent-to 0
.7 and afterwards is decreased to vanish at theta = 0. Whereas for the
ta greater than or similar to 1 the storage capacity alpha(c) coincide
s with the value alpha(c) determined by the SCSNA as the upper bound o
f a ensuring the existence of retrieval solutions, for theta less than
or similar to 1 the alpha(c) is shown to differ from the alpha(c) wit
h the result that the retrieval solutions claimed by the SCSNA are uns
table for alpha(c) < alpha < alpha(c). Second, in the case of theta <
1 the network can exhibit a new type of phase which appears as a resul
t of a phase transition with respect to the non-Gaussian distribution
of the local fields of neurons: the standard type of retrieval state w
ith r not-equal 0 (i.e., finite width of the local field distribution)
, which is implied by the order-parameter equations of the SCSNA, disa
ppears at a certain critical loading rate alpha0, and for alpha less-t
han-or-equal-to 0 a qualitatively different type of retrieval state co
mes into existence in which the width of the local field distribution
vanishes (i.e., r = 0+). As a consequence, memory retrieval without er
rors becomes possible even in the saturation limit alpha not-equal 0.
Results of the computer simulations on the statistical properties of t
he novel phase with alpha less-than-or-equal-to alpha0 are shown to be
in satisfactory agreement with the theoretical results. The effect of
introducing self-couplings on the storage capacity is also analyzed f
or the two types of networks. It is conspicuous for the networks with
F(NM), where the self-couplings increase the stability of the retrieva
l solutions of the SCSNA with small values of theta, leading to a rema
rkable enhancement of the storage capacity.