A. Turiel et al., The mutual information of a stochastic binary channel: validity of the replica symmetry ansatz, J PHYS A, 32(10), 1999, pp. 1875-1894
We calculate the mutual information (MI) of a two-layered neural network wi
th noiseless, continuous inputs and binary, stochastic outputs under severa
l assumptions on the synaptic efficiencies. The interesting regime correspo
nds to the limit where the number of both input and output units is large h
ut their ratio is kept fixed at a value alpha. We first present a solution
for the MI using the replica technique with a replica symmetric (RS) ansatz
. Then we find an exact solution for this quantity valid in a neighbourhood
of alpha = 0. An analysis of this solution shows that the system must have
a phase transition at some finite value of alpha. This transition shows a
singularity in the third derivative of the MI. As the RS solution turns out
to be infinitely differentiable, it could be regarded as a smooth approxim
ation to the MI. This is checked numerically in the validity domain of the
exact solution.