The mutual information of a stochastic binary channel: validity of the replica symmetry ansatz

We calculate the mutual information (MI) of a two-layered neural network wi th noiseless, continuous inputs and binary, stochastic outputs under severa l assumptions on the synaptic efficiencies. The interesting regime correspo nds to the limit where the number of both input and output units is large h ut their ratio is kept fixed at a value alpha. We first present a solution for the MI using the replica technique with a replica symmetric (RS) ansatz . Then we find an exact solution for this quantity valid in a neighbourhood of alpha = 0. An analysis of this solution shows that the system must have a phase transition at some finite value of alpha. This transition shows a singularity in the third derivative of the MI. As the RS solution turns out to be infinitely differentiable, it could be regarded as a smooth approxim ation to the MI. This is checked numerically in the validity domain of the exact solution.