CAPACITY OF 2-LAYER FEEDFORWARD NEURAL NETWORKS WITH BINARY WEIGHTS

The low er and upper bounds for the information capacity of two-layer feedforward neural networks with binary interconnections, integer thre sholds for the hidden units, and zero threshold for the output unit is obtained through two steps, First, through a constructive approach ba sed on statistical analysis, it is shown that a specifically construct ed (N -2L -1) network with N input units, 2L hidden units, and one out put unit is capable of implementing, with almost probability one, any dichotomy of O(W/1n W) random samples drawn from some continuous distr ibutions, where W is the total number of weights of the network, This quantity is then used as a lower bound for the information capacity C of all (N -2L -1) networks with binary weights, Second, an upper bound is obtained and shown to be O(W) by a simple counting argument. There fore, we have Omega(W/ln W) less than or equal to C less than or equal to O(W).