STORAGE CAPACITIES OF COMMITTEE MACHINES WITH OVERLAPPING AND NONOVERLAPPING RECEPTIVE-FIELDS

We present theoretical investigations via the replica theory of the st orage capacities of committee machines with a large number M of hidden units and spherical weights. Difficulties arise in the solution of th is problem in the limit of large M. In the case of overlapping recepti ve fields, as the number of patterns increases, both permutation symme try and replica symmetry are broken, which leads to the appearance of many order parameters and causes additional difficulty. We observe tha t the relations among these order parameters yield a set of quantities which are small in the limit of large M, making the asymptotic calcul ation tractable. Using the one-step replica symmetry breaking scheme, we compute the asymptotic value alpha(c) of the storage capacity per i nput unit in the limit of large M. We find that alpha(c) similar or eq ual to (8 root 2/(pi-2))M root ln M. The shift to the case of non-over lapping receptive fields can be made easily; we then find alpha c simi lar or equal to (8 root 2/pi)root ln M. Both values satisfy the bound of Mitchison and Durbin.