HOW INITIAL CONDITIONS AFFECT GENERALIZATION PERFORMANCE IN LARGE NETWORKS

Generalization is one of the most important problems in neural-network research, It is influenced by several factors in the network design, such as network size, weight decay factor, and others, We show here th at the initial weight distribution (for gradient decent training algor ithms) is one other factor that influences generalization, The initial conditions guide the training algorithm to search particular places o f the weight space, For instance small initial weights tend to result in low complexity networks, and therefore can effectively act as a reg ularization factor. We propose a novel network complexity measure, whi ch is helpful in shedding insight into the phenomenon, as well as in s tudying other aspects of generalization.