On the Quality of the Initial Basin in Overspecified Neural Networks

Over the past few years, artificial neural networks have seen a dramatic resurgence in popularity as a tool for solving hard learning problems in AI applications. While it is widely known that neural networks are computationally hard to train in the worst case, in practice, neural networks are trained efficiently using SGD methods and a variety of techniques which accelerate the learning process. One mechanism which has been suggested to explain this is overspecification, which is the training of a network larger than what would be needed with unbounded computational power. Empirically, despite worst-case NP-hardness results, large networks tend to achieve a smaller error over the training set. In this work, we aspire to understand this phenomenon. In particular, we wish to better understand the behavior of the error over the sample as a function of the weights of the network, where we focus mostly on neural nets comprised of 2 layers, although we will also consider single neuron nets and nets of arbitrary depth, investigating properties such as the number of local minima the function has, and the probability of initializing from a basin with a given minimal value, with the goal of finding reasonable conditions under which efficient learning of the network is possible.