On the Expressive Power of Overlapping Operations of Deep Networks

Expressive Efficiency with respect to a network architectural attribute P refers to the property where an architecture without P must grow exponentially large in order to approximate the expressivity of a network with attribute P. For example, it is known that depth is an architectural attribute that generates exponential efficiency in the sense that a shallow network must grow exponentially large in order to approximate the functions represented by a deep network of polynomial size. In this paper we extend the study of expressive efficiency to the attribute of network connectivity and in particular to the effect of "overlaps" in the convolutional process, i.e., when the stride of the convolution is smaller than its kernel size (receptive field). Our analysis shows that having overlapping local receptive fields, and more broadly denser connectivity, results in an exponential increase in the expressive capacity of neural networks. Moreover, while denser connectivity can increase the expressive capacity, we show that the most common types of modern architectures already exhibit exponential increase in expressivity, without relying on fully-connected layers.