Learned-norm pooling for deep neural networks

In this paper we proposed a novel nonlinear unit, which is called as $L_p$ unit, for a multi-layer perceptron (MLP). The proposed $L_p$ unit receives signal from several projections of the layer below and computes the normalized $L_p$ norm. We notice two interesting interpretations of the $L_p$ unit. First, we note that the proposed unit is a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks(CNN), HMAX models and neocognitrons. Furthermore, under certain constraints, the $L_p$ unit is a generalization of the recently proposed maxout unit (Goodfellow et al, 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Second, we provide a geometrical interpretation of the activation function. Each $L_p$ unit defines a spherical boundary, with its exact shape defined by the order $p$. We claim that this makes it possible to obtain arbitrarily shaped, curved boundaries more efficiently by combining just a few $L_p$ units of different orders. We empirically evaluate the proposed $L_p$ units on a number of datasets and show that MLPs consisting of the $L_p$ units achieves the state-of-the-art results on a number of benchmark datasets.