Piecewise Linear Activation Functions For More Efficient Deep Networks

Activation functions play important roles in state-of-the-art neural networks and a great example is Rectified Linear Units(ReLU). In this work, we propose a Piece-wise Linear Activation function generalizing the Rectified Linear function by adding multiple linear segments with trainable variables. The new activation improves fitting and makes the neural network more efficient by projecting the data to multiple spaces according to input data range. This new activation function enables us to train the deep models with higher accuracy compared to ReLU. We can also train a shallower network with the same accuracy compared to a deeper network with ReLU. We plan to test our models on CIFAR10 and the ImageNet 2012 classification datasets. Both are representative datasets in computer vision using deep neural networks.