Learning Time-Efficient Deep Architectures with Budgeted Super Networks

Learning neural network architectures is a way to discover new highly predictive models. We propose to focus on this problem from a different perspective where the goal is to discover architectures efficient in terms of both prediction quality and computation cost, e.g time in milliseconds, number of operations... For instance, our approach is able to solve the following task: find the best neural network architecture (in a very large set of possible architectures) able to predict well in less than 100 milliseconds on my mobile phone. Our contribution is based on a new family of models called Budgeted Super Networks that are learned using reinforcement-learning inspired techniques applied to a budgeted learning objective function which includes the computation cost during disk/memory operations at inference. We present a set of experiments on computer vision problems and show the ability of our method to discover efficient architectures in terms of both predictive quality and computation time.