Representational Distance Learning for Deep Neural Networks

We propose representational distance learning (RDL), a technique that allows transferring knowledge from an arbitrary model with task related information to a deep neural network (DNN). This method seeks to maximize the similarity between the representational distance matrices (RDMs) of a model with desired knowledge, the teacher, and a DNN currently being trained, the student. The knowledge contained in the information transformations performed by the teacher are transferred to a student using auxiliary error functions. This allows a DNN to simultaneously learn from a teacher model and learn to perform some task within the framework of backpropagation. We test the use of RDL for knowledge distillation, also known as model compression, from a large teacher DNN to a small student DNN using the MNIST and CIFAR-10 datasets. Also, we test the use of RDL for knowledge transfer between tasks using the CIFAR-10 and CIFAR-100 datasets. For each test, RDL significantly improves performance when compared to traditional backpropagation alone and performs similarly to, or better than, recently proposed methods for model compression and knowledge transfer.