Learning Multiple Tasks with Deep Relationship Networks

Deep neural networks trained on large-scale dataset can learn transferable features that promote learning multiple tasks for inductive transfer and labeling mitigation. As deep features eventually transition from general to specific along the network, a fundamental problem is how to exploit the relationship structure across different tasks while accounting for the feature transferability in the task-specific layers. In this work, we propose a novel Deep Relationship Network (DRN) architecture for multi-task learning by discovering correlated tasks based on multiple task-specific layers of a deep convolutional neural network. DRN models the task relationship by imposing matrix normal priors over the network parameters of all task-specific layers, including higher feature layers and classifier layer that are not transferable safely. By jointly learning the transferable features and task relationships, DRN is able to alleviate the dilemma of negative-transfer in the feature layers and under-transfer in the classifier layer. Empirical evidence shows that DRN yields state-of-the-art classification results on standard multi-domain object recognition datasets.