Beyond Sharing Weights for Deep Domain Adaptation

Deep Neural Networks have demonstrated outstanding performance in many Computer Vision tasks but typically require large amounts of labeled training data to achieve it. This is a serious limitation when such data is difficult to obtain. In traditional Machine Learning, Domain Adaptation is an approach to overcoming this problem by leveraging annotated data from a source domain, in which it is abundant, to train a classifier to operate in a target domain, in which labeled data is either sparse or even lacking altogether. In the Deep Learning case, most existing methods use the same architecture with the same weights for both source and target data, which essentially amounts to learning domain invariant features. Here, we show that it is more effective to explicitly model the shift from one domain to the other. To this end, we introduce a two-stream architecture, one of which operates in the source domain and the other in the target domain. In contrast to other approaches, the weights in corresponding layers are related but not shared to account for differences between the two domains. We demonstrate that this both yields higher accuracy than state-of-the-art methods on several object recognition and detection tasks and consistently outperforms networks with shared weights in both supervised and unsupervised settings.