MixPath: A Unified Approach for One-shot Neural Architecture Search

Blending multiple convolutional kernels is proved advantageous in neural architectural design. However, current neural architecture search approaches are mainly limited to stacked single-path search space. How can the one-shot doctrine search for multi-path models remains unresolved. Specifically, we are motivated to train a multi-path supernet to accurately evaluate the candidate architectures. In this paper, we discover that in the studied search space, feature vectors summed from multiple paths are nearly multiples of those from a single path, which perturbs supernet training and its ranking ability. In this regard, we propose a novel mechanism called Shadow Batch Normalization(SBN) to regularize the disparate feature statistics. Extensive experiments prove that SBN is capable of stabilizing the training and improving the ranking performance (e.g. Kendall Tau 0.597 tested on NAS-Bench-101). We call our unified multi-path one-shot approach as MixPath, which generates a series of models that achieve state-of-the-art results on ImageNet.