StacNAS: Towards Stable and Consistent Differentiable Neural Architecture Search

Differentiable Neural Architecture Search algorithms such as DARTS have attracted much attention due to the low search cost and competitive accuracy. However, it has been observed that DARTS can be unstable, especially when applied to new problems. One cause of the instability is the difficulty of two-level optimization. In addition, we identify two other causes: (1) Multicollinearity of correlated/similar operations leads to unpredictable change of the architecture parameters during search; (2) The optimization complexity gap between the proxy search stage and the final training leads to suboptimal architectures. Based on these findings, we propose a two-stage grouped variable pruning algorithm using one-level optimization. In the first stage, the best group is activated, and in the second stage, the best operation in the activated group is selected. Extensive experiments verify the superiority of the proposed method both for accuracy and for stability. For the DARTS search space, the proposed strategy obtains state-of-the-art accuracies on CIFAR-10, CIFAR-100 and ImageNet. Code is available at https://github.com/susan0199/stacnas.