Max-Margin Adversarial (MMA) Training: Direct Input Space Margin Maximization through Adversarial Training

We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. In theory, we show that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation". This max-margin perspective also provides an alternative interpretation on adversarial training with a fixed perturbation magnitude $\epsilon$: adversarial training is maximizing either a lower bound or an upper bound of the margin. Motivated by our theoretical analysis, we propose Max-Margin Adversarial (MMA) training to directly maximize the margins. Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by selecting the margin as the "correct" $\epsilon$ individually for each point. We demonstrate MMA training's efficacy and analyze its properties on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.