Stable Distribution Alignment Using the Dual of the Adversarial Distance

Learning to align distributions by minimizing an adversarial distance between them has recently achieved impressive results. However, such models are difficult to optimize with gradient descent and they often do not converge without very careful parameter tuning and initialization. We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment. Our empirical results suggest that using the dual formulation for linear and kernelized discriminators results in a more stable convergence to a desirable solution. We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem using a subset of MNIST and USPS. In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.