Interval Universal Approximation for Neural Networks

To certify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using interval bound propagation (IBP). IBP is an incomplete calculus that over-approximates the set of possible predictions of a neural network. In this paper, we introduce the interval universal approximation (IUA) theorem, which sheds light on the power and limits of IBP. First, IUA shows that neural networks not only can approximate any continuous function $f$ (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrary close approximation of the set semantics of $f$ (the result of applying $f$ to a set of inputs). We call this notion of approximation interval approximation. Our result (1) extends the recent result of Baader et al. (2020) from ReLUs to a rich class of activation functions that we call squashable functions, and (2) implies that we can construct certifiably robust neural networks under $\ell_\infty$-norm using almost any practical activation function. Our construction and that of Baader et al. (2020) are exponential in the size of the function's domain. The IUA theorem additionally establishes a limit on the capabilities of IBP. Specifically, we show that there is no efficient construction of a neural network that interval-approximates any $f$, unless P=NP. To do so, we present a novel reduction from 3SAT to interval-approximation of neural networks. It implies that it is hard to construct an IBP-certifiably robust network, even if we have a robust network to start with.