Depth Separation in ReLU Networks for Approximating Smooth Non-Linear Functions

We provide a depth-based separation result for feed-forward ReLU neural networks, showing that a wide family of non-linear, twice-differentiable functions on $[0,1]^d$, which can be approximated to accuracy $\epsilon$ by ReLU networks of depth and width $\mathcal{O}(\text{poly}(\log(1/\epsilon)))$, cannot be approximated to similar accuracy by constant-depth ReLU networks, unless their width is at least $\Omega(1/\epsilon)$.