Coresets For Monotonic Functions with Applications to Deep Learning

Coreset (or core-set) in this paper is a small weighted \emph{subset} $Q$ of the input set $P$ with respect to a given \emph{monotonic} function $f:\mathbb{R}\to\mathbb{R}$ that \emph{provably} approximates its fitting loss $\sum_{p\in P}f(p\cdot x)$ to \emph{any} given $x\in\mathbb{R}^d$. Using $Q$ we can obtain approximation to $x^*$ that minimizes this loss, by running \emph{existing} optimization algorithms on $Q$. We provide: (i) a lower bound that proves that there are sets with no coresets smaller than $n=|P|$ , (ii) a proof that a small coreset of size near-logarithmic in $n$ exists for \emph{any} input $P$, under natural assumption that holds e.g. for logistic regression and the sigmoid activation function. (iii) a generic algorithm that computes $Q$ in $O(nd+n\log n)$ expected time, (iv) novel technique for improving existing deep networks using such coresets, (v) extensive experimental results with open code.oving existing deep networks using such coresets, (v) extensive experimental results with open code.