From exp-concavity to variance control: O(1/n) rates and online-to-batch conversion with high probability

We present an algorithm for the statistical learning setting with a bounded exp-concave loss in $d$ dimensions that obtains excess risk $O(d/n)$ with high probability: the dependence on the confidence parameter $\delta$ is polylogarithmic in $1/\delta$. The core technique is to boost the confidence of recent in-expectation $O(d/n)$ excess risk bounds for empirical risk minimization (ERM), without sacrificing the rate, by leveraging a Bernstein condition which holds due to exp-concavity. This Bernstein condition implies that the variance of excess loss random variables are controlled in terms of their excess risk. Using this variance control, we further show that a regret bound for any online learner in this setting translates to a high probability excess risk bound for the corresponding online-to-batch conversion of the online learner. We also show that with probability $1 - \delta$ the standard ERM method obtains excess risk $O(d (\log(n) + \log(1/\delta))/n)$.