Stochastic Optimization of a Locally Smooth Function under Correlated Bandit Feedback

In this paper we consider the problem of online stochastic optimization of a locally smooth function under bandit feedback. We introduce the high confidence tree (HCT) algorithm, a novel any-time $\mathcal X$-armed bandit algorithm, and derive regret bounds matching the performance of existing state-of-the-art in terms of dependency on number of steps $n$ and near-optimality dimensions $d$. The main advantage of HCT is that it handles the challenging case of correlated arms, whereas existing methods require that rewards to be conditionally independent of each others. HCT also improves on the state-of-the-art in terms of space complexity as well as requiring a weaker smoothness assumption on the mean-reward function in compare to the previous anytime algorithms. Finally, we discuss how HCT can be applied to the problem of policy search in reinforcement learning and we report preliminary empirical results.