Efficient Constrained Regret Minimization

Online learning constitutes a mathematical framework to analyze sequential decision making problems in adversarial environments. The learner repeatedly chooses an action, the environment responds with an outcome, and then the learner receives a reward for the played action. The goal of the learner is to maximize his total reward. However, there are situations in which, in addition to maximizing the cumulative reward, there are some additional constraints/goals on the sequence of decisions that must be satisfied by the learner. For example, in \textit{online marketing}, simultaneously maximizing the cumulative reward and the number of buyers to take advantage of word-of-mouth advertising for future marketing seems to be a more ambitious goal than only maximizing cumulative reward. As another example, learning from costly expert advice captures more realistic settings than the original setting in applications such as routing in networks with power constraint. In this paper we study an extension to the online learning where the learner aims to maximize the total reward given that some additional constraints need to be satisfied. We propose Lagrangian exponentially weighted average (\textbf{LEWA}) algorithm, an efficient algorithm to solve constrained online learning, which is a primal dual variant of the well known exponentially weighted average algorithm and inspired by the theory of Lagrangian method in constrained optimization. We establish the regret and the violation of the constraint bounds in full information and bandit feedback models.