Adaptive Sequential Experiments with Unknown Information Arrival Processes

Sequential experiments are often designed to strike a balance between maximizing immediate payoffs based on available information, and acquiring new information that is essential for maximizing future payoffs. This trade-off is captured by the multi-armed bandit (MAB) framework that has been studied and applied, typically when at each time epoch feedback is received only on the action that was selected at that epoch. However, in many practical settings, including product recommendations, dynamic pricing, retail management, and health care, additional information may become available between decision epochs. We introduce a generalized MAB formulation in which auxiliary information may appear arbitrarily over time. By obtaining matching lower and upper bounds, we characterize the minimax complexity of this family of problems as a function of the information arrival process, and study how salient characteristics of this process impact policy design and achievable performance. In terms of achieving optimal performance, we establish that: $(i)$ upper confidence bound and posterior sampling policies possess natural robustness with respect to the information arrival process without any adjustments, which uncovers a novel property of these policies and further lends credence to their appeal; and $(ii)$ policies with exogenous exploration rate do not possess such robustness. For such policies, we devise a novel virtual time indices method for dynamically controlling the effective exploration rate. We apply our method for designing $\epsilon_t$-greedy-type policies that, without any prior knowledge on the information arrival process, attain the best performance that is achievable when the information arrival process is a priori known. We use data from a large media site to analyze the value that may be captured in practice by leveraging auxiliary information for designing content recommendations.