Contextual Fixed-Budget Best Arm Identification: Adaptive Experimental Design with Policy Learning

Individualized treatment recommendation is a crucial task in evidence-based decision-making. In this study, we formulate this task as a fixed-budget best arm identification (BAI) problem with contextual information. In this setting, we consider an adaptive experiment given multiple treatment arms. At each round, a decision-maker observes a context (covariate) that characterizes an experimental unit and assigns the unit to one of the treatment arms. At the end of the experiment, the decision-maker recommends a treatment arm estimated to yield the highest expected outcome conditioned on a context (best treatment arm). The effectiveness of this decision is measured in terms of the worst-case expected simple regret (policy regret), which represents the largest difference between the conditional expected outcomes of the best and recommended treatment arms given a context. Our initial step is to derive asymptotic lower bounds for the worst-case expected simple regret, which also implies ideal treatment assignment rules. Following the lower bounds, we propose the Adaptive Sampling (AS)-Policy Learning recommendation (PL) strategy. Under this strategy, we randomly assign a treatment arm with a ratio of a target assignment ratio at each round. At the end of the experiment, we train a policy, a function that recommends a treatment arm given a context, by maximizing the counterfactual empirical policy value. Our results show that the AS-PL strategy is asymptotically minimax optimal, with its leading factor of expected simple regret converging with our established worst-case lower bound. This research has broad implications in various domains, and in light of existing literature, our method can be perceived as an adaptive experimental design tailored for policy learning, on-policy learning, or adaptive welfare maximization.