Learning Adversarially Robust Policies in Multi-Agent Games

Robust decision-making in multiplayer games requires anticipating what reactions a player policy may elicit from other players. This is difficult in games with three or more players: when one player fixes a policy, it induces a subgame between the remaining players with many possible equilibria outcomes. Predicting the worst-case outcome of a policy is thus an equilibrium selection problem -- one known to be generally NP-Hard. We show that worst-case coarse-correlated equilibria can be efficiently approximated in smooth games and propose a framework that uses the worst-case evaluation scheme to learn robust player policies. We further prove the framework can be extended to handle uncertainty about the bounded rationality of other players. In experiments, our framework learns robust policies in repeated N-player matrix games and, when applied to deep multi-agent reinforcement learning, can scale to complex spatiotemporal games. For example, it learns robust AI tax policies that improve welfare by up to 15%, even when taxpayers are boundedly rational.