Modelling User's Theory of AI's Mind in Interactive Intelligent Systems

Multi-armed bandits provide a sample- and computationally efficient approach to developing assisting agents for interactive systems. Yet, they cannot capture strategic behaviour of an intelligent user, be it human or artificial, who forms a mental model of the system. We propose a new probabilistic multi-agent model that endows bandits with a theory of mind: the system has a model of the user having a model of the system. This is implemented as a nested bandit--Markov decision process--bandit model. We show that inference in the model reduces to probabilistic inverse reinforcement learning. Results show improved performance in simulations and in a user experiment. The improvements when users can form accurate mental models that the system can capture imply that predictability of the interactive intelligent system is important not only for the user experience but also for the design of the system's statistical models.