Efficient Learning and Planning with Compressed Predictive States

Predictive state representations (PSRs) offer an expressive framework for modelling partially observable dynamical systems. By compactly representing systems as functions of observable quantities, the PSR learning approach avoids using local-minima prone expectation-maximization. Moreover, since PSRs do not require a predetermined latent state structure as an input, they offer an attractive framework for agnostic model-based reinforcement learning, where agents must plan in domains without prior knowledge. Unfortunately, the expressiveness of PSRs comes with significant computational cost, and this cost is a major factor inhibiting the use of PSRs in applications. In order to alleviate this shortcoming, we introduce the notion of compressed PSRs (CPSRs). The CPSR learning approach synthesizes recent advancements in dimensionality reduction, incremental matrix decomposition, and compressed sensing. We show how this approach drastically reduces the computational cost of learning PSRs while also providing effective regularization. Going further, we propose a planning framework which exploits these learned models. And, we show that this compressed model-based approach facilitates planning in large, complex partially observable domains without prior knowledge, a task that is infeasible without the principled use of compression.