Ray RLlib: A Framework for Distributed Reinforcement Learning

Reinforcement learning (RL) training involves the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation. Current RL libraries offer parallelism at the level of the entire program, coupling all algorithm components together and making existing implementations difficult to scale, combine, and reuse. We argue for distributing RL components in a composable way by adapting algorithms for top-down hierarchical control, thereby encapsulating parallelism and resource requirements within short-running compute tasks. We demonstrate this principle by building RLlib on top of a task-based framework and show that we can implement a wide range of state-of-the art algorithms on top of a small set of general abstractions. These abstractions are key to composability and reuse in RLlib and do not come at the cost of performance---in our experiments, RLlib matches or exceeds the performance of highly optimized reference implementations. Ray RLlib is available as part of Ray at https://github.com/ray-project/ray/.