Deep Incremental Model Informed Reinforcement Learning for Continuous Robotic Control

Model-based reinforcement learning attempts to use an available or learned model to improve the data efficiency of reinforcement learning. This work proposes a one-step lookback approach that jointly learns the deep incremental model and the policy to realize the sample-efficient continuous robotic control, wherein the control-theoretical knowledge is utilized to decrease the model learning difficulty and facilitate efficient training. Specifically, we use one-step backward data to facilitate the deep incremental model, an alternative structured representation of the robotic evolution model, that accurately predicts the robotic movement but with low sample complexity. This is because the formulated deep incremental model degrades the model learning difficulty into a parametric matrix learning problem, which is especially favourable to high-dimensional robotic applications. The imagined data from the learned deep incremental model is used to supplement training data to enhance the sample efficiency. Comparative numerical simulations on benchmark continuous robotics control problems are conducted to validate the efficiency of our proposed one-step lookback approach.