222

Large Video Planner Enables Generalizable Robot Control

Boyuan Chen
Tianyuan Zhang
Haoran Geng
Kiwhan Song
Caiyi Zhang
Peihao Li
William T. Freeman
Jitendra Malik
Pieter Abbeel
Russ Tedrake
Vincent Sitzmann
Yilun Du
Main:13 Pages
16 Figures
Bibliography:8 Pages
4 Tables
Appendix:8 Pages
Abstract

General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available atthis https URL.

View on arXiv
Comments on this paper