Learning Universal Policies via Text-Guided Video Generation

31 January 2023

Pieter Abbeel

Papers citing "Learning Universal Policies via Text-Guided Video Generation"

43 / 43 papers shown

Title
Pixel Motion as Universal Representation for Robot Control Kanchana Ranasinghe Xiang Li Cristina Mata J. Park Michael S. Ryoo VGen 27 0 0 12 May 2025
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions Qingwen Bu Y. Yang Jisong Cai Shenyuan Gao Guanghui Ren Maoqing Yao Ping Luo Hongyang Li 96 0 0 09 May 2025
CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations Anthony Liang Pavel Czempin Matthew Hong Yutai Zhou Erdem Biyik Stephen Tu 47 0 0 08 May 2025
Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability L. Wang Senmao Li Fei Yang Jianye Wang Ziheng Zhang Y. Liu Y. Wang Jian Yang DiffM 56 0 0 06 May 2025
Learning 3D Persistent Embodied World Models Siyuan Zhou Yilun Du Yuncong Yang Lei Han Peihao Chen Dit-Yan Yeung Chuang Gan VGen 47 0 0 05 May 2025
CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation Xiaoqi Li Lingyun Xu M. Zhang Jiaming Liu Yan Shen ... Jiahui Xu Liang Heng Siyuan Huang S. Zhang Hao Dong LM&Ro 44 0 0 04 May 2025
T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generation Xuyang Guo Jiayan Huo Zhenmei Shi Zhao-quan Song Jiahao Zhang Jiale Zhao EGVM VGen PINN 82 1 0 01 May 2025
A Survey of Interactive Generative Video Jiwen Yu Yiran Qin Haoxuan Che Quande Liu X. Wang Pengfei Wan Di Zhang Kun Gai Hao Chen Xihui Liu VGen 60 0 0 30 Apr 2025
Solving New Tasks by Adapting Internet Video Knowledge Calvin Luo Zilai Zeng Yilun Du Chen Sun 21 0 0 21 Apr 2025
AdaWorld: Learning Adaptable World Models with Latent Actions Shenyuan Gao Siyuan Zhou Yilun Du Jun Zhang Chuang Gan VGen 59 3 0 24 Mar 2025
Object-Centric World Model for Language-Guided Manipulation Youngjoon Jeong Junha Chun S. Cha Taesup Kim OCL VGen 138 1 0 08 Mar 2025
Generative Artificial Intelligence in Robotic Manipulation: A Survey Kun Zhang Peng Yun Jun Cen Junhao Cai DiDi Zhu ... Qifeng Chen Jia Pan Wei K. Zhang Bo Yang Hua Chen 59 1 0 05 Mar 2025
Unified Video Action Model Shuang Li Yihuai Gao Dorsa Sadigh Shuran Song VGen 48 1 0 28 Feb 2025
View-Invariant Policy Learning via Zero-Shot Novel View Synthesis Stephen Tian Blake Wulfe Kyle Sargent Katherine Liu Sergey Zakharov Vitor Campagnolo Guizilini Jiajun Wu 73 10 0 21 Feb 2025
Learning Human Skill Generators at Key-Step Levels Yilu Wu Chenhui Zhu Shuai Wang Hanlin Wang Jing Wang Zhaoxiang Zhang Limin Wang VGen 112 0 0 12 Feb 2025
DMWM: Dual-Mind World Model with Long-Term Imagination Lingyi Wang Rashed Shelim Walid Saad Naren Ramakrishnan LRM 139 1 0 11 Feb 2025
VILP: Imitation Learning with Latent Video Planning Zhengtong Xu Qiang Qiu Yu She VGen 72 1 0 03 Feb 2025
Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning Jiange Yang Haoyi Zhu Y. Wang Gangshan Wu Tong He Limin Wang 92 2 0 21 Nov 2024
Grounding Video Models to Actions through Goal Conditioned Exploration Yunhao Luo Yilun Du LM&Ro VGen 79 1 0 11 Nov 2024
NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields Eric Zhu Mara Levy M. Gwilliam Abhinav Shrivastava 40 0 0 04 Nov 2024
SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation Cheng-Chun Hsu Bowen Wen Jie Xu Yashraj S. Narang Xiaolong Wang Yuke Zhu Joydeep Biswas Stan Birchfield DiffM 35 8 0 01 Nov 2024
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation Hyungjoo Chae Namyoung Kim Kai Tzu-iunn Ong Minju Gwak Gwanwoo Song Jihoon Kim S. Kim Dongha Lee Jinyoung Yeo LLMAG 33 14 0 17 Oct 2024
Latent Action Pretraining from Videos Seonghyeon Ye Joel Jang Byeongguk Jeon Sejune Joo Jianwei Yang ... Kimin Lee J. Gao Luke Zettlemoyer Dieter Fox Minjoon Seo 35 27 0 15 Oct 2024
VideoAgent: Self-Improving Video Generation Achint Soni Sreyas Venkataraman Abhranil Chandra Sebastian Fischmeister Percy Liang Bo Dai Sherry Yang LM&Ro VGen 55 7 0 14 Oct 2024
DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering Jiaxu Wang Jingkai Sun Junhao He Ziyi Zhang Qiang Zhang Mingyuan Sun Renjing Xu AI4CE 30 0 0 11 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 82 25 0 04 Oct 2024
Embodiment-Agnostic Action Planning via Object-Part Scene Flow Weiliang Tang Jia-Hui Pan Wei Zhan Jianshu Zhou Huaxiu Yao Yun-Hui Liu M. Tomizuka Mingyu Ding Chi-Wing Fu 45 0 0 16 Sep 2024
Generative Hierarchical Materials Search Sherry Yang Simon L. Batzner Ruiqi Gao Muratahan Aykol Alexander L. Gaunt Brendan McMorrow Danilo J. Rezende Dale Schuurmans Igor Mordatch E. D. Cubuk AI4CE 32 5 0 10 Sep 2024
MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model Junjie Li Yang Liu Weiqing Liu Shikai Fang Lewen Wang Chang Xu Jiang Bian VGen 38 4 0 04 Sep 2024
VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation Wentao Zhao Jiaming Chen Ziyu Meng Donghui Mao Ran Song Wei Zhang 37 8 0 13 Jul 2024
Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators Wentao Zhang Junliang Guo Tianyu He Li Zhao Linli Xu Jiang Bian 34 3 0 10 Jul 2024
ARDuP: Active Region Video Diffusion for Universal Policies Shuaiyi Huang Mara Levy Zhenyu Jiang Anima Anandkumar Yuke Zhu Linxi Fan De-An Huang Abhinav Shrivastava VGen 42 2 0 19 Jun 2024
ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories Qianlan Yang Yu-Xiong Wang OnRL 34 1 0 06 Jun 2024
ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation Guanxing Lu Zifeng Gao Tianxing Chen Wen-Dao Dai Ziwei Wang Yansong Tang Yansong Tang DiffM 68 14 0 03 Jun 2024
Do's and Don'ts: Learning Desirable Skills with Instruction Videos Hyunseung Kim ByungKun Lee Hojoon Lee Dongyoon Hwang Donghu Kim Jaegul Choo 37 1 0 01 Jun 2024
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability Shenyuan Gao Jiazhi Yang Li Chen Kashyap Chitta Yihang Qiu Andreas Geiger Jun Zhang Hongyang Li 60 75 0 27 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 71 41 0 23 May 2024
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation Hongxin Zhang Zeyuan Wang Qiushi Lyu Zheyuan Zhang Sunli Chen Tianmin Shu Yilun Du Kwonjoon Lee Yilun Du Chuang Gan 41 12 0 16 Apr 2024
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains Rohan Myer Krishnan Zitian Tang Zhiqiu Yu Chen Sun 51 1 0 30 Nov 2023
Probabilistic Adaptation of Text-to-Video Models Mengjiao Yang Yilun Du Bo Dai Dale Schuurmans J. Tenenbaum Pieter Abbeel VGen DiffM 40 24 0 02 Jun 2023
DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics Ivan Kapelyukh Vitalis Vosylius Edward Johns LM&Ro DiffM 110 144 0 05 Oct 2022
Planning with Diffusion for Flexible Behavior Synthesis Michael Janner Yilun Du J. Tenenbaum Sergey Levine DiffM 202 627 0 20 May 2022
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets F. Ebert Yanlai Yang Karl Schmeckpeper Bernadette Bucher G. Georgakis Kostas Daniilidis Chelsea Finn Sergey Levine 161 218 0 27 Sep 2021