Video as the New Language for Real-World Decision Making

27 February 2024

Pieter Abbeel

Papers citing "Video as the New Language for Real-World Decision Making"

44 / 44 papers shown

Title
Latent Diffusion Planning for Imitation Learning Amber Xie Oleh Rybkin Dorsa Sadigh Chelsea Finn 33 1 0 23 Apr 2025
Solving New Tasks by Adapting Internet Video Knowledge Calvin Luo Zilai Zeng Yilun Du Chen Sun 21 0 0 21 Apr 2025
Exploration-Driven Generative Interactive Environments N. Savov Naser Kazemi Mohammad Mahdi Danda Pani Paudel Xi Wang Luc Van Gool VGen 3DV 38 0 0 03 Apr 2025
Exploring the Evolution of Physics Cognition in Video Generation: A Survey Minghui Lin Xiang Wang Y. Wang Shu Wang Fengqi Dai ... Cunxiang Wang Zhengrong Zuo Nong Sang Siteng Huang Donglin Wang EGVM VGen 78 3 0 27 Mar 2025
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models Qingqing Zhao Yao Lu Moo Jin Kim Zipeng Fu Zhuoyang Zhang ... Ankur Handa Ming-Yu Liu Donglai Xiang Gordon Wetzstein Tsung-Yi Lin LM&Ro LRM 43 10 0 27 Mar 2025
Synthetic Video Enhances Physical Fidelity in Video Synthesis Qi Zhao Xingyu Ni Ziyu Wang Feng Cheng Ziyan Yang Lu Jiang Bohan Wang VGen 41 2 0 26 Mar 2025
AdaWorld: Learning Adaptable World Models with Latent Actions Shenyuan Gao Siyuan Zhou Yilun Du Jun Zhang Chuang Gan VGen 54 3 0 24 Mar 2025
Position: Interactive Generative Video as Next-Generation Game Engine Jiwen Yu Yiran Qin Haoxuan Che Quande Liu Xintao Wang Pengfei Wan Di Zhang Xihui Liu VGen 45 1 0 21 Mar 2025
Impossible Videos Zechen Bai Hai Ci Mike Zheng Shou EGVM VGen 70 0 0 18 Mar 2025
LuciBot: Automated Robot Policy Learning from Generated Videos Xiaowen Qiu Yian Wang Jiting Cai Zhehuan Chen Chunru Lin Tsun-Hsuan Wang Chuang Gan LM&Ro VGen 67 0 0 12 Mar 2025
PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop Chenyu Li Oscar Michel Xichen Pan Sainan Liu Mike Roberts Saining Xie VGen 53 3 0 12 Mar 2025
InfinityDrive: Breaking Time Limits in Driving World Models Xi Guo C. Ding Haoxuan Dou Xin Zhang Weixuan Tang Wei Yu Wu VGen 81 5 0 02 Dec 2024
Generating 3D-Consistent Videos from Unposed Internet Photos Gene Chou Kai Zhang Sai Bi Hao Tan Zexiang Xu Fujun Luan Bharath Hariharan Noah Snavely 3DGS VGen 75 3 0 20 Nov 2024
Grounding Video Models to Actions through Goal Conditioned Exploration Yunhao Luo Yilun Du LM&Ro VGen 77 1 0 11 Nov 2024
Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling Mingtong Zhang Kaifeng Zhang Yunzhu Li 3DGS AI4CE 23 3 0 24 Oct 2024
Latent Action Pretraining from Videos Seonghyeon Ye Joel Jang Byeongguk Jeon Sejune Joo Jianwei Yang ... Kimin Lee Jianfeng Gao Luke Zettlemoyer Dieter Fox Minjoon Seo 30 27 0 15 Oct 2024
Asymptotic Analysis of Sample-averaged Q-learning Saunak Kumar Panda Ruiqi Liu Yisha Xiang OnRL 52 8 0 14 Oct 2024
VideoAgent: Self-Improving Video Generation Achint Soni Sreyas Venkataraman Abhranil Chandra Sebastian Fischmeister Percy Liang Bo Dai Sherry Yang LM&Ro VGen 48 7 0 14 Oct 2024
Learning Truncated Causal History Model for Video Restoration Amirhosein Ghasemabadi Muhammad Kamran Janjua Mohammad Salameh Di Niu VGen CML 21 2 0 04 Oct 2024
Generative Modeling of Molecular Dynamics Trajectories Bowen Jing Hannes Stärk Tommi Jaakkola Bonnie Berger AI4CE 27 14 0 26 Sep 2024
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation Homanga Bharadhwaj Debidatta Dwibedi Abhinav Gupta Shubham Tulsiani Carl Doersch Ted Xiao Dhruv Shah Fei Xia Dorsa Sadigh Sean Kirmani VGen LM&Ro 35 27 0 24 Sep 2024
Learning Generative Interactive Environments By Trained Agent Exploration Naser Kazemi N. Savov Danda Paudel Luc Van Gool 18 2 0 10 Sep 2024
Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance Renming Huang Shaochong Liu Yunqiang Pei Peng Wang Guoqing Wang Yang Yang Hengtao Shen OffRL 19 0 0 06 Sep 2024
MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model Junjie Li Yang Liu Weiqing Liu Shikai Fang Lewen Wang Chang Xu Jiang Bian VGen 31 4 0 04 Sep 2024
Controlling the World by Sleight of Hand Sruthi Sudhakar Ruoshi Liu Basile Van Hoorick Carl Vondrick Richard Zemel 27 3 0 13 Aug 2024
Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics Ruining Li Chuanxia Zheng Christian Rupprecht Andrea Vedaldi DiffM VGen 36 9 0 08 Aug 2024
Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions Ashkan Taghipour Morteza Ghahremani Bennamoun Aref Miri Rekavandi Zinuo Li Hamid Laga F. Boussaïd VGen 68 2 0 27 Jul 2024
Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators Wentao Zhang Junliang Guo Tianyu He Li Zhao Linli Xu Jiang Bian 32 3 0 10 Jul 2024
ARDuP: Active Region Video Diffusion for Universal Policies Shuaiyi Huang Mara Levy Zhenyu Jiang Anima Anandkumar Yuke Zhu Linxi Fan De-An Huang Abhinav Shrivastava VGen 40 2 0 19 Jun 2024
Do's and Don'ts: Learning Desirable Skills with Instruction Videos Hyunseung Kim ByungKun Lee Hojoon Lee Dongyoon Hwang Donghu Kim Jaegul Choo 37 1 0 01 Jun 2024
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability Shenyuan Gao Jiazhi Yang Li Chen Kashyap Chitta Yihang Qiu Andreas Geiger Jun Zhang Hongyang Li 60 75 0 27 May 2024
Position: Foundation Agents as the Paradigm Shift for Decision Making Xiaoqian Liu Xingzhou Lou Jianbin Jiao Junge Zhang OffRL LLMAG 31 5 0 27 May 2024
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World Yifei Huang Guo Chen Jilan Xu Mingfang Zhang Lijin Yang ... Hongjie Zhang Lu Dong Yali Wang Limin Wang Yu Qiao EgoV 52 35 0 24 Mar 2024
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding Yi Wang Kunchang Li Xinhao Li Jiashuo Yu Yinan He ... Hongjie Zhang Yifei Huang Yu Qiao Yali Wang Limin Wang 27 44 0 22 Mar 2024
Probabilistic World Modeling with Asymmetric Distance Measure Meng Song 19 0 0 16 Mar 2024
Lumiere: A Space-Time Diffusion Model for Video Generation Omer Bar-Tal Hila Chefer Omer Tov Charles Herrmann Roni Paiss ... T. Michaeli Oliver Wang Deqing Sun Tali Dekel Inbar Mosseri VGen 101 214 0 23 Jan 2024
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets A. Blattmann Tim Dockhorn Sumith Kulal Daniel Mendelevitch Maciej Kilian ... Zion English Vikram S. Voleti Adam Letts Varun Jampani Robin Rombach VGen 150 985 0 25 Nov 2023
Video Language Planning Yilun Du Mengjiao Yang Peter R. Florence Fei Xia Ayzaan Wahid ... Pieter Abbeel Josh Tenenbaum L. Kaelbling Andy Zeng Jonathan Tompson PINN LM&Ro 89 83 0 16 Oct 2023
Vision-Language Models as Success Detectors Yuqing Du Ksenia Konyushkova Misha Denil A. Raju Jessica Landon Felix Hill Nando de Freitas Serkan Cabi MLLM LRM 84 76 0 13 Mar 2023
Foundation Models for Decision Making: Problems, Methods, and Opportunities Sherry Yang Ofir Nachum Yilun Du Jason W. Wei Pieter Abbeel Dale Schuurmans LM&Ro OffRL LRM AI4CE 90 148 0 07 Mar 2023
Diffusion Models in Vision: A Survey Florinel-Alin Croitoru Vlad Hondru Radu Tudor Ionescu M. Shah DiffM VLM MedIm 186 1,098 0 10 Sep 2022
Score-based Generative Models for Calorimeter Shower Simulation Vinicius Mikuni Benjamin Nachman DiffM 22 75 0 17 Jun 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022