Title
SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models Cansu Sancaktar Christian Gumbsch Andrii Zadaianchuk Pavel Kolev Georg Martius LM&Ro VLM 55 0 0 03 Mar 2025
SFO: Piloting VLM Feedback for Offline RL Jacob Beck OffRL 26 0 0 02 Mar 2025
AppVLM: A Lightweight Vision Language Model for Online App Control Georgios Papoudakis Thomas Coste Zhihao Wu Jianye Hao J. Wang Kun Shao 43 1 0 10 Feb 2025
STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft Nicholas Lenzen Amogh Raut Andrew Melnik VGen 66 0 0 01 Dec 2024
LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble Yujeong Lee Sangwoo Shin Wei-Jin Park Honguk Woo OffRL 3DV 70 1 0 26 Nov 2024
Vision Language Models are In-Context Value Learners Yecheng Jason Ma Joey Hejna Ayzaan Wahid Chuyuan Fu Dhruv Shah ... Dinesh Jayaraman Wenhao Yu Tingnan Zhang Dorsa Sadigh Fei Xia 46 4 0 07 Nov 2024
Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos Harsh Mahesheka Zhixian Xie Z. Wang Wanxin Jin 23 0 0 11 Oct 2024
Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models Alain Andres Javier Del Ser OffRL 19 0 0 09 Oct 2024
Continuously Improving Mobile Manipulation with Autonomous Real-World RL Russell Mendonca Emmanuel Panov Bernadette Bucher Jiuguang Wang Deepak Pathak OffRL 25 4 0 30 Sep 2024
Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RL Eduardo Pignatelli Johan Ferret Tim Rockäschel Edward Grefenstette Davide Paglieri Samuel Coward Laura Toni 30 2 0 19 Sep 2024
MotIF: Motion Instruction Fine-tuning Minyoung Hwang Joey Hejna Dorsa Sadigh Yonatan Bisk 42 1 0 16 Sep 2024
Multimodal foundation world models for generalist embodied agents Pietro Mazzaglia Tim Verbelen Bart Dhoedt Aaron C. Courville Sai Rajeswar OffRL LM&Ro 32 1 0 26 Jun 2024
Open-Endedness is Essential for Artificial Superhuman Intelligence Edward Hughes Michael Dennis Jack Parker-Holder Feryal M. P. Behbahani Aditi Mavalankar Yuge Shi Tom Schaul Tim Rocktaschel LRM 32 18 0 06 Jun 2024
FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning Yuwei Fu Haichao Zhang Di Wu Wei-ping Xu Benoit Boulet VLM 24 12 0 02 Jun 2024
Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics Minttu Alakuijala Reginald McLean Isaac Woungang Nariman Farsad Samuel Kaski Pekka Marttinen Kai Yuan LM&Ro 27 0 0 30 May 2024
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning Yuexiang Zhai Hao Bai Zipeng Lin Jiayi Pan Shengbang Tong ... Alane Suhr Saining Xie Yann LeCun Yi-An Ma Sergey Levine LLMAG LRM 29 54 0 16 May 2024
BlenderAlchemy: Editing 3D Graphics with Vision-Language Models Ian Huang Guandao Yang Leonidas J. Guibas 29 3 0 26 Apr 2024
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code Ziniu Hu Ahmet Iscen Aashi Jain Thomas Kipf Yisong Yue David A. Ross Cordelia Schmid Alireza Fathi LLMAG 34 23 0 02 Mar 2024
Video as the New Language for Real-World Decision Making Sherry Yang Jacob Walker Jack Parker-Holder Yilun Du Jake Bruce Andre Barreto Pieter Abbeel Dale Schuurmans VGen 16 45 0 27 Feb 2024
Code as Reward: Empowering Reinforcement Learning with VLMs David Venuto Sami Nur Islam Martin Klissarov Doina Precup Sherry Yang Ankit Anand VLM 13 9 0 07 Feb 2024
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences Xiyao Wang Yuhang Zhou Xiaoyu Liu Hongjin Lu Yuancheng Xu ... Taixi Lu Gedas Bertasius Mohit Bansal Huaxiu Yao Furong Huang LRM VLM 73 65 0 19 Jan 2024
RePLan: Robotic Replanning with Perception and Language Models Marta Skreta Zihan Zhou Jia Lin Yuan Kourosh Darvish Alán Aspuru-Guzik Animesh Garg LM&Ro LRM 27 26 0 08 Jan 2024
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning Juan Rocamonde Victoriano Montesinos Elvis Nava Ethan Perez David Lindner VLM 31 73 0 19 Oct 2023
Vision-Language Models as Success Detectors Yuqing Du Ksenia Konyushkova Misha Denil A. Raju Jessica Landon Felix Hill Nando de Freitas Serkan Cabi MLLM LRM 82 76 0 13 Mar 2023
Foundation Models for Decision Making: Problems, Methods, and Opportunities Sherry Yang Ofir Nachum Yilun Du Jason W. Wei Pieter Abbeel Dale Schuurmans LM&Ro OffRL LRM AI4CE 87 148 0 07 Mar 2023
VideoGPT: Video Generation using VQ-VAE and Transformers Wilson Yan Yunzhi Zhang Pieter Abbeel A. Srinivas ViT VGen 237 482 0 20 Apr 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 220 450 0 11 Feb 2021