Learning to Model the World with Language

31 July 2023

Pieter Abbeel

Papers citing "Learning to Model the World with Language"

49 / 49 papers shown

Title
Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows Wenhao Li Bo Jin Mingyi Hong Changhong Lu Xiangfeng Wang 46 0 0 07 May 2025
Looking beyond the next token Abitha Thankaraj Yiding Jiang J. Zico Kolter Yonatan Bisk LRM 51 1 0 15 Apr 2025
Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning Q. Wang Z. Zhang Baao Xie Xin Jin Y. Wang Shiyu Wang Liaomo Zheng Xiaokang Yang Wenjun Zeng OffRL 63 0 0 11 Mar 2025
Learning Actionable World Models for Industrial Process Control Peng Yan Ahmed Abdulkadir Gerrit A. Schatte Giulia Anguzzi Joonsu Gha Nikola Pascher Matthias Rosenthal Yunlong Gao Benjamin Grewe Thilo Stadelmann DRL AI4CE 49 0 0 03 Mar 2025
Zero-shot Model-based Reinforcement Learning using Large Language Models Abdelhakim Benechehab Youssef Attia El Hili Ambroise Odonnat Oussama Zekri Albert Thomas Giuseppe Paolo Maurizio Filippone I. Redko Balázs Kégl OffRL 62 1 0 17 Feb 2025
DMWM: Dual-Mind World Model with Long-Term Imagination Lingyi Wang Rashed Shelim Walid Saad Naren Ramakrishnan LRM 97 1 0 11 Feb 2025
Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards Shresth Verma Niclas Boehmer Lingkai Kong Milind Tambe 69 2 0 17 Jan 2025
Navigation World Models Amir Bar G. Zhou Danny Tran Trevor Darrell Yann LeCun VGen EgoV 80 14 0 04 Dec 2024
Evaluating World Models with LLM for Decision Making Chang Yang Xinrun Wang Junzhe Jiang Qinggang Zhang Xiao Huang LLMAG ELM 36 2 0 13 Nov 2024
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use Jiajun Xi Yinong He Jianing Yang Yinpei Dai Joyce Chai LM&Ro 24 2 0 31 Oct 2024
Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions Kyungmin Kim JB Lanier Pierre Baldi Charless C. Fowlkes Roy Fox 23 1 0 13 Oct 2024
Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments Zoya Volovikova A. Skrynnik Petr Kuderov Aleksandr I. Panov LLMAG LM&Ro 30 0 0 12 Jul 2024
Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models Logan Cross Violet Xiang Agam Bhatia Daniel L. K. Yamins Nick Haber LM&Ro LRM LLMAG 46 4 0 09 Jul 2024
iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvement Aoyu Pang Maonan Wang Man-On Pun Chung Shue Chen Xi Xiong 43 9 0 08 Jul 2024
From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models Eleni Nisioti Claire Glanois Elias Najarro Andrew Dai Elliot Meyerson J. Pedersen Laetitia Teodorescu Conor F. Hayes Shyam Sudhakaran Sebastian Risi AI4CE LM&Ro 35 2 0 14 Jun 2024
World Models with Hints of Large Language Models for Goal Achieving Zeyuan Liu Ziyu Huan Xiyao Wang Jiafei Lyu Jian Tao Xiu Li Furong Huang Huazhe Xu LM&Ro LRM AI4CE 29 1 0 11 Jun 2024
SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation Junjie Zhang Chenjia Bai Haoran He Wenke Xia Zhigang Wang Bin Zhao Xiu Li Xuelong Li 35 12 0 30 May 2024
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability Shenyuan Gao Jiazhi Yang Li Chen Kashyap Chitta Yihang Qiu Andreas Geiger Jun Zhang Hongyang Li 60 75 0 27 May 2024
Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search Nicola Dainese Matteo Merler Minttu Alakuijala Pekka Marttinen LLMAG 36 7 0 24 May 2024
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration Yang Zhang Shixin Yang Chenjia Bai Fei Wu Xiu Li Zhen Wang Xuelong Li LLMAG 31 25 0 23 May 2024
Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods Yuji Cao Huan Zhao Yuheng Cheng Ting Shu Guolong Liu Gaoqi Liang Junhua Zhao Yun Li LLMAG KELM OffRL LM&Ro 27 48 0 30 Mar 2024
ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation Guanxing Lu Shiyi Zhang Ziwei Wang Changliu Liu Jiwen Lu Yansong Tang 44 49 0 13 Mar 2024
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation Guosheng Zhao Xiaofeng Wang Zheng Zhu Xinze Chen Guan Huang Xiaoyi Bao Xingang Wang VGen 33 14 0 11 Mar 2024
Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation Joseph Cho Fachrina Dewi Puspitasari Sheng Zheng Jingyao Zheng Lik-Hang Lee Tae-Ho Kim Choong Seon Hong Chaoning Zhang EGVM VGen 36 40 0 08 Mar 2024
Real-World Robot Applications of Foundation Models: A Review Kento Kawaharazuka T. Matsushima Andrew Gambardella Jiaxian Guo Chris Paxton Andy Zeng OffRL VLM LM&Ro 41 45 0 08 Feb 2024
Vision-Language Models Provide Promptable Representations for Reinforcement Learning William Chen Oier Mees Aviral Kumar Sergey Levine VLM LM&Ro 39 22 0 05 Feb 2024
Language-Guided World Models: A Model-Based Approach to AI Control Alex Zhang Khanh Nguyen Jens Tuyls Albert Lin Karthik Narasimhan LLMAG 29 5 0 24 Jan 2024
WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens Xiaofeng Wang Zheng Zhu Guan Huang Boyuan Wang Xinze Chen Jiwen Lu VGen 27 32 0 18 Jan 2024
QUAR-VLA: Vision-Language-Action Model for Quadruped Robots Pengxiang Ding Han Zhao Wenxuan Song Zhitao Wang Zhenyu Wei Shangke Lyu Ningxi Yang Donglin Wang 30 19 0 22 Dec 2023
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX Alexander Nikulin Vladislav Kurenkov Ilya Zisman Artem Agarkov Viacheslav Sinii Sergey Kolesnikov 24 23 0 19 Dec 2023
Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning Sabrina McCallum Max Taylor-Davies Stefano V. Albrecht Alessandro Suglia 14 1 0 07 Dec 2023
FoMo Rewards: Can we cast foundation models as reward functions? Ekdeep Singh Lubana Johann Brehmer P. D. Haan Taco S. Cohen OffRL LRM 33 2 0 06 Dec 2023
LanGWM: Language Grounded World Model Rudra P. K. Poudel Harit Pandya Chao Zhang Roberto Cipolla 13 5 0 29 Nov 2023
LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Model Siwei Chen Anxing Xiao David Hsu LM&Ro 16 5 0 29 Nov 2023
Applications of Large Scale Foundation Models for Autonomous Driving Yu Huang Yue Chen Zhu Li ELM AI4CE LRM ALM LM&Ro 46 15 0 20 Nov 2023
DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving Xiaofeng Wang Zheng Hua Zhu Guan Huang Xinze Chen Jiagang Zhu Jiwen Lu VGen 16 148 0 18 Sep 2023
LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Querying T. G. Karimpanal Laknath Semage Santu Rana Hung Le T. Tran Sunil R. Gupta Svetha Venkatesh RALM LLMAG 10 2 0 21 Aug 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 13 2 0 23 May 2023
ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments Dongyan An H. Wang Wenguan Wang Zun Wang Yan Huang Keji He Liang Wang 52 62 0 06 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 262 4,223 0 30 Jan 2023
Improving Policy Learning via Language Dynamics Distillation Victor Zhong Jesse Mu Luke Zettlemoyer Edward Grefenstette Tim Rocktaschel OffRL 32 15 0 30 Sep 2022
Improving Intrinsic Exploration with Language Abstractions Jesse Mu Victor Zhong Roberta Raileanu Minqi Jiang Noah D. Goodman Tim Rocktaschel Edward Grefenstette 95 63 0 17 Feb 2022
Waypoint Models for Instruction-guided Navigation in Continuous Environments Jacob Krantz Aaron Gokaslan Dhruv Batra Stefan Lee Oleksandr Maksymets LM&Ro 126 76 0 05 Oct 2021
Skill Induction and Planning with Latent Language Pratyusha Sharma Antonio Torralba Jacob Andreas LM&Ro 194 108 0 04 Oct 2021
TEACh: Task-driven Embodied Agents that Chat Aishwarya Padmakumar Jesse Thomason Ayush Shrivastava P. Lange Anjali Narayan-Chen Spandana Gella Robinson Piramithu Gökhan Tür Dilek Z. Hakkani-Tür LM&Ro 155 180 0 01 Oct 2021
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks Cristian-Paul Bara Sky CH-Wang J. Chai 65 61 0 13 Sep 2021
Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning H. Wang Victor Zhong Karthik Narasimhan 76 53 0 19 Jan 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 275 1,583 0 18 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 406 2,576 0 03 Sep 2019