Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling

28 January 2023

Kolby Nottingham

Prithviraj Ammanabrolu

Yejin Choi

Papers citing "Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling"

50 / 62 papers shown

Title
Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach Minting Pan Yitao Zheng J. Li Yunbo Wang Xiaokang Yang OffRL 41 0 0 10 May 2025
Multi-agent Embodied AI: Advances and Future Directions Zhaohan Feng Ruiqi Xue Lei Yuan Yang Yu Ning Ding M. Liu Bingzhao Gao Jian-jun Sun Gang Wang AI4CE 52 0 0 08 May 2025
Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning Isadora White Kolby Nottingham Ayush Maniar Max Robinson Hansen Lillemark Mehul Maheshwari Lianhui Qin Prithviraj Ammanabrolu LLMAG LM&Ro 115 0 0 24 Apr 2025
Exploring 3D Activity Reasoning and Planning: From Implicit Human Intentions to Route-Aware Planning Xueying Jiang Wenhao Li Xiaoqin Zhang Ling Shao Shijian Lu LRM 42 0 0 17 Mar 2025
Uncertainty in Action: Confidence Elicitation in Embodied Agents Tianjiao Yu Vedant Shah Muntasir Wahed Kiet A. Nguyen Adheesh Sunil Juvekar Tal August Ismini Lourentzou 40 0 0 13 Mar 2025
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions Mourad Gridach Jay Nanavati Khaldoun Zine El Abidine Lenon Mendes Christina Mack 50 3 0 12 Mar 2025
DMWM: Dual-Mind World Model with Long-Term Imagination Lingyi Wang Rashed Shelim Walid Saad Naren Ramakrishnan LRM 112 1 0 11 Feb 2025
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web Hiroki Furuta Yutaka Matsuo Aleksandra Faust Izzeddin Gur CLL 85 13 0 03 Jan 2025
Project Sid: Many-agent simulations toward AI civilization Altera. AL Andrew Ahn Nic Becker Stephanie Carroll Nico Christie ... Shuying Luo Peter Y Wang Mathew Willows Feitong Yang Guangyu Robert Yang LLMAG LM&Ro AI4CE 58 12 0 31 Oct 2024
Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Driving Sihao Wu Jiaxu Liu Xiangyu Yin Guangliang Cheng Xingyu Zhao Meng Fang Xinping Yi Xiaowei Huang 26 0 0 16 Oct 2024
EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment Chen Gao Baining Zhao Weichen Zhang Jinzhu Mao Jun Zhang ... Jianjie Fang Zile Zhou Jinqiang Cui X. Chen Yong Li LM&Ro 35 10 0 12 Oct 2024
On the Modeling Capabilities of Large Language Models for Sequential Decision Making Martin Klissarov Devon Hjelm Alexander Toshev Bogdan Mazoure LM&Ro ELM OffRL LRM 29 2 0 08 Oct 2024
Open-World Reinforcement Learning over Long Short-Term Imagination Jiajian Li Q. Wang Yunbo Wang Xin Jin Yang Li Wenjun Zeng Xiaokang Yang OCL VLM 47 1 0 04 Oct 2024
Making Large Language Models into World Models with Precondition and Effect Knowledge Kaige Xie Ian Yang John Gunerli Mark Riedl 29 5 0 18 Sep 2024
Visual Grounding for Object-Level Generalization in Reinforcement Learning Haobin Jiang Zongqing Lu LM&Ro 25 2 0 04 Aug 2024
Enhancing Agent Learning through World Dynamics Modeling Zhiyuan Sun Haochen Shi Marc-Alexandre Côté Glen Berseth Xingdi Yuan Bang Liu 47 3 0 25 Jul 2024
KUNPENG: An Embodied Large Model for Intelligent Maritime Naiyao Wang Tongbang Jiang Ye Wang Shaoyang Qiu Bo Zhang ... Yiyang Wang Hongxiang Ren Ruili Wang Hongjun Shan Hongbo Liu 36 3 0 12 Jul 2024
Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models Logan Cross Violet Xiang Agam Bhatia Daniel L. K. Yamins Nick Haber LM&Ro LRM LLMAG 46 4 0 09 Jul 2024
From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models Eleni Nisioti Claire Glanois Elias Najarro Andrew Dai Elliot Meyerson J. Pedersen Laetitia Teodorescu Conor F. Hayes Shyam Sudhakaran Sebastian Risi AI4CE LM&Ro 37 2 0 14 Jun 2024
Can Language Models Serve as Text-Based World Simulators? Ruoyao Wang Graham Todd Ziang Xiao Xingdi Yuan Marc-Alexandre Côté Peter Clark Peter Alexander Jansen LLMAG 33 16 0 10 Jun 2024
CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning Xinrui Lin Yangfan Wu Huanyu Yang Yu Zhang Yanyong Zhang Jianmin Ji 19 2 0 05 Jun 2024
From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems Jianliang He Siyu Chen Fengzhuo Zhang Zhuoran Yang LM&Ro LLMAG 40 2 0 30 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 67 41 0 23 May 2024
Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs Bahar Radmehr Adish Singla Tanja Kaser LLMAG AI4CE 35 5 0 29 Apr 2024
Evolving Agents: Interactive Simulation of Dynamic and Diverse Human Personalities Jiale Li Jiayang Li Jiahao Chen Yifan Li Shijie Wang Hugo Zhou Minjun Ye Yunsheng Su AI4CE 40 4 0 03 Apr 2024
Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods Yuji Cao Huan Zhao Yuheng Cheng Ting Shu Guolong Liu Gaoqi Liang Junhua Zhao Yun Li LLMAG KELM OffRL LM&Ro 27 48 0 30 Mar 2024
Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoning Zhouhang Xie Bodhisattwa Prasad Majumder Mengjie Zhao Yoshinori Maeda Keiichi Yamada Hiromi Wakaki Julian McAuley 32 3 0 23 Mar 2024
Scaling Instructable Agents Across Many Simulated Worlds Sima Team Maria Abi Raad Arun Ahuja Catarina Barros F. Besse ... Daan Wierstra Duncan Williams Nathaniel Wong Sarah York Nick Young LM&Ro 110 38 0 13 Mar 2024
RL-GPT: Integrating Reinforcement Learning and Code-as-policy Shaoteng Liu Haoqi Yuan Minda Hu Yanwei Li Yukang Chen Shu Liu Zongqing Lu Jiaya Jia LLMAG 40 14 0 29 Feb 2024
WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment Hao Tang Darren Key Kevin Ellis LLMAG 18 27 0 19 Feb 2024
AgentLens: Visual Analysis for Agent Behaviors in LLM-based Autonomous Systems Jiaying Lu Bo Pan Jieyi Chen Yingchaojie Feng Jingyuan Hu Yuchen Peng Wei Chen 34 13 0 14 Feb 2024
Large Language Models as Minecraft Agents Chris Madge Massimo Poesio LLMAG 24 6 0 13 Feb 2024
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback Yufei Wang Zhanyi Sun Jesse Zhang Zhou Xian Erdem Biyik David Held Zackory M. Erickson VLM 53 48 0 06 Feb 2024
Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills Kolby Nottingham Bodhisattwa Prasad Majumder Bhavana Dalvi Sameer Singh Peter Clark Roy Fox 32 7 0 05 Feb 2024
Vision-Language Models Provide Promptable Representations for Reinforcement Learning William Chen Oier Mees Aviral Kumar Sergey Levine VLM LM&Ro 42 22 0 05 Feb 2024
Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models Xingzhou Lou Junge Zhang Ziyan Wang Kaiqi Huang Yali Du 25 3 0 15 Jan 2024
On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS) Vishal Pallagani Kaushik Roy Bharath Muppasani F. Fabiano Andrea Loreggia K. Murugesan Biplav Srivastava F. Rossi L. Horesh Amit P. Sheth 18 38 0 04 Jan 2024
MobileAgent: enhancing mobile control via human-machine interaction and SOP integration Tinghe Ding LLMAG LM&Ro 34 6 0 04 Jan 2024
LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers Taewook Nam Juyong Lee Jesse Zhang Sung Ju Hwang Joseph J. Lim Karl Pertsch OffRL LRM 43 4 0 14 Dec 2023
Learning adaptive planning representations with natural language guidance L. Wong Jiayuan Mao Pratyusha Sharma Zachary S. Siegel Jiahai Feng Noa Korneev Joshua B. Tenenbaum Jacob Andreas LM&Ro 21 21 0 13 Dec 2023
See and Think: Embodied Agent in Virtual Environment Zhonghan Zhao Wenhao Chai Xuan Wang Li Boyi Shengyu Hao Shidong Cao Tianbo Ye Gaoang Wang LM&Ro LLMAG 19 32 0 26 Nov 2023
RoboCLIP: One Demonstration is Enough to Learn Robot Policies S. Sontakke Jesse Zhang Sébastien M. R. Arnold Karl Pertsch Erdem Biyik Dorsa Sadigh Chelsea Finn Laurent Itti OffRL 14 63 0 11 Oct 2023
Compositional Semantics for Open Vocabulary Spatio-semantic Representations Robin Karlsson Francisco Lepe-Salazar K. Takeda VLM 40 1 0 08 Oct 2023
AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback Wanpeng Zhang Zongqing Lu LLMAG 21 6 0 29 Sep 2023
A Survey on Large Language Model based Autonomous Agents Lei Wang Chengbang Ma Xueyang Feng Zeyu Zhang Hao-ran Yang ... Xu Chen Yankai Lin Wayne Xin Zhao Zhewei Wei Ji-Rong Wen LLMAG AI4CE LM&Ro 39 1,101 0 22 Aug 2023
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis Izzeddin Gur Hiroki Furuta Austin Huang Mustafa Safdari Yutaka Matsuo Douglas Eck Aleksandra Faust LM&Ro LLMAG 25 195 0 24 Jul 2023
Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors Kolby Nottingham Yasaman Razeghi Kyungmin Kim JB Lanier Pierre Baldi Roy Fox Sameer Singh 18 8 0 21 Jul 2023
Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learning Michael Villarreal Bibek Poudel Weizi Li 22 23 0 13 Jun 2023
User Behavior Simulation with Large Language Model based Agents Lei Wang Jingsen Zhang Hao-ran Yang Zhiyuan Chen Jiakai Tang ... Wayne Xin Zhao Jun Xu Zhicheng Dou Jun Wang Ji-Rong Wen LM&Ro LLMAG 22 39 0 05 Jun 2023
STEVE-1: A Generative Model for Text-to-Behavior in Minecraft Shalev Lifshitz Keiran Paster Harris Chan Jimmy Ba Sheila A. McIlraith LM&Ro 19 66 0 01 Jun 2023